The 4 Ways Intelligent Automation Helps Achieve DataOps
To accommodate the growing popularity of self-service data preparation among our customers, we focused our most recent 2019 release on fusing the generation of data prep flows and the operationalization of them. The larger the team of data analysts, data engineers, and data practitioners, the more data preparation flows get created and reused across the organization. Instead of making operationalization of these flows a separate task, we wanted the system to handle that intelligently.
Say hello to Intelligent Automation.
In speaking with customers who are using the new release, it is fulfilling to hear them express the benefits of our recent innovation. I repeatedly hear customers articulate one of four particular aspects of Paxata Intelligent Automation.
1) Automated Runtime and Dependency Management
With a single click, Paxata automatically detects and sequences all related data prep projects into a single data flow. For example, take one data preparation project that is dependent on a dozen other projects, all curated from the teamwork and collaboration of several data practitioners.
Then, instead of manually linking the various data flows together—typically a task done by IT or a data engineer—Paxata Intelligent Automation will recursively map all the dependencies. That way, when a single project is set to run, the rest execute accordingly.
“Paxata’s new Intelligent Automation greatly improves our organization’s ability to accelerate the continuous delivery of our customer master and data quality projects. We can now automate any curated data preparation workflow with a single click, and Paxata intelligently discovers and links all the dependent projects together, which helps us save time and simplify our data operations.” — Byron Hernandez, Senior Data Analyst, Cox Automotive.
2) Visual Impact Analysis
The new Automated Project Flow graph is useful to see how incoming data is impacting downstream processes. This view gives full visibility to all upstream sources and downstream outputs for any given data preparation project, helping the user visualize the impact before making any changes.
3) Tracking the History and Analyzing Data Snapshots
For an operationalized process, the ability to record (what ran when to generate what results), audit, and even troubleshoot is very important. Paxata Intelligent Automation has this functionality built in, so business analysts or data engineers can set it and forget it. For every automation job that runs, Paxata automatically snapshots the version of Dataset and Projects that were executed. If you need to go back to the last quarter and analyze what data and transformations resulted in the answers, you have the ability to do so quickly and easily.
Paxata also automatically versions the results so you can whip up a periodic trend report.
4) Integration with Other Data Processes
In enterprises where an orchestration engine already exists, Paxata’s Intelligent Automation can be invoked within that context using REST APIs. This helps processing of data preparation flows within a larger data architecture. The same automated recursive map ensures that all the dependencies are sequenced appropriately for every run.
Many of these points reinforce the core principles of DataOps, ensuing the orchestration of data to production use cases and agile deployment of new changes. To scale data preparation at the enterprise, one can either hire and assemble a team of DataOps professionals or take advantage of intelligence algorithms that facilitate the same process with no or little DataOps oversight. The choice, I think you’ll agree, is an easy one.