Birth of the Data Prep Market
In the last several years there has been an explosive growth in the number of data preparation tools. It seems to me that more vendors are strategically positioning and marketing themselves as the solution to the amount of time end users spend cleaning and preparing data for business insight. As a data prep enthusiast this is exciting and refreshing to me. I believe the growth and proliferation of these tools marks the birth of the ‘Data Prep Market’. My goal in this blog is not to tell you which tool is the best but to help promote the discipline. So how did we get here? Why the upward trend in the number of data prep tools? Just as diamonds are forged in intense heat and pressure over time, so too have data prep tools been forged from the intense heat of business demands to compete against peers, and pressing demands for data insight, which occurred frequently over the last two or three decades of Traditional Business Intelligence & Analytics.
How Data Prep Positively Disrupts Traditional BI/Analytics.
Here’s a short list of typical needs both Business and IT:
Business Demands
Access (Access to all types and formats of data necessary)
Agility (Ability to keep pace with constantly evolving demands)
Autonomy (Without IT Support)
Automation (Repeat-ability of processes)
Collaboration (Ability to work seamlessly with internal and external teams)
Ease of Use (Minimal to No-Code)
Iterative (Freedom to explore and ask more advanced questions of data)
Scale (Ability to work with larger datasets)
Speed (Fast time to Insight)
IT Requirements
Audit (Ability to identify events and access to data)
Control (Ability to Control Access to Data)
‘Governance (Foster shared ownership and management of data as an asset)
Procedure (Follow the right process to request access to information)
Project (Determining the ROI vs Costs to prioritize data projects)
Security (protecting information)
When we take a look at traditional tools and platforms, not many of them met the demands and requirements of both the Business and IT. Unfortunately someone usually had to sacrifice. For example, if a Business leader believes that he has the opportunity to have access to all necessary data, apply agility in the process, work autonomously without IT support, automate mundane tasks, have the freedom to collaborate with internal and external teams, work with easy to use tools that require minimal to no-coding, the ability to interrogate the data, work with larger datasets, all with faster speed to insight, he will take the opportunity, no matter the cost. In extreme cases I believe that the Business will sabotage (knowingly and unknowingly) IT requirements if it will help them get what they need. Naturally my lists of Business demands and IT requirements are not exhaustive, neither are all Business leaders looking to work against IT to accomplish their goals, but my goal was to highlight this real and pervasive conflict. From this conflict arose the need for tools that would appease both parties: Enter Data Prep.
As the data prep market continues to grow I believe the next step in maturity will be the creation of dedicated Data Prep Professionals who will be well aligned with both the needs of the Business and the Chief Data Officer.