The team comes from a variety of backgrounds. While some spent a decade building technology to detect financial crimes, others were architecting data fabrics at fortune 100 companies.
Regardless of the industry or experience, we all faced similar challenges as it related to data quality.
These unique vantage points have allowed us to understand the most common data quality challenges organizations are facing.
We tried many of the traditional tools and techniques. The biggest problem was always the amount of time it took to do everything needed to implement and maintain data quality controls.
You get left with half-baked data quality coverage and the right controls are added only after issues occur.
It turned out teams were doing the same tasks for every dataset and for each department, building the exact same tools over and over.
Traditional approaches are very manual.
Start by opening a sample or spreadsheet and conduct analysis (Table-by-table, column-by-column, query-by-query, and item-by-item).
Next, manually craft the rules, conditions, tolerances, and thresholds.
Then stitch together the dashboards, scoring, workflows, alerts, and reporting. And you wonder why bare-minimum coverage is common.
Now that the surface area of the data in an organization is so much larger, these traditional techniques don't hold up.
What we needed didn't exist. As lifelong data enthusiasts, we wanted a tool that could alert us when data changes occurred without complicated setup and lengthy analysis. We sought something that could work on big, medium, and small data and across all storage formats. Upon evaluating all the commercially available tools, and assessing costs and time of homegrown solutions, there were no great options.