There is a specific moment that tells you a data platform has failed, and it is not an outage. It is a meeting where someone presents a number, someone else says that does not look right, and the room agrees to check it offline. Nobody trusts the chart. Everyone keeps their own spreadsheet.
The usual response is to build a better dashboard. That almost never works, because the problem was never presentation. The pipeline delivered wrong data and nothing objected, so the dashboard faithfully rendered something false. Adding another view of untrusted data produces another untrusted view.
Silence is the actual defect
Most pipelines are built to succeed. A job runs, transforms what it is given, writes the result, exits zero. If a source system starts sending nulls in a required column, the job succeeds. If a join silently drops 30% of rows because an upstream key format changed, the job succeeds. If a currency column switches from dollars to cents, the job succeeds enthusiastically.
In every case the pipeline did what it was told. It was never told to have an opinion about whether the data was plausible. That absence of opinion is the defect - and it is why data quality tests in data engineering services return more than almost any other investment.
Four test types that catch nearly everything
You do not need an elaborate framework. Four categories, applied at the point data enters your control, catch the overwhelming majority of real incidents.
- Schema tests. Does this table have the columns expected, with the types expected? Catches upstream changes nobody announced - the single most common cause of a quiet corruption.
- Volume tests. Is today's row count within a plausible band of recent history? A table that normally gains 40,000 rows and gains 4 has a broken source, regardless of whether the job succeeded.
- Freshness tests. What is the newest timestamp, and is it recent enough to be usable? Stale data presented as current is more dangerous than missing data, because it looks fine.
- Distribution tests. Null rates, cardinality, and value ranges against recent norms. This is what catches the unit change, the new enum value, and the region that stopped reporting.
Together these are usually a day or two of work per critical pipeline, and they are the difference between finding out from a test and finding out from a board meeting.
Fail the pipeline, do not just log a warning
The decision that matters most is what happens when a test fails. Teams often start with warnings, because stopping a pipeline feels dangerous. Warnings get ignored within about three weeks. This is reliable enough to plan around.
Failing loudly is the right default. When a quality test fails, the pipeline stops and the downstream table keeps yesterday's known-good data rather than being overwritten with something suspect. Stale-but-correct beats fresh-but-wrong in almost every business context, because people can reason about a table that is a day behind. They cannot reason about a table that is subtly poisoned.
A dashboard that is a day stale gets a caveat. A dashboard that is quietly wrong gets a decision made on it.
The exception is genuinely non-critical enrichment - a nice-to-have attribute where partial data is better than none. Mark those explicitly as warn-only. The point is that it becomes a deliberate per-test decision rather than a blanket default nobody chose.
What actually changes operationally
The visible effect is fewer bad numbers. The larger effect is on how the team spends its time.
Before tests, the discovery path for a data problem runs: someone notices an implausible number, raises it days or weeks later, an analyst investigates, traces it back through several transformations, and eventually finds an upstream change from three weeks ago. That is often days of work, and the intervening decisions were made on bad data.
After tests, the same problem surfaces as a specific failure on a specific table within one pipeline run, naming the column and the expectation it violated. Diagnosis is minutes, not days, and no bad number ever reached anyone.
There is also a second-order effect worth naming. Once tests exist, upstream teams start hearing about their changes immediately instead of silently breaking things for weeks. That feedback loop tends to improve upstream discipline on its own, without anyone having to run a governance program to enforce it.
Where to start
Not everywhere at once. Pick the three tables that feed the numbers leadership actually looks at, and put all four test categories on those. That is usually under a week, and it protects the data whose wrongness would be most expensive.
Then add ownership. Every tested table gets a named owner who receives the failure. A test that alerts a shared channel with no owner is a test that gets muted. The technical work is the easy half; the half that determines whether it survives is deciding who is responsible when it fires.
