The problem
The client operates a large manufacturing plant with several thousand IoT sensors across its production lines, tracking everything from vibration and temperature on machinery to throughput counters on the line, and generating roughly twelve terabytes of raw data per day. The data had technically been collected for years. But the ingestion pipeline feeding the data warehouse dropped a meaningful and inconsistent share of readings, had no reliable schema across the 340 sensor types in use, and had accumulated so many quiet failures that the plant's reliability engineers had largely stopped querying it. They relied instead on spreadsheets built from a handful of sensors they'd verified by hand.
This is a specific and common failure mode: data infrastructure that exists, generates a real-looking volume number, and is functionally useless for the analysis it was supposedly built to support. The plant wanted to use this data for predictive maintenance (catching equipment issues before a failure), but that only works if the data is complete and trustworthy enough to build a model on, and it wasn't.
What we actually did
We rebuilt the ingestion pipeline from the sensor gateways through to the warehouse, treating schema consistency and ingestion completeness as the primary engineering problems, ahead of anything related to analysis or modeling on top of the data.
Weeks 1-2: auditing what was actually being lost, and why
Before rebuilding anything, we instrumented the existing pipeline to measure how much data was being dropped and where; nobody had a real number, just a sense that "some data goes missing." Actual completeness was about 81%, worse than the plant's own estimate. The losses were concentrated at one ingestion-queue stage that silently dropped messages during load spikes instead of applying backpressure or retrying, a bug that appeared to date from close to the pipeline's original build.
Weeks 3-5: schema normalization across 340 sensor types
The 340 sensor types, added over years by different vendors and plant engineers, used inconsistent units and field names, and in a few cases the same field name meant different things on different models. We built a normalization layer that maps each sensor type's raw output to a consistent schema at ingestion. The mapping rules are version-controlled and reviewed by the plant's reliability engineering team, because a silently wrong unit conversion or field mapping would have reintroduced the same trust problem.
Weeks 6-8: backpressure-aware ingestion and completeness monitoring
The rebuilt ingestion layer handles load spikes (which happen predictably during shift changes and certain production events) with proper backpressure and retry logic instead of silent drops. We also built a live completeness dashboard broken down by sensor type and production line, so a drop anywhere is visible immediately rather than discovered months later.
Challenges and tradeoffs
- Finding the silent-drop bug required instrumentation the original pipeline simply didn't have. The existing system had no meaningful completeness monitoring, which is exactly how a bug dropping data for years went unnoticed. We built the audit instrumentation as separate infrastructure from the production pipeline, so measuring the old system's behavior wouldn't change what we were observing.
- Getting plant engineers to trust newly normalized historical data took more than fixing the pipeline going forward. Reliability engineers who'd been burned by bad data for years were reasonably skeptical that the rebuild had fixed anything. We ran a joint validation exercise: we reprocessed three months of historical raw sensor logs through the new normalization layer, and the plant's engineers spot-checked the output against equipment events they remembered. That did more for trust than any completeness percentage we could report.
- The two-tier ingestion split meant two things to maintain instead of one, a real ongoing cost. Isolating high-volume sensors in their own ingestion path meant the plant's data engineering team had to understand and maintain two related systems instead of one. We documented the rationale for the split and shared tooling where possible (the normalization and monitoring layers serve both tiers) so the maintenance burden didn't double.
Results
Ingestion completeness rose from roughly 81% to 99.7%, measured over the two months after full production rollout. The remaining gap is concentrated in a few physically remote sensors with known intermittent connectivity problems unrelated to the pipeline. All 340 sensor types are now normalized into one consistent, queryable schema, and within the first month the reliability engineering team used it to build the predictive-maintenance analysis that motivated the project.
The plant's head of reliability engineering credited the historical-data validation exercise, as much as the new completeness numbers, with getting the broader engineering team to use the data again after years of working around it. That mattered as much to the project's success as the pipeline's technical correctness.
What we'd do differently
We'd make completeness monitoring the very first step of the audit, in the first days of week one, rather than starting with a general code review and adding instrumentation midweek. We found the silent-drop bug quickly regardless, but monitoring first would have given a cleaner baseline to compare the rebuild against.