Janus, The Missing CI for ML Supply Chains

Janus is a reliability agent for the boundary between a warehouse and the models trained on it, a boundary DataHub is the only catalog that holds both sides of but never joins. We built it over five weeks for "Build with DataHub: The Agent Hackathon," entered in the Production ML Agents category; judging is still open as of this writing. Janus reads DataHub's column-level lineage and its ML metadata, writes the join between them that no ingestion source produces on its own, and walks that join to catch what live monitoring structurally cannot see: a feature quietly leaking the label it's supposed to predict, a model still serving on a table that stopped refreshing, a schema that drifted between training and now. Every finding is written back into DataHub itself, as an incident, a trust score, and a guarding assertion, so the same failure is caught automatically the next time. None of it is a screenshot: there's a live DataHub instance seeded with a real failure and continuously watched, a self-contained reference-manual documentation site, and a 1,053-test suite standing behind all of it.

Software engineering solved "does this change break something" with CI: every change runs tests before it ships. The machine learning data supply chain has no equivalent. Data-quality tools stop at the warehouse edge, model monitoring starts at the model, and the dangerous middle, where a feature is derived from a table and a schema change three hops upstream reaches a live endpoint, belongs to neither. DataHub is the one catalog that holds both graphs at once, column-level lineage across the warehouse and ML metadata for the models, but out of the box nothing joins them: an MLflow ingest produces a model whose training run records no inputs at all, and a dbt ingest produces excellent lineage between tables with no edge reaching a model. A detector that wants to walk from a feature back to its source column has nowhere to start.

1. 01

   Wrote the join nothing else writes

   `janus link` declares which warehouse column each of a model's features came from, which column carries the label, and what the input schema looked like at training time, inferred from what the graph already holds, imported from a Feast repo or a dbt manifest, or typed by hand. Once it exists, every check below is a graph traversal instead of a guess.

2. 02

   Seven detectors sharing one traversal engine

   Target leakage, schema drift, and blast radius ask whether the data is correct; sensitive-source, deprecated-input, and proxy-candidate ask what the organization already decided elsewhere in DataHub that nothing joins back to the model. Each rests on a published result rather than a rule of thumb: target leakage follows Kaufman, Rosset & Perlich's KDD 2011 definition of leakage as illegitimate information about the target; schema drift follows Breck et al.'s MLSys 2019 training/serving contract; blast radius follows Sculley et al.'s NeurIPS 2015 notion of undeclared consumers.

3. 03

   A degraded mode for the common case, nobody has linked anything yet. Rather than staying silent, a scan says what it can still see at table level, quotes the 0.25 precision that answer is measured at, and never lets it outrank or be confused with a real column-level finding. `coverage`, `inventory`, and `finops` fold the same honesty into catalog-wide sweeps a platform lead can act on, including the one report that ever suggests deleting something.

4. 04

   Idempotent by construction

   Every incident, tag, trust score, and guarding assertion is read-before-write and keyed, so a scan rerun on an unchanged graph writes nothing new, measured by reading the graph back after a rerun rather than asserted.

5. 05

   A CI gate with an honest third exit code

   `janus gate` runs the same detectors against a policy and answers 0 shippable, 1 policy violated, or 2 couldn't reach a verdict, because a gate that reports "DataHub was unreachable" as a violation teaches a team to wave through every red build. Bundled as a GitHub Action that posts findings straight to the pull request's own job summary.

6. 06

   Generated the paperwork instead of maintaining it by hand

   A model card, an EU AI Act evidence pack, a per-feature Data Card, and a NIST AI RMF crosswalk are all regenerated from the graph on request, so they're current by construction instead of accurate-until-the-model-next-changes. The evidence pack's first heading says what it isn't, "This is not a compliance certification," followed immediately by what it could not establish.

7. 07

   Built Argos, a desktop watchdog

   A 32×32-pixel dog patrols the corner of the screen, barks with a red collar the moment a finding lands, turns into a translucent ghost when it can't reach DataHub, and walks a finding's blast radius across the desktop, one lineage hop at a time, when you double-click it.

8. 08

   Measured it against ourselves, not just asserted it

   Janus-Bench scores every detector against a live DataHub, compares column-level lineage to a table-level baseline on the identical graph, applies every finding's counterfactual fix and re-checks that it actually clears, and runs mutation testing against the detection code itself.

9. 09

   Hit DataHub's own limits and reported them instead of quietly working around them. Incidents can't attach to an mlModel (GMS answers 500), so model risk lives in structured properties instead, filed upstream as an RFC. Smart assertions are DataHub Cloud-only, so Janus renders its own portable open-assertions YAML. Sixteen reproducible bugs went back in total, plus a datahub-ml-guard skill and an incident-write MCP tool DataHub's own MCP server didn't have.

Janus is a read, reason, write-back loop. A trigger, the CLI, a training script's API, an MCP client, or `watch`, starts a run. Detection queries DataHub's column-level and ML lineage and returns typed findings, with no LLM anywhere in that decision. Orchestration sequences the work and, on the human-approval path, pauses for consent. Reasoning asks a language model only to explain and rank what detection already found, never to decide anything. Write-back commits the result to DataHub idempotently, and Argos turns the same run's event stream into a desktop watchdog.

1. 01

   The join was the entire bet. Column-level lineage and ML metadata already both lived in DataHub; the reason nothing used them together is that nothing wrote the edge between them. Writing that one join is what turned every detector into a graph traversal instead of a heuristic.

2. 02

   Deterministic detection is what made the LLM safe to add, not the other way around. Because the model only explains and ranks what a pure function already decided, a prompt injection sitting in catalog text can at worst affect wording, it's downstream of every finding, never upstream of one.

3. 03

   Idempotency isn't a nice-to-have when something runs forever. `watch` polls or streams DataHub's own change log continuously; without keyed, read-before-write mutations, a `watch` left running over a weekend would file the same incident a thousand times instead of once.

4. 04

   The benchmark had to be willing to fail to be worth trusting. The freshness sweep plants trials on either side of the SLA boundary, not just the obvious 30-hour case, and mutation testing is reported at 76%, not rounded up to a clean number, because a suite that can't fail hasn't been tested.

5. 05

   Argos taught us an honest status display has to be willing to look bad. A cheerful pet that keeps patrolling while it's actually disconnected from DataHub is the kind of lie that gets ambient status indicators switched off, so it turns into a translucent ghost the moment it can't reach the graph instead of pretending everything's fine.

6. 06

   Real platform limits are worth reporting, not just quietly working around. Finding that GMS 500s on an incident attached to an mlModel, or that search hides older model versions Janus still has to write to, could each have stayed a private workaround. Filing them as sixteen reproducible bugs, plus an RFC and a skill for DataHub's own ecosystem, is what turned five weeks of hitting a platform's edges into something the next team building on it doesn't have to rediscover.

7. 07

   Generated documentation earns trust by admitting its limits, not by asserting authority. The EU AI Act evidence pack's first heading is what it isn't, not what it is; a compliance document that implied certainty it didn't have would be worse than no document at all.