AI National Summit, MLOps Win & Sentinel

The AI National Summit ran two tracks overnight, and I entered both. With Ahmed Ben Rejeb, I won the MLOps competition and its 1000 DT prize. With Moetez Fradi and Ahmed Ben Rejeb, I built Sentinel, a platform that keeps AI agents reliable once they're running inside Jira and Confluence. Both came down to the thing I care about most in this work: not training a model in a notebook, but keeping a real system alive in production. Serve it fast enough. Scale it under load. Keep the pipeline running. Notice when the AI quietly gets something wrong.

Models are easy to train and hard to run, and both tracks pushed on exactly that. Every MLOps challenge handed you something broken. A model with no server in front of it. An inference script that reloaded the model on every single request. A data pipeline that was wired up but never actually connected to its model. A grader then hit your machine from the outside and cared about two things only: were the answers right, and did the service stay up as the traffic climbed. Sentinel took on the quieter version of the same problem. An AI agent can finish every step without an error and still set the wrong priority on a ticket, and normal monitoring can't tell a 60% reliable agent from a 100% reliable one. None of this is about the model. It's about whether the system around it holds.

1. 01

   Serve under a deadline

   I stood up an inference endpoint that matched the model's reference outputs exactly and answered fast enough to clear the target. A wrong answer scores nothing no matter how quick it is, so I got it correct first and only then made it fast.

2. 02

   Optimize and scale out

   The starter service was slow on purpose: it reloaded the model on every request and ran single-threaded. I loaded the model once, added batching, served it across a pool of workers, containerized it, and ran several copies behind one nginx entry point so it kept up as the traffic got heavier.

3. 03

   Repair the pipeline

   I inherited a broken data pipeline meant to score comments with a toxicity model and report the mix per community. I reconnected the scoring step, finished the pipeline, and put a clean service in front of it. The grader ran it through the pipeline tool's own API, so a pipeline that only looked finished counted for nothing.

4. 04

   Record and replay

   Sentinel records every model and tool call an agent makes, without touching the agent's code, then replays a run exactly and offline, with no real emails or ticket comments fired again. So a failure that happened in production can be re-run, narrowed down to the first step that went wrong, and fixed safely. Every run is signed so it can't be quietly altered.

5. 05

   Evaluate what happened

   Three graders check each run: a fast one for correctness that confirms the ticket actually changed, a language-model judge that scores against a rubric, and a safety filter. Reliability is reported as pass-of-k, meaning every one of k tries has to succeed. That surfaces the flakiness a single lucky run would hide.

6. 06

   The agent itself

   A live agent on Atlassian that, when a new incident comes in, finds similar past incidents, pulls up the right runbooks, and drafts a first guess at the root cause with its evidence. It is both the agent we watch and the place the verdicts land.

The diagram is Sentinel. Three parts sit on one shared stream of traces. The Atlassian agent does its work and emits traces as it goes; the recorder captures them and can replay them later; the evaluation engine judges them with its three graders and writes a verdict; and the verdicts land back in Jira and Confluence as tickets and incident reports, closing the loop on the same agent that produced them. The whole thing runs on Kubernetes, autoscales the evaluation engine under load, and stayed 100% available through a chaos test that killed its pods mid-traffic, behind 167 tests. The way it measures reliability follows published research on evaluating agents rather than anything I made up. The MLOps competition was not one system like this. It was three separate broken machines, so it is covered above.

1. 01

   The hard part of machine learning isn't the model, it's everything around it. Every challenge and every part of Sentinel was about serving, scaling, fixing, and checking systems, never about training one.

2. 02

   In the MLOps track, a slow right answer and a fast wrong answer both lose. Treating speed and correctness as one target instead of two is what got us to first place.

3. 03

   An agent can say it did the job and not have done it. That's why Sentinel checks that the world actually changed, not just what the agent reported.

4. 04

   You can't really debug something you can't re-run. Being able to replay a failure offline, without setting off its side effects again, changed how I think about building AI systems.