Model operations · inference terminal
The AI Port watches your models the way a terminal watches aircraft: every inference job gets a gate, a status, and an ETA. Delays and holds have a reason a human can read out loud. It is not another MLOps dashboard with forty tabs — it is one board and a radio someone still owns.
TOWER: "D08 eval-nightly, hold at the gate, quota releases 14:30. Everyone else is on time. Over."
Green boarded on time. Amber is delayed with a stated cause. Red is held — never a silent drop. Every red line has a name attached and a controller who acked it.
The terminal, not the cockpit
Your models already emit metrics. The problem is nobody can glance at them and know if the day is fine. The AI Port turns that into a departure board an ops lead, a controller, and an on-call engineer read the same way — and a radio log so a human owns every hold.
Every job is a flight with a gate, a status, and an ETA. On time, delay with a cause, or hold. Read it from across the room.
Gates map to GPU pools. When a pool is full the board says HOLD and names the quota, not a stack trace. Controllers reassign gates by hand when they need to.
A delay is never just "slow." It is weight-load, cold-start, tool-timeout, or upstream. The cause is on the board next to the ETA.
Holds require an ack. Releases are logged with a name. The AI Port never auto-launches a red flight — it waits for the controller to clear it.
Each route carries its own on-time target. The money route can demand 5s p95 while eval-nightly just has to land by morning.
Every flight leaves a manifest: model, version, prompt hash, cost, and the controller who cleared it. Replay any hour like a flight recorder.
Route types we board today:
Ground control
Above 85% the tower starts holding low-priority flights first.
84% of flights left the gate inside their SLO window today.
The bump at frame 60 is a cold start after a deploy. It cleared in two minutes and the board said so.
58% realtime, 27% batch, 15% training. Batch yields the gate to realtime automatically.
The board runs on wall-clock, not epoch. Humans schedule in "before the 9 a.m. standup," so the terminal does too.
Anatomy of a row
No fourth state, no "degraded-ish yellow-orange." A room can only act on three colors, so the board only has three. Everything else is a number you can drill into, not a status you have to interpret.
Ops flow
voice-tts cold start after a deploy. Cause posted, ETA +9m. No page — inside grace.
eval-nightly wanted 4×A100, pool full. Controller K. Ruiz acked the hold, release at 14:30.
finetune-v7 hit a maintenance window on the train pool. Auto-hold, human-cleared for 15:00.
fraud-score and rag-legal both under SLO. Board all green on realtime.
12 boarding, 2 delayed, 2 held, 0 unacked. Controller signs the hour.
Manifest
| Route | Model | Flights | On-time | p95 | Owner |
|---|---|---|---|---|---|
| chat / support | qwen-reason-32b | 18,204 | 96.2% | 3.9s | Support ops |
| realtime / fraud | fraud-score-v4 | 402,118 | 99.1% | 180ms | Risk desk |
| batch / ocr | invoice-ocr-2 | 61,540 | 98.8% | 1.2s | Finance |
| realtime / voice | voice-tts-lite | 9,860 | 91.4% | 640ms | Product |
| batch / eval | eval-nightly | 1 | held | — | ML platform |
| train / finetune | finetune-v7 | 3 | 2/3 | — | ML platform |
The fraud route ran 402,118 flights at 99.1% on-time and 180ms p95 — and the one time it slipped, the board said "tool timeout on the device-graph call," a controller acked it in 40 seconds, and the money never stopped moving. That is the whole product: not fewer incidents, but no silent ones.
Peak dips because the tower holds batch to protect realtime. On purpose.
Two peaks: the 9 a.m. rush and the nightly batch push. The board never went dark.
Nothing sits red without a name next to it.
Board vs. dashboard
| Question at 9 a.m. standup | 40-tab MLOps suite | The AI Port |
|---|---|---|
| Is today okay? | Open 6 dashboards and guess | Glance at the board |
| Why is voice slow? | Grep logs, correlate graphs | DELAY: cold-start, on the row |
| What is held and who owns it? | No concept of a hold | Red row + named controller |
| Did we auto-retry into a storm? | Usually, then it pages you | Never — holds wait for a human |
| What ran last Tuesday at 3? | If retention allows | Black-box replay of the hour |
| Can ops read it without ML? | No | Yes — it is a departure board |
We are not trying to replace your training stack or your feature store. We replace the six tabs nobody agrees on at standup with one board the whole room reads the same way.
Proof & boundaries
“Our old dashboard had forty panels and nobody could answer ‘is today okay?’ The board answers it from the doorway. Standup went from twenty minutes to two.”
— Head of ML platform, fintech (Terminal plan)“The part that sold my SRE was that it never auto-launches a held job. A red flight waits for a human. That’s the opposite of every autoscaler that paged us at 3 a.m.”
— On-call lead, e-commerce (Hub, air-gapped)Pricing
No. It is the arrivals/departures board on top of whatever you already run — vLLM, Triton, Ray, Kubernetes. We watch flights and own the radio. We do not train models or store features.
Never. A held flight stays red until a named controller acks and clears it. Automation can propose a release; only a human executes one. That rule does not have a toggle.
Metrics in from Prometheus or an OpenAI-compatible proxy; pages out to PagerDuty, Slack, or a webhook. Gates map to your existing GPU pools. Most Terminals board their first flight the same week.
Yes — that is the Hub plan. No outbound calls, SSO against your IdP, and the board can drive a NOC wall display. Everything, including the black box, stays inside your network.
A named controller per shift. The board pages that person, not a Slack channel that everyone ignores.
Ground control to…
Tell us how many models you fly and which route is the money route. We will show you your board.