Part Three: The Ghost Switcher

I was sitting at the conference registration desk at the Park Plaza London Riverbank, just outside the ballroom where the main session was in full swing, when the venue network disappeared.

Not slowed down. Not got a little flaky. Disappeared.

A few seconds later, the familiar little horror show began. The remote producer was kicked off the livestream laptop. The stream dropped. The remote audience, quite reasonably, started reporting what they could see from their side of the void.

"Black screen."

"Lost audio."

"Where is the video?"

There is a special kind of adrenaline reserved for livestream failure. It is not loud, exactly. It is colder than that. Your body knows before your mind finishes assembling the sentence. Something that was supposed to be continuous has broken. The room is still happening. The speaker is still speaking. The science is still moving forward. But somewhere between the ballroom and the online audience, the event has vanished.

To the venue AV team's credit, they got us back online relatively quickly, especially by the standards of hotel network failures, a genre of incident that seems governed by weather, folklore, and whatever is happening in the walls. But "relatively quickly" is still an eternity when audience comments are piling up in the chat. A livestream is not like a file upload. It does not patiently resume with dignity. It either exists in the moment or it does not.

That London failure clarified something for me.

The remote producer problem was not only a staffing problem. It was not only a latency problem. It was not only a bandwidth problem. It was a dependency problem.

The production system was sitting in the room. The laptop was in the room. The ATEM was in the room. The camera was in the room. The slides were in the room. The audio was in the room. But the operational intelligence needed to run the show was somewhere else (Toronto, specifically), reaching back through a network connection that could disappear without warning.

That is a bad shape for a live system.

The first two parts of this series traced how we got here. The original kits were built to make livestreaming portable after scientific conferences returned in person but still needed to serve remote audiences. Later, in a hotel room in Killarney, the idea began to take on a more formal architecture: a local system that could observe the show through audio, video, device state, and a run of show, instead of relying on a remote human clicking through a delayed desktop.

This final part is about what happens next.

Because once the system can observe the show, the question becomes unavoidable: what is it allowed to do?

The naive answer is: let it run everything.

That is also the dangerous answer.

Live production is full of actions that are simple but consequential. Cut to camera. Cut to slides. Show a lower third. Hide a lower third. Start recording. Stop recording. Show slate. End session. Upload file. None of these actions are conceptually complex. Many are just API calls. But their timing and context matter. A bad cut is not catastrophic, but it is visible. A missed recording is worse. A stream interrupted at the wrong moment can erase the value of the whole system.

So the architecture has to respect a distinction that AI products often blur: recommendation is not execution.

A model can observe. A model can classify. A model can suggest. But the switching decision needs to pass through a production policy that understands the rules of the show, the current state of the system, the confidence of the evidence, and the cost of being wrong.

That is why the system needs a ghost switcher before it needs an autopilot.

By ghost switcher, I mean a shadow technical director that watches the show and says what it would do, without doing it. It observes the current program state, the camera feed, the slide feed, the transcript, the run of show, the recent history of cuts, and the health of each source. Then it produces a recommendation:

Q&A cue detected.
Slides show acknowledgements.
Camera feed usable.
Recommendation: switch to camera.
Confidence: 0.88.

In shadow mode, nothing happens. The human keeps operating the show. The system logs the recommendation. Later, those recommendations can be compared against what the human did, what the show actually needed, and what the production standard says should have happened.

That is the quiet machinery of trust.

Not vibes. Not a demo. Not "the AI seemed pretty good." Logs.

Every recommendation should produce a record. What phase did the system think it was in? What did the transcript say? What did the slide observer see? Was the camera usable? What did the run of show expect? What was live before the recommendation? What action did the human take afterward? Was the recommendation accepted, rejected, ignored, or wrong?

That record is how autonomy earns its keep.

It also prevents a common mistake: training the system to imitate mediocre production. I do not want an autonomous producer that learns every bad habit of a rushed remote operator. I want one that is measured against a better standard:

Intro = camera
Talk = slides
Q&A = camera
Lower third = first 8-12 seconds of speaker camera intro
Break = slate

The point is not to reproduce whatever a person happened to do. The point is to encode what should happen, then use human operation and replay evaluation to test whether the machine can recognize those moments reliably.

This is where replay becomes important.

A live session is stressful because everything is happening once. Replay gives the system a second life. Feed it a recorded session, sample the frames, transcribe the audio, align the run of show, and ask the ghost switcher what it would have recommended at each moment. Would it have stayed on slides during the talk? Would it have caught the transition into Q&A? Would it have avoided cutting to a frozen source? Would it have held when the evidence was ambiguous?

Replay turns live production into a test bench.

That matters because the path to autonomy should not be a single leap. It should be a ladder.

The first rung is command relay. A staff member watches the multiview and types constrained commands: cam, slides, l3, slate. The local system validates and executes those commands through OBS or ATEM APIs. No remote desktop. No GUI clicking. No freeform computer control. Just a narrow vocabulary of production actions.

The second rung is shadow mode. The system watches and recommends, but does not execute. It becomes a ghost in the booth, building a record of what it would have done.

The third rung is human-confirmed switching. The system recommends an action and waits. The human confirms with a keystroke. The command executes through the same constrained control layer as any typed command. The difference is that the cue came from the machine.

The fourth rung is narrow autopilot. Not general autonomy. Not "AI, run the conference." Just the moments that are formulaic enough, well-observed enough, and low-risk enough to automate: camera for intros, slides for presentations, camera for Q&A, lower thirds during the first seconds of a speaker introduction, slate during breaks, hold when confidence is low.

The fifth rung, eventually, is mostly autonomous operation with human override. One person monitoring multiple rooms. The system handling routine switching. Alerts only when something looks wrong.

But that future only works if the local system can survive what examples like London taught me to fear.

The point of Autonomous Producer is not merely to make switching smarter. It is to make the production system less dependent on fragile remote control. If the remote producer is disconnected, the system in the room should not become brainless. If the venue network drops, the local kit should still know what show it is in. It should still know the current block, the expected source, the speaker, the talk title, the camera state, the slide state, the transcript buffer, and the safest next action. When the network comes back, the production log should explain what happened.

That is why local inference matters.

Cloud systems are useful, but live event infrastructure cannot assume perfect connectivity. The more venue networks I encounter, the more I think "assume the network works" is less an architecture than a superstition with a budget line. For this kind of system, the critical loop should run on the livestream laptop or on hardware physically attached to the kit. The machine in the room should be able to observe, decide, and act without asking a distant server whether the slide feed is black.

Local does not mean isolated. A remote dashboard can still exist. A human can still monitor. Logs can still sync. Recordings can still upload. Larger models can still review sessions later. But the live-critical path should be as local as possible.

The same principle applies to control. The system should not operate OBS or ATEM Software Control by moving a mouse around the screen. That is just remote desktop with extra mythology. Live-critical actions should be explicit, typed, and narrow:

{
  "action": "switch_program",
  "target": "camera"
}

The executor translates that into the appropriate OBS or ATEM command. The action is logged. The state before and after is recorded. If the command fails, the failure is visible. If the source is black, the policy can suppress the cut. If confidence is low, the policy can hold. If the human overrides, the override becomes part of the record.

This is not as glamorous as a model "controlling the computer." Good. Glamour is not the job. Reliability is the job.

The distributed model hierarchy fits into that philosophy. A small visual model does not need to direct the show. It only needs to say, "This frame looks like a content slide," or "This frame looks like a camera shot," or "This source may be unusable." OCR reads text. Audio transcription catches transition language. Cheap heuristics detect black frames, frozen frames, motion, text density, and slide changes. The run of show supplies the plan. The director policy decides what the evidence means.

The larger intelligence is not necessarily a larger model. It is the structure of the system.

That was the insight that became clearest after the Killarney work and the London failure sat next to each other in my head. In Killarney, I was building the senses. In London, I was reminded why the body needed to be local.

A remote producer can be excellent. A remote producer can also be disconnected by a hotel network, trapped behind latency, or forced to operate a show through a delayed representation of the room. The autonomous producer does not replace human expertise by pretending the human never mattered. It replaces a fragile operational shape with a more resilient one.

The human expertise moves into the system design:

• the show grammar
• the run-of-show schema
• the source-state contract
• the command vocabulary
• the timing rules
• the confidence thresholds
• the override behavior
• the production logs
• the replay tests
• the refusal to cut when the evidence is bad

That last one may be the most important. A good autonomous producer should know when not to switch. It should hold through ambiguity. It should alert when the world does not match the plan. It should fail quiet before it fails theatrical.

That is why I think the project is less about artificial intelligence than operational intelligence. The intelligence is not a personality. It is not a chat interface. It is not a single model making heroic decisions from pixels. It is a local system that understands enough of the event structure to keep the audience connected to what is happening in the room.

That brings us back to the registration desk in London.

When the network dropped, the failure was not mysterious. It was infrastructural. A remote operator lost access to a local show. The audience saw the result immediately. For a few minutes, the conference split in two: the room where the session continued, and the online space where the event had disappeared.

Autonomous Producer is an attempt to close that split.

Not completely. Not magically. Not all at once. But through a set of practical moves: move the control loop local, make the show observable, give the system a run of show, constrain its actions, log its decisions, evaluate it in shadow mode, require confirmation before trust, and only then let it automate the narrow parts of production that were never as creative as they looked.

A yellow hard case made the livestream portable.

A hotel-room prototype gave the system a spine.

The ghost switcher is how it learns to act.