Part Two: The Show Has a Spine

The temporary lab in Killarney was a hotel room windowsill.

On the left sat the livestream laptop, open and humming. In the middle, a compact ATEM Mini Pro. On the right, a BirdDog PTZ camera, squat and watchful, pointed out the window at nothing in particular. Cables looped across the sill in loose black curves, held together with improvised ties, as if the whole assembly had been sketched into existence faster than it could be properly arranged. Outside the leaded window, Ireland carried on in its usual green indifference. Inside, on a long three-country work trip, I was trying to teach a small machine how to produce a livestream.

This was not the glamorous phase of the project. No reveal, no product shot, no clean bench. Just a cramped hotel room in Killarney, a laptop, a switcher, some testing sources, OBS, a WebSocket connection, and the strange, familiar feeling of building something in the wrong place because it needed to be built now. Some of the most consequential technical work happens under conditions that would not photograph well. It happens in transit, between meetings, after sessions, before dinner, in rooms that were not designed for deep concentration. What matters is not the backdrop so much as the pressure. The work follows the pressure.

By that point, the project had already moved beyond the original question of whether a local agent could operate a formulaic livestream without a remote producer. That much was beginning to seem plausible. OBS could be controlled directly. A switcher could be given constrained commands. Lower thirds could be prepared in a structured way instead of improvised at the last second. The simple grammar of the show was already visible:

Intro = camera
Talk = slides
Q&A = camera
Lower third = first 8–12 seconds of speaker camera intro
Break = slate

The deeper question was what it would mean for the system to understand the show.

That word can get mystical very quickly, so it helps to pin it down. A livestream production system does not need to understand a conference in the human sense. It does not need to appreciate the science or grasp the emotional arc of a keynote. It needs to know something narrower and more operational. What phase are we in? Who is supposed to be speaking? Should the audience be seeing the speaker or the slides? Has the talk actually started, or are we still in introductions? Is this a Q&A cue or just a stray mention of the word "question" in the middle of a presentation? Is the slide feed alive? Is the camera usable? Is the lower third supposed to be on screen right now?

A human producer answers those questions by fusing many kinds of context without really thinking about it. They have a run of show. They know the session title, the speaker, the expected length, the general rhythm of a conference talk. They hear the chair introducing someone. They see the speaker approach the podium. They notice when the slide deck appears. They pick up on phrases like "today I'm going to talk about..." or "I'd be happy to take questions." They compare what is happening to what was supposed to happen, and they make a decision.

That is the real problem. Not switching, exactly. Interpretation.

The first version of the idea is always temptingly simple. You imagine one large model watching the show like a little digital technical director, absorbing the audio and video and deciding what to do next. There is a science-fiction elegance to that picture. It is also probably the wrong architecture.

The more I worked on it, the clearer it became that the system should not be one giant eye. It should be a collection of smaller senses feeding a structured understanding of the show. The system does not need to "watch" the livestream the way a human does. It needs to maintain a rolling state.

That state begins with the obvious production surfaces: current OBS or ATEM state, what source is live, what source is in preview, whether recording is active, whether the slate is available, whether the lower third is visible. But that is only the skeleton. To become useful, the system also needs a run of show and a stream of observations.

The run of show is the planned spine. That matters more than it might seem. A human producer is not doing pure improvisation. They usually know the session title, the chair, the speaker, the talk title, the rough duration of the talk, the likely transition into Q&A, and the approximate moment when a break should occur. The machine should know that too. Not because the plan is always correct, but because it provides a strong prior. It tells the system what is supposed to be happening before the evidence of the room begins to drift.

Then come the observations.

Audio is one of the most valuable signals because it is often the earliest one. If the system continuously captures program audio, breaks it into short chunks, transcribes it, and keeps a rolling buffer of recent speech, it can begin to detect transition phrases and production cues. "I'd like to introduce..." means something different from "first slide." "Thank you" means something different from "I'm happy to take questions." The distinction between these moments is the distinction between camera and slides, between slides and camera again, between confidence and guesswork.

Video matters too, but not in the usual exaggerated way people talk about machine vision. The goal is not to pour a full livestream through a giant model like gasoline through an engine block. The goal is to sample frames and extract compact signals. Is the frame black? Frozen? Slide-like? Camera-like? Text-heavy? Has the slide changed recently? Does the frame look like a title slide, a content slide, a Q&A slide, an acknowledgements slide? Is a person visible on camera? Is the feed usable at all?

This is where the architecture began to sharpen. The system should not depend on a single model to do everything. It should use cheaper, narrower signals first, and only escalate when they are insufficient. A frame can be checked for black or frozen status without any grand intelligence. OCR can read visible text from slides. A small visual model can classify the broad role of a frame. A larger model can remain in reserve for harder cases, ambiguity, or offline evaluation. The output of all of these tools should be normalized into the same thing: a source-state update.

That phrase sounds bureaucratic, but it is quietly important. The production agent does not need raw perceptions piled at its feet. It needs a clean contract. For each source, at each point in time, what do we believe? Is this source a camera, slides, or program output? Is it usable? Does it contain readable text? Has it changed recently? Does it look like the talk is in the intro, the presentation, Q&A, or a break? What visible evidence supports that?

Once you begin to think this way, the system stops looking like an AI stunt and starts looking like a control room made of software. There is no mystical "understanding layer." There are just multiple observers and a decision policy.

That was the point in the project where the shape of the thing became much clearer to me, even if the exact implementation was still evolving. The autonomous producer would not be a chatbot glued to a switcher. It would be an orchestration layer. A persistent ingest system. A run-of-show-aware, stateful, logged, local production engine that could watch, infer, recommend, and eventually act.

And yet the architecture did not fully arrive in the hotel room. It kept arriving in pieces.

At some point that evening, I had to stop working and go out to dinner with colleagues. This is one of the minor absurdities of building technical systems while traveling for work: the mind does not respect the schedule. The body leaves the hotel; the architecture comes with it. Somewhere between dinner and the bar, while the day was trying to become something more social and less schematic, another part of the structure clicked into place. The source understanding should probably be distributed. Not one omniscient model watching everything, but multiple smaller source-level observers producing structured polling data, and a larger, smarter layer above them reasoning over that shared state, much like a technical director.

I am not sure this is the place to fully unpack that hierarchy. It may belong more properly in the next part, when the focus shifts from system shape to model strategy and implementation choices. But it matters as a moment of recognition. The design was getting more specific. The project was shedding one of the most common AI delusions, namely that intelligence becomes more useful the more centrally and opaquely it is concentrated. In practice, the opposite is often true. You get farther with narrow observers, explicit contracts, and a clear policy layer than you do with one giant machine asked to hallucinate a control room.

That, I think, is the real content of this phase of the project.

The first article was about the origin of the problem: the livestream kit, the remote producer, the portability, the latency, the workflow that wanted to move back onto the local machine. The second phase is where the problem becomes architectural. Once you stop asking, "Can the laptop switch the show?" and start asking, "How does the laptop know what show it is in?", the whole project deepens.

Because a scientific livestream is not just a sequence of cuts. It is a sequence of expectations, cues, timings, identities, and transitions. The run of show is the planned version of that sequence. Audio and video are the live version. Production control is the act of reconciling the two.

That is why the system needs a spine.

Not because the show is rigid, but because it drifts. The chair runs long. The speaker opens with thanks. The first slide appears later than expected. A question gets asked early. The transition into Q&A is messy. The break starts three minutes late. The machine has to know not only what should happen, but how to remain useful when the world declines to cooperate.

This is also why the project became more interesting to me as it became more technical. It stopped being just a clever replacement for an awkward remote workflow and started becoming a more general inquiry into operational intelligence. What does it mean to encode a production instinct? Which judgments belong in rules, which in models, and which in the run of show itself? How much can be inferred from cheap signals? Where should certainty come from? How narrow can autonomy be and still be useful?

Those are engineering questions, but they are also aesthetic ones in disguise. A good producer is not only reacting to the room. They are maintaining a form. They are keeping the event legible to the audience. In a small way, that is what this system is trying to do too.

The hotel room in Killarney was not where the idea began. But it was one of the places where it started to acquire a proper internal structure. The yellow case had long since ceased to be the whole story. The project now had a laptop, a switcher, a camera, a command layer, testing sources, OBS WebSocket control, early observers, and a growing sense that the real work was no longer just operating the show. It was modeling it.

That is the threshold I wanted this part to capture. The moment when a livestream system stops being a bag of tools and starts becoming a machine with a point of view about what is happening.

And once a system has that point of view, the next question is unavoidable: what kind of intelligence should be assigned to each layer, and how do you keep the whole arrangement fast, local, explainable, and safe?

That is where the next part begins.