LLM Chatbot with Streamed Replies

Context

Progressive rendering was a requirement; the transport was left open. Replies stream in as they are written, follow-up questions keep their context, and conversations are listed, renamed, switched, and deleted from a sidebar.

Challenge

Once the first byte of a streamed reply has gone out, the HTTP status code is already spent. A model that fails halfway through has no status left to fail with, and a client that trusts the 200 will render a truncated answer as a complete one.

Non-negotiables

• A reply interrupted mid-stream must be distinguishable from a finished one
• Proxies must not close a connection while the model is still thinking
• Billed tokens that never appear in the reply still have to be accounted for

Calls I made

1. 01

   Server-Sent Events, not WebSockets

   The traffic is one-directional, so a bidirectional protocol would have bought nothing and cost a second transport, Django Channels, a Redis channel layer, and separate auth and scaling stories. Plain chunked text was the other option and was rejected because it cannot carry structured terminal information — message id, usage, finish reason — or tell a completed reply apart from a dropped connection.

2. 02

   The terminal frame is the authority

   Events are typed as meta, delta, done, and error. Because errors after the first byte arrive on an already-successful response, clients are required to treat the terminal frame as the source of truth and ignore the status code.

3. 03

   Heartbeats through the thinking gap

   Idle comment frames keep intermediaries from closing a connection while the model reasons. With time-to-first-token around three seconds and thinking budgets above that, the silence is long enough to look like a dead connection.

4. 04

   Reasoning configured by level, not budget

   Setting a thinking budget of zero was rejected outright by both Gemini flash models with an HTTP 400, despite being valid in the SDK's type system — so nothing catches it before a live call. Reasoning is configured by level instead, and an integration test asserts the model accepts the generation config.

Trade-offs accepted

• Streaming requires ASGI. The Django development server and any buffering proxy will silently break progressive rendering, which makes the deployment topology part of the feature rather than an operational detail.
• Reasoning cannot be switched off on these models. The lowest setting still consumed around 94 thinking tokens on a trivial prompt — billed, invisible in the reply, and stored separately from output tokens because folding them together would misreport both.