How to stream LLM responses in a web app

A 500-token response at 40 tok/s takes 12.5 seconds without streaming. With streaming, the user sees the first words in under a second. Research on user perception shows that responses that start appearing within 1 second feel 'instant' regardless of total generation time.

Streaming is not just a UX feature, it also allows users to stop generation early if the model is going in the wrong direction, reducing wasted API spend.

LLM APIs stream responses using Server-Sent Events (SSE), a simple HTTP protocol where the server keeps the connection open and sends `data: {...}\n\n` events as tokens are generated. The client reads the stream line by line and processes each event.

Each event in the OpenAI format is a JSON object with a `choices[0].delta.content` field containing the new token(s). The final event is `data: [DONE]\n\n`, signalling the end of the stream.

On the server, set SSE headers: `res.setHeader('Content-Type', 'text/event-stream'); res.setHeader('Cache-Control', 'no-cache'); res.setHeader('Connection', 'keep-alive'); res.flushHeaders();`.

Stream from OpenAI: `const stream = await openai.chat.completions.stream({...}); for await (const chunk of stream) { const token = chunk.choices[0]?.delta?.content; if (token) { res.write('data: ' + JSON.stringify({token}) + '\n\n'); } } res.write('data: [DONE]\n\n'); res.end();`

Use the `EventSource` API or the `fetch` API with a readable stream. The `fetch` approach is more flexible: `const response = await fetch('/api/stream', {method: 'POST', body: JSON.stringify({prompt})}); const reader = response.body.getReader(); const decoder = new TextDecoder();`

Read chunks in a loop: `while (true) { const {done, value} = await reader.read(); if (done) break; const text = decoder.decode(value); const lines = text.split('\n'); for (const line of lines) { if (line.startsWith('data: ') && line !== 'data: [DONE]') { const data = JSON.parse(line.slice(6)); setMessage(prev => prev + data.token); } } }`

Implement abort on the client: create an `AbortController`, pass `signal: controller.signal` to `fetch`, and call `controller.abort()` when the user cancels. On the server, listen for the `close` event on the response object and abort the OpenAI stream.

Always wrap stream reading in a try-finally block. If the client disconnects mid-stream, you will get an error, catch it gracefully rather than letting it propagate. Log the error for monitoring but do not crash the process.