Streaming responses
Pros:
- Lower perceived latency: Users see the beginning of the answer immediately, which can make interactions feel faster.
- Long responses feel more manageable: Text appears progressively, making it easier to follow or stop early.
Cons:
- More complex implementation: Clients must handle partial output, reconnection, and interrupted generations.
- Harder to validate and process: The full response is not available at once, complicating structured-output parsing, error handling, and some workflows.
Non-streaming responses
Pros:
- Simpler integration: The entire response is returned in one payload, making parsing, validation, and downstream processing easier.
- More predictable completion: The request finishes atomically, which is useful for batch jobs and applications that require the complete result before proceeding.
Cons:
- Higher perceived latency: Users may wait without seeing any output until generation is complete.
- Poor experience for long answers: A long response can appear frozen and provides no incremental feedback.