Streaming synthesis and the gRPC frontend

lazy (the default) — synthesizes one sentence per call to the stream’s next(), on a single thread, on demand. Lowest latency to the first chunk, no parallelism.

parallel — synthesizes every sentence concurrently across a rayon thread pool, then yields them in original order once all of them are done. Higher throughput for long, multi-sentence text, but nothing is emitted until the whole request finishes.

batched(N) — synthesizes sentences in concurrent groups of up to N (parallel within a group, sequential across groups), yielding each group’s audio before starting the next. A middle ground between lazy and parallel: partial output streams out sooner than parallel’s "wait for everything," with less per-request thread/memory pressure than firing every sentence at once. Because every backend serializes the actual inference call behind a `Mutex, `batched(N)’s benefit is bounded resource pressure and earlier partial output, not raw inference throughput. Delivery is bursty by design: the first chunk of each group waits for the whole group to finish, then the remaining chunks in that group arrive immediately from a buffer, before the next group’s stall.

realtime — requests audio from the model in fixed-size windows (chunk_size, in a backend-specific unit — e.g. mel frames for Piper) as it’s generated, with chunk_padding extra frames of context trimmed from each window’s edges to avoid decoder boundary artifacts. Runs on a background thread and supports cancellation mid-stream. Chunk size grows for later sentences in the same request (up to 4x the base size) to trade a little latency for fewer, larger chunks as the stream goes on.