01 / Architectural objective
Reducing the latency added by the application.
In an AI assistant, time to first token does not depend solely on the model. Before contacting the provider, the frontend may issue several requests, while the backend loads the conversation, prepares the execution graph and retrieves relevant context. Once the first token arrives, it still needs to reach the client and be displayed.
In Ariath, I worked across this path to bring the user’s wait closer to the provider’s latency. Getting below one second is a useful user-experience reference, but the engineering objective is more precise: minimise the time added by the application, even when model response times vary.
This approach requires examining dependencies between operations. An operation may be necessary without having to block everything that follows. The work involves removing unnecessary round trips, overlapping independent preparation tasks and reducing the cost of operations that remain on the critical path.
02 / Measurement boundaries
From sending a message to the first visible token.
I distinguish the TTFT reported for a provider request from end-to-end latency in the interface. The latter starts when the user sends a message and ends when the first response token is displayed. A frontend optimisation can improve this delay without changing the TTFT recorded by OpenRouter.
- FrontendSend message
- Backend
chat:requestConversation and message persistence - PreparationGraph ∥ memory / RAGOverlapping independent work
- ProviderQueueing, prefill, first token
- FrontendReceive and display
A functional diagram, not a timing scale. Transport and routing add time between components; necessary dependencies remain respected.
The interval from the backend sending a request to receiving its first token still includes transport and, where applicable, intermediate routing. It does not measure model computation alone. Provider TTFT includes factors such as queueing and initial prompt processing, or prefill. OpenRouter documentation
A reliable breakdown requires explicit measurement boundaries. A client-side timer covers the complete experience; server-side measurements isolate preparation and inference requests. Directly subtracting timestamps from different machines requires synchronised clocks. Traces also need to connect stages belonging to the same request.
03 / Simplifying frontend requests
From three sequential requests to one.
The original flow split three operations across separate frontend requests: creating the conversation, storing the message and starting generation. These requests ran sequentially. The client waited for a response before sending the next request, even though the operations served one user intention.
I consolidated this orchestration behind the chat:request socket entry point. The frontend sends one request; the backend handles conversation creation when needed, message persistence and the execution that leads to the provider call.
The improvement does not come from eliminating database writes: they remain necessary. It comes from removing round trips and client-side waits between stages. Orchestration runs closer to the data and the agent’s execution.
On the frontend request-submission path, observed timings went from roughly 700 ms to 200 ms. This describes that part of the flow, not the total time before the response appears.
04 / Backend preparation
Overlapping graph preparation and memory retrieval.
Once a request arrives, the backend prepares the LangGraph execution graph and inference context. RAG retrieval from user memory is among the operations that can delay dispatch to the model.
I worked on graph preparation and on starting memory / RAG preparation alongside graph operations that can proceed independently. Their results are joined when they become necessary to construct the model request.
For two independent tasks, strictly sequential execution adds their durations. Overlapping them can bring the elapsed time closer to the longer of the two, plus any remaining dependent stages. This principle does not justify parallelising operations whose order determines data validity.
Optimisations in this phase brought preparation from roughly 900 ms to 400 ms in development observations. This result covers backend preparation as a whole; it does not attribute the improvement to a single operation or to parallelisation alone.
| Phase | Before | After |
|---|---|---|
| Frontend submission path | ≈ 700 ms | ≈ 200 ms |
| Backend preparation | ≈ 900 ms | ≈ 400 ms |
Interpreting these figures
The timings are approximate and describe different stages. Exact timing boundaries and measurement conditions are not documented here as they would be in a controlled benchmark. Adding them does not establish an end-to-end improvement, particularly without checking phase boundaries and overlap. These figures come from development experience, not calculations based on the OpenRouter export.
05 / Provider and prompt caching
Optimising the inference stage too.
One improvement less directly related to application code was selecting a less congested provider on OpenRouter. Inference infrastructure availability and load matter alongside model choice for observed responsiveness. The fastest provider in one trial is not necessarily the fastest at every hour.
What the observed timings show.
The OpenRouter export provides a view of this variability over several months. The median describes the central case; P90 exposes longer waits, which also matter to the user experience. The table below is ordered by request volume.
26 April–15 November 2026 · The 12 most-used providers represent 81.2% of the 60,095 included requests, across all account uses.
| Provider | Apr.* | May | Jun. | Jul. | Aug. | Sep. | Oct. | Nov.* |
|---|---|---|---|---|---|---|---|---|
| OpenAI | — | — | 0.89† (1 request) | 0.79† (19 requests) | 1.83 (754 requests) | 1.63 (5,780 requests) | 2.18 (7,401 requests) | 2.34 (5,055 requests) |
| GMICloud | — | 2.87 (358 requests) | 2.99 (1,762 requests) | 2.70 (2,905 requests) | 2.67 (1,341 requests) | 5.06 (30 requests) | 3.05† (10 requests) | — |
| StreamLake | — | 2.46† (2 requests) | 1.84 (685 requests) | 1.97 (1,062 requests) | 1.98 (3,862 requests) | 2.66 (281 requests) | 5.46 (192 requests) | — |
| 1.41† (1 request) | 2.03† (11 requests) | 1.78 (1,271 requests) | 0.37 (644 requests) | 0.32 (905 requests) | 0.84 (496 requests) | 2.79 (166 requests) | 2.92 (90 requests) | |
| DeepInfra | — | 0.57 (458 requests) | 0.99 (225 requests) | 1.11 (65 requests) | 0.96 (806 requests) | 0.98 (949 requests) | 1.75 (37 requests) | 0.92† (23 requests) |
| Baidu | 7.01 (206 requests) | 6.31 (709 requests) | 0.63 (344 requests) | 1.00 (312 requests) | 1.03 (777 requests) | 1.94† (9 requests) | 0.69 (66 requests) | 0.87† (5 requests) |
| CoreWeave | — | — | — | 2.15† (13 requests) | 0.57 (73 requests) | 0.61 (270 requests) | 0.80 (1,017 requests) | 0.41 (457 requests) |
| Relace | — | — | — | — | 2.64† (10 requests) | 1.49 (763 requests) | 1.16 (799 requests) | 0.67† (6 requests) |
| Alibaba | 8.21† (1 request) | 0.75 (295 requests) | 0.78 (725 requests) | 1.50 (315 requests) | 1.51 (65 requests) | 1.81† (12 requests) | 0.86† (11 requests) | 0.67† (4 requests) |
| SiliconFlow | 6.16 (317 requests) | 1.26 (299 requests) | 1.41 (604 requests) | 3.00 (95 requests) | 5.62 (43 requests) | 3.66† (14 requests) | 3.98† (14 requests) | — |
| Fireworks | — | 2.48† (5 requests) | 0.82 (60 requests) | 1.64 (35 requests) | 0.87 (738 requests) | 0.85 (463 requests) | 0.29† (2 requests) | — |
| Wafer | — | — | 0.76 (38 requests) | 1.71† (7 requests) | — | 1.75 (106 requests) | 0.48 (932 requests) | 0.31 (135 requests) |
* April: 26–30. November: 1–15. † Fewer than 30 requests in the month. — No included requests. Values are monthly medians in seconds; each heatmap uses its own scale.
| Provider | Requests | TTFT P50 | TTFT P90 | Generation P50 | Generation P90 |
|---|---|---|---|---|---|
| OpenAI | 19,010 | 1.97 s | 5.92 s | 3.01 s | 10.60 s |
| GMICloud | 6,406 | 2.75 s | 6.49 s | 6.52 s | 20.78 s |
| StreamLake | 6,084 | 2.02 s | 6.87 s | 10.28 s | 33.76 s |
| 3,584 | 0.89 s | 3.06 s | 1.70 s | 7.98 s | |
| DeepInfra | 2,563 | 0.92 s | 3.16 s | 7.45 s | 41.59 s |
| Baidu | 2,428 | 2.42 s | 8.69 s | 5.77 s | 15.70 s |
| CoreWeave | 1,830 | 0.60 s | 2.69 s | 6.10 s | 25.12 s |
| Relace | 1,578 | 1.31 s | 3.55 s | 7.02 s | 36.94 s |
| Alibaba | 1,428 | 0.91 s | 4.25 s | 5.69 s | 22.53 s |
| SiliconFlow | 1,386 | 1.81 s | 9.43 s | 8.13 s | 33.08 s |
| Fireworks | 1,303 | 0.87 s | 5.12 s | 5.45 s | 23.80 s |
| Wafer | 1,218 | 0.48 s | 1.07 s | 8.57 s | 27.80 s |
This covers all account uses, unlike the Prisma/Mentat scope of the prompt-caching article. Models, context sizes, cache state and response lengths differ: these results are not a like-for-like provider ranking. TTFT and generation time are OpenRouter fields, excluding additional application stages. Generation duration is neither TTFT nor tokens-per-second throughput; the two durations are not added together here.
Also view monthly generation durations
| Provider | Apr.* | May | Jun. | Jul. | Aug. | Sep. | Oct. | Nov.* |
|---|---|---|---|---|---|---|---|---|
| OpenAI | — | — | 17.52† (1 request) | 1.67† (19 requests) | 3.15 (754 requests) | 2.52 (5,780 requests) | 3.22 (7,401 requests) | 3.41 (5,055 requests) |
| GMICloud | — | 10.59 (358 requests) | 6.19 (1,762 requests) | 6.20 (2,905 requests) | 6.62 (1,341 requests) | 13.90 (30 requests) | 6.50† (10 requests) | — |
| StreamLake | — | 7.43† (2 requests) | 5.89 (685 requests) | 6.16 (1,062 requests) | 12.11 (3,862 requests) | 12.50 (281 requests) | 11.28 (192 requests) | — |
| 4.36† (1 request) | 3.96† (11 requests) | 2.52 (1,271 requests) | 0.59 (644 requests) | 0.50 (905 requests) | 2.69 (496 requests) | 10.13 (166 requests) | 12.57 (90 requests) | |
| DeepInfra | — | 1.71 (458 requests) | 10.16 (225 requests) | 4.80 (65 requests) | 8.90 (806 requests) | 9.16 (949 requests) | 8.48 (37 requests) | 3.59† (23 requests) |
| Baidu | 7.39 (206 requests) | 7.95 (709 requests) | 2.32 (344 requests) | 3.41 (312 requests) | 4.86 (777 requests) | 18.04† (9 requests) | 3.13 (66 requests) | 3.45† (5 requests) |
| CoreWeave | — | — | — | 10.61† (13 requests) | 6.26 (73 requests) | 5.83 (270 requests) | 7.04 (1,017 requests) | 4.38 (457 requests) |
| Relace | — | — | — | — | 11.44† (10 requests) | 6.38 (763 requests) | 8.13 (799 requests) | 4.12† (6 requests) |
| Alibaba | 34.61† (1 request) | 5.33 (295 requests) | 5.99 (725 requests) | 5.68 (315 requests) | 5.58 (65 requests) | 8.90† (12 requests) | 1.69† (11 requests) | 0.83† (4 requests) |
| SiliconFlow | 13.93 (317 requests) | 5.39 (299 requests) | 6.15 (604 requests) | 9.64 (95 requests) | 8.34 (43 requests) | 14.38† (14 requests) | 9.14† (14 requests) | — |
| Fireworks | — | 11.72† (5 requests) | 1.64 (60 requests) | 4.37 (35 requests) | 5.44 (738 requests) | 6.01 (463 requests) | 0.29† (2 requests) | — |
| Wafer | — | — | 5.27 (38 requests) | 1.71† (7 requests) | — | 6.36 (106 requests) | 10.02 (932 requests) | 1.12 (135 requests) |
Source and calculation method
Personal OpenRouter activity export. Streaming only, excluding embeddings, cancellations, errors and nonfinite or nonpositive timings. Both timings are required for each included request. Percentiles use individual requests, with linear interpolation for P90, rather than averages of monthly medians. P50 is the median; P90 describes the duration below which approximately 90% of observations fall. Dates as exported, without timezone conversion. Provider names are preserved as exported.
- OpenAI: median TTFT rises from 1.63 s in September to 2.34 s in November. An overall improvement does not necessarily extend to every provider.
- StreamLake: from 1.98 s in August to 5.46 s in October. Routing choices need reassessment as observed timings evolve.
- Baidu: from 6.31 s in May to 0.63 s in June. The size of the change calls for examining models and requests before attributing it to infrastructure.
These changes justify continuous monitoring, but do not by themselves prove a change in provider load. Nor do they measure the application improvements described above.
Caching as a complementary improvement.
Prompt caching also contributed to reducing latency in this work. Reusing a stable prefix can reduce input processing before generation. The changes to system prompts, tool catalogues and history reconstruction are covered in the dedicated caching study.
Prompt caching: AI agent costs and latency
These improvements act at different levels: reducing application preparation reaches the provider sooner; caching and provider availability affect what follows. No single improvement guarantees constant end-to-end latency.
06 / Diagnosis and validation
Connecting Langfuse, application timings and OpenRouter logs.
I use Langfuse to analyse executions and performance metrics, alongside OpenRouter logs to inspect inference requests. Traces help explain agent steps, RAG retrieval and tool use; frontend measurements remain essential to capture the delay the user actually experiences.
Diagnosis needs to locate the wait before changing the system. Slow server preparation, a congested provider and delayed stream delivery can produce the same visible symptom while requiring different fixes.
- Define boundaries. Distinguish the user action, backend admission, provider dispatch, first-token reception and display.
- Compare similar conditions. Document model, provider, context size and cache state, along with cold starts and resumed conversations.
- Inspect distributions. Track the median and higher latencies, such as p90, to check that an improvement extends beyond the best cases.
- Preserve functional behaviour. Check that gains do not result from skipping necessary work, providing incomplete context or deferring persistence without handling the consequences.
These criteria define the validation protocol for further optimisation. They do not turn the earlier approximate observations into controlled experimental results.
07 / Planned investigations
Further reducing time outside inference.
Several options remain to be evaluated. They concern both the network path and local data processing; they are not presented here as improvements already achieved.
- Direct provider access. Compare direct requests to the same model and provider against requests through OpenRouter to measure intermediate routing overhead. Any gain needs comparable conditions, accounting for routing and recovery functions that would then need to be handled elsewhere.
- Serialisation, deserialisation and decryption. Profile the transformations needed to load and reconstruct context, then target repeated copying, conversion or processing. Optimisation concerns pipeline cost while preserving data-protection guarantees.
- Faster embeddings. Evaluate a faster model, potentially running locally, while also measuring retrieval quality. A model change may require re-embedding the corpus and validating index compatibility.
- Better-sized infrastructure. Test a more powerful machine for local embeddings and CPU work, including decryption. SQL timings need separate analysis: performance also depends on queries, indexes and data access.
08 / Architectural lesson
The latency budget spans the whole request path.
The work in Ariath began before inference by simplifying requests issued by the frontends. It continued through backend preparation, provider selection and context reuse. This progression shows that a more immediate experience depends on several consistent decisions across the architecture.
Saving ten milliseconds remains useful when it removes real work from the critical path, but priorities should follow measurements: eliminate the largest avoidable waits first, then optimise residual processing. The durable reference is the latency added by Ariath, not a universal threshold independent of the provider.
The objective is to spend most of the user’s wait producing the response, with as little delay as possible around the model request.
