01 / Objectives and constraints
Optimising inference cost and response latency.
Ariath is an AI assistant for iOS and the web that retains conversation context and uses tools. Its development involves two related constraints: controlling the cost of model requests and reducing the delay before a response begins. French language quality remains a requirement when selecting a model.
My initial approach focused mainly on input and output token prices. I chose an inexpensive Chinese model while rapidly developing features, expanding tools and enriching the context supplied to the agent. Request stability was not yet an explicit architectural objective.
The limitations I encountered with the models and providers I tested concerned both French language quality and variable latency. ChatGPT’s responsiveness reinforced the goal of a more immediate response, without providing a comparison under equivalent conditions. The work then shifted towards context construction and reuse between requests.
02 / Stack and observability
LangGraph for execution, Langfuse for diagnosis.
Ariath uses LangGraph to orchestrate the agent, its model requests and its tools. Execution state is persisted in PostgreSQL through checkpoints so a conversation can resume from its recorded state. These checkpoints belong to the application; prefix caching is handled by the inference provider. Preserving the former therefore does not guarantee reuse of the latter. LangGraph persistence
Langfuse is used to inspect executions and understand what is actually sent to the model. In this work, its role goes beyond aggregate metrics: I use it to investigate cache breaks, context construction and tool use. Langfuse observability
- Investigating prefix changes. Compare the context of successive requests to identify rewritten information, a moved block or a change introduced when conversation history is restored.
- Checking the transmitted context. Inspect which information enters the prompt, where it is inserted and how it evolves between turns: history, technical footers and information retrieved from memory.
- Inspecting tools. Check calls, arguments and results, as well as the size of the catalogue exposed to the model. Tool definitions contribute to input context; their changes need to be examined alongside those of the messages.
- Connecting context to execution. Examine durations, errors and available token counters to distinguish a prompt construction issue from time spent on a model request, RAG retrieval or tool execution.
A drop in the cache rate therefore triggers request inspection rather than immediate attribution to the model. A prefix difference provides a verifiable lead; if none is found, cache conditions and the provider used also need investigation. The charts in this article are calculated from the OpenRouter export, separately from Langfuse diagnostic traces.
03 / Inference performance
Separating TTFT from total response time.
TTFT measures the delay before the first token. For an individual request, it depends on factors including network transit, provider queueing and initial prompt processing, known as prefill. Generating subsequent tokens is a separate phase. OpenRouter latency documentation
Simplified processing sequence
Context preparation → request transmission → queueing and prefill → first token → subsequent generation.
Perceived latency in the application also includes its own processing. An agent turn can involve several model requests and tool executions.
Prompt caching reuses a prefix during input processing. It can reduce prefill work and therefore contribute to a lower TTFT. On its own, it does not guarantee faster output generation or stable end-to-end latency.
This distinction matters for optimisation: improving caching removes neither the wait at a busy provider nor the time spent executing a tool. The same cache rate can therefore coexist with very different response times.
04 / Cost model
Accounting for cache reads and writes.
Prefix caching reuses the processing of an initial portion of a prompt previously sent to the model. It does not return a stored answer. Reuse requires an identical prefix, subject to provider conditions such as minimum length and retention time. Anthropic documentation
An economic comparison needs to distinguish four components: uncached inputs, cache reads, cache writes and output tokens. Read and write rates depend on the model and provider; a cache write can cost more than an ordinary input token. OpenRouter caching documentation
Inference cost per request
C = Ninput × Pinput + Nread × Pread + Nwrite × Pwrite + Noutput × Poutput
N is a token count and P its unit price. Input categories must be disjoint under the provider’s billing rules: a cache-write token must not also be counted as an ordinary input token. Any ancillary charges are outside this breakdown.
A low input price is therefore insufficient to determine conversation cost. Actual reuse, write frequency and the number of requests needed to complete a task also matter. The cache rate remains a technical metric, distinct from the percentage saved on the bill.
05 / Request construction
Preserving a stable prefix between requests.
The first change to Ariath was to stabilise the system prompt and tool definitions. Reusable elements need a consistent representation across requests. A change near the start of the context can prevent reuse of the prefix that follows it.
This constraint applies to the content actually presented to the model: instructions, tool definitions and history. Two requests that appear semantically equivalent can still differ enough to lose reuse opportunities. Stability needs to be checked after the application’s transformations, not only in the stored data model.
I introduced a technical footer at the end of user messages to enrich the new turn’s context. New information is appended to the conversation without changing technical blocks already attached to previous turns.
Solid blue: retained content. Dashed border: appended content. A logical diagram, not token proportions; exact block order depends on the API and cache eligibility on the provider.
The footer is a context construction convention. Its benefit depends on applying the same rule to subsequent requests: do not regenerate previous footers using the latest information.
06 / Persistence and context freshness
Preserving history while keeping knowledge current.
I also revised conversation storage and decoding to preserve context when a conversation resumes. Persistence needs to reconstruct previously transmitted messages without implicitly rewriting their technical information.
During LLM-assisted development, coding assistants regularly proposed updating context inside the history. This prioritised information freshness but changed the replayed prefix. I made the constraint explicit: refreshing context must not silently mutate previous exchanges.
Ariath handles freshness through complementary mechanisms:
- Prioritising recent information. The prompt asks the model to give more weight to fresh information when the context contains several states of the same information.
- Searching user memory. RAG searches run when relevant to retrieve context useful to the request.
- Context compaction. Once the context reaches a certain volume, it is compacted to limit its size.
Compaction introduces a trade-off: it changes the context presented to the model and can therefore require part of the cache to be rebuilt. The objective is to keep a stable prefix between these transitions while controlling total volume and information relevance. Maximising the cache rate is not sufficient reason to accumulate context indefinitely.
07 / Measurements and results
73% caching in October, with daily peaks above 85%.
Early in this work, my recollection was that roughly 25% of input tokens were served from cache under favourable conditions. After the changes, I regularly observed rates around 70–75%. Analysing an OpenRouter export provides more precise evidence.
Within the selected scope, the token-weighted rate rose from 20.6% in July to 73.0% in October. It was 67.8% for 1–15 November. The full series shows monthly variations and the two days above 85%.
1 October 2026
85.4%
629 included requests
36,745,694 input tokens
27 October 2026
85.7%
548 included requests
10,707,605 input tokens
Daily rate weighted by tokens · 26 April to 15 November 2026. Ringed points mark days at or above 85%; gaps indicate days with no included requests. These daily peaks are not monthly averages. They use the same filters and the same ratio of token sums.
View daily values
| Period in 2026 | Weighted rate | Included requests |
|---|---|---|
| 2026-04-29 | 0.0 % | 2 |
| 2026-04-30 | 0.5 % | 2 |
| 2026-05-05 | 0.0 % | 1 |
| 2026-05-11 | 0.0 % | 9 |
| 2026-05-12 | 7.0 % | 181 |
| 2026-05-13 | 17.1 % | 256 |
| 2026-05-14 | 35.8 % | 10 |
| 2026-05-15 | 28.8 % | 70 |
| 2026-05-16 | 38.5 % | 93 |
| 2026-05-18 | 11.8 % | 179 |
| 2026-05-19 | 18.3 % | 294 |
| 2026-05-20 | 18.3 % | 144 |
| 2026-05-21 | 24.1 % | 162 |
| 2026-05-22 | 20.1 % | 39 |
| 2026-05-23 | 6.4 % | 19 |
| 2026-05-24 | 9.1 % | 6 |
| 2026-05-25 | 5.1 % | 94 |
| 2026-05-26 | 36.2 % | 258 |
| 2026-05-27 | 58.8 % | 158 |
| 2026-05-28 | 62.3 % | 378 |
| 2026-05-29 | 64.1 % | 283 |
| 2026-05-30 | 44.1 % | 47 |
| 2026-05-31 | 48.2 % | 129 |
| 2026-06-01 | 46.9 % | 1,341 |
| 2026-06-02 | 77.8 % | 347 |
| 2026-06-03 | 56.5 % | 275 |
| 2026-06-04 | 52.6 % | 113 |
| 2026-06-05 | 28.2 % | 559 |
| 2026-06-06 | 23.3 % | 78 |
| 2026-06-07 | 32.5 % | 225 |
| 2026-06-08 | 25.4 % | 339 |
| 2026-06-09 | 42.9 % | 243 |
| 2026-06-10 | 24.6 % | 118 |
| 2026-06-11 | 40.2 % | 83 |
| 2026-06-12 | 20.8 % | 198 |
| 2026-06-13 | 25.8 % | 140 |
| 2026-06-14 | 16.0 % | 288 |
| 2026-06-15 | 15.6 % | 284 |
| 2026-06-16 | 35.9 % | 368 |
| 2026-06-17 | 13.4 % | 93 |
| 2026-06-18 | 19.7 % | 351 |
| 2026-06-19 | 14.8 % | 342 |
| 2026-06-20 | 36.0 % | 211 |
| 2026-06-21 | 37.6 % | 139 |
| 2026-06-22 | 23.6 % | 113 |
| 2026-06-23 | 23.2 % | 75 |
| 2026-06-24 | 23.8 % | 113 |
| 2026-06-25 | 21.8 % | 181 |
| 2026-06-26 | 27.6 % | 329 |
| 2026-06-27 | 21.2 % | 349 |
| 2026-06-28 | 28.7 % | 398 |
| 2026-06-29 | 23.7 % | 307 |
| 2026-06-30 | 22.9 % | 282 |
| 2026-07-01 | 34.6 % | 156 |
| 2026-07-02 | 27.2 % | 662 |
| 2026-07-03 | 16.2 % | 532 |
| 2026-07-04 | 37.5 % | 229 |
| 2026-07-05 | 18.6 % | 97 |
| 2026-07-06 | 17.8 % | 465 |
| 2026-07-07 | 32.5 % | 213 |
| 2026-07-08 | 16.9 % | 322 |
| 2026-07-09 | 13.6 % | 310 |
| 2026-07-10 | 23.1 % | 296 |
| 2026-07-11 | 12.3 % | 172 |
| 2026-07-12 | 16.6 % | 195 |
| 2026-07-13 | 43.6 % | 196 |
| 2026-07-14 | 5.6 % | 126 |
| 2026-07-15 | 16.4 % | 67 |
| 2026-07-16 | 7.2 % | 211 |
| 2026-07-17 | 6.3 % | 147 |
| 2026-07-18 | 3.9 % | 127 |
| 2026-07-19 | 15.2 % | 93 |
| 2026-07-20 | 15.0 % | 207 |
| 2026-07-21 | 12.9 % | 170 |
| 2026-07-22 | 17.5 % | 130 |
| 2026-07-23 | 11.5 % | 174 |
| 2026-07-24 | 23.7 % | 484 |
| 2026-07-25 | 10.9 % | 142 |
| 2026-07-26 | 7.1 % | 71 |
| 2026-07-27 | 23.7 % | 112 |
| 2026-07-28 | 14.9 % | 161 |
| 2026-07-29 | 15.6 % | 113 |
| 2026-07-30 | 39.6 % | 177 |
| 2026-07-31 | 31.3 % | 241 |
| 2026-08-01 | 32.1 % | 333 |
| 2026-08-02 | 47.5 % | 132 |
| 2026-08-03 | 17.4 % | 203 |
| 2026-08-04 | 38.0 % | 451 |
| 2026-08-05 | 19.3 % | 275 |
| 2026-08-06 | 29.5 % | 322 |
| 2026-08-07 | 36.4 % | 300 |
| 2026-08-08 | 43.5 % | 127 |
| 2026-08-09 | 24.4 % | 106 |
| 2026-08-10 | 29.0 % | 149 |
| 2026-08-11 | 21.7 % | 212 |
| 2026-08-12 | 41.7 % | 619 |
| 2026-08-13 | 41.6 % | 687 |
| 2026-08-14 | 49.3 % | 776 |
| 2026-08-15 | 53.1 % | 736 |
| 2026-08-16 | 49.4 % | 420 |
| 2026-08-17 | 49.6 % | 477 |
| 2026-08-18 | 42.7 % | 579 |
| 2026-08-19 | 44.0 % | 469 |
| 2026-08-20 | 31.9 % | 105 |
| 2026-08-21 | 48.4 % | 136 |
| 2026-08-22 | 41.2 % | 137 |
| 2026-08-23 | 47.9 % | 159 |
| 2026-08-24 | 48.5 % | 332 |
| 2026-08-25 | 48.6 % | 345 |
| 2026-08-26 | 50.1 % | 236 |
| 2026-08-27 | 39.5 % | 270 |
| 2026-08-28 | 53.2 % | 254 |
| 2026-08-29 | 42.3 % | 90 |
| 2026-08-30 | 38.3 % | 219 |
| 2026-08-31 | 38.5 % | 237 |
| 2026-09-01 | 29.9 % | 350 |
| 2026-09-02 | 38.2 % | 757 |
| 2026-09-03 | 45.1 % | 214 |
| 2026-09-04 | 46.0 % | 389 |
| 2026-09-05 | 40.1 % | 199 |
| 2026-09-06 | 53.7 % | 342 |
| 2026-09-07 | 44.8 % | 206 |
| 2026-09-08 | 47.8 % | 281 |
| 2026-09-09 | 53.7 % | 364 |
| 2026-09-10 | 59.5 % | 392 |
| 2026-09-11 | 81.9 % | 326 |
| 2026-09-12 | 64.3 % | 134 |
| 2026-09-13 | 67.2 % | 215 |
| 2026-09-14 | 61.5 % | 259 |
| 2026-09-15 | 65.7 % | 241 |
| 2026-09-16 | 63.0 % | 111 |
| 2026-09-17 | 64.6 % | 204 |
| 2026-09-18 | 70.5 % | 390 |
| 2026-09-19 | 54.7 % | 119 |
| 2026-09-20 | 56.6 % | 134 |
| 2026-09-21 | 62.8 % | 544 |
| 2026-09-22 | 63.8 % | 626 |
| 2026-09-23 | 68.4 % | 520 |
| 2026-09-24 | 69.3 % | 313 |
| 2026-09-25 | 60.6 % | 313 |
| 2026-09-26 | 69.3 % | 567 |
| 2026-09-27 | 53.0 % | 158 |
| 2026-09-28 | 55.8 % | 171 |
| 2026-09-29 | 69.4 % | 602 |
| 2026-09-30 | 72.5 % | 642 |
| 2026-10-01 | 85.4 % | 629 |
| 2026-10-02 | 74.8 % | 639 |
| 2026-10-03 | 68.1 % | 494 |
| 2026-10-04 | 63.4 % | 305 |
| 2026-10-05 | 68.2 % | 414 |
| 2026-10-06 | 70.3 % | 493 |
| 2026-10-07 | 73.6 % | 429 |
| 2026-10-08 | 51.1 % | 494 |
| 2026-10-09 | 77.0 % | 436 |
| 2026-10-10 | 74.8 % | 242 |
| 2026-10-11 | 62.0 % | 346 |
| 2026-10-12 | 40.8 % | 2,625 |
| 2026-10-13 | 62.5 % | 496 |
| 2026-10-14 | 75.5 % | 290 |
| 2026-10-15 | 85.0 % | 359 |
| 2026-10-16 | 81.0 % | 558 |
| 2026-10-17 | 67.9 % | 193 |
| 2026-10-18 | 67.9 % | 191 |
| 2026-10-19 | 80.1 % | 405 |
| 2026-10-20 | 70.0 % | 276 |
| 2026-10-21 | 77.4 % | 130 |
| 2026-10-22 | 77.2 % | 171 |
| 2026-10-23 | 80.1 % | 360 |
| 2026-10-24 | 61.9 % | 94 |
| 2026-10-25 | 65.6 % | 88 |
| 2026-10-26 | 74.5 % | 257 |
| 2026-10-27 | 85.7 % | 548 |
| 2026-10-28 | 83.9 % | 651 |
| 2026-10-29 | 74.8 % | 245 |
| 2026-10-30 | 75.6 % | 432 |
| 2026-10-31 | 58.3 % | 237 |
| 2026-11-01 | 69.7 % | 844 |
| 2026-11-02 | 69.8 % | 667 |
| 2026-11-03 | 65.7 % | 150 |
| 2026-11-04 | 63.0 % | 370 |
| 2026-11-05 | 61.0 % | 542 |
| 2026-11-06 | 63.6 % | 600 |
| 2026-11-07 | 62.4 % | 313 |
| 2026-11-08 | 51.2 % | 204 |
| 2026-11-09 | 72.6 % | 852 |
| 2026-11-10 | 72.7 % | 453 |
| 2026-11-11 | 65.1 % | 175 |
| 2026-11-12 | 58.3 % | 582 |
| 2026-11-13 | 70.2 % | 448 |
| 2026-11-14 | 82.7 % | 272 |
| 2026-11-15 | 77.1 % | 334 |
| Period in 2026 | Weighted rate | Included requests | Median TTFT (s) |
|---|---|---|---|
| 26–30 April | 0.2 % | 4 | 6.52 |
| May | 38.3 % | 2,810 | 1.20 |
| June | 37.4 % | 8,282 | 1.58 |
| July | 20.6 % | 6,798 | 2.19 |
| August | 42.1 % | 9,893 | 1.77 |
| September | 60.1 % | 10,083 | 1.55 |
| October | 73.0 % | 13,527 | 1.66 |
| 1–15 November | 67.8 % | 6,806 | 1.90 |
Method and scope
OpenRouter export dated 15 November 2026: 82,970 rows, with 58,203 requests retained from Prisma and Mentat keys. OpenClaw, OpenRouter Chatroom, embeddings, cancelled or failed requests, nonpositive inputs and missing cache counters are excluded. Periods follow the exported dates, which do not specify a timezone.
Rate = sum of tokens read from cache ÷ sum of input tokens. Requests with zero cached tokens remain included; a missing counter is not treated as zero.
Improving the cache rate does not automatically produce a continuous fall in TTFT. In this export, median TTFT per request was 2.19 s in July, 1.55 s in September, 1.66 s in October and 1.90 s in the first half of November. Cache rate and latency describe different dimensions of the system.
These are observational results. Models, providers, context sizes and workloads change during the period; the export cannot isolate the causal effect of each modification. Reported TTFT concerns requests recorded by OpenRouter, not full latency measured in Ariath’s interface. No percentage reduction in cost or response time is inferred from the cache rate.
08 / Architectural lessons
Measuring reuse, cost and latency separately.
This optimisation covered the entire context construction chain: system prompts, tool definitions, message enrichment, storage and conversation replay. Caching became an explicit design constraint, with deliberate transitions when context needs compaction.
To attribute gains precisely, the validation step is to compare representative requests with a fixed model, provider and configuration, first with a cold cache and then with reuse. Relevant measures include tokens read and written, billed cost, TTFT distributions — especially median and p90 — and the full latency of an agent execution.
The technical decision needs to keep three objectives together: limiting cost per task, improving responsiveness and preserving response quality. A higher cache rate is useful when it contributes to that balance.
This study continues with an analysis of the full request path, from frontend requests and backend preparation to the first visible token. Reducing latency between the user and the model
