Gautier Billard
← AI case studies

Case study 01 / Ariath / Agent engineering

Prompt caching: optimising AI agent costs and latency.

Prefix stability, conversation persistence and context freshness: an engineering case study on inference costs and time to first token in Ariath.

01 / Objectives and constraints

Optimising inference cost and response latency.

Ariath is an AI assistant for iOS and the web that retains conversation context and uses tools. Its development involves two related constraints: controlling the cost of model requests and reducing the delay before a response begins. French language quality remains a requirement when selecting a model.

My initial approach focused mainly on input and output token prices. I chose an inexpensive Chinese model while rapidly developing features, expanding tools and enriching the context supplied to the agent. Request stability was not yet an explicit architectural objective.

The limitations I encountered with the models and providers I tested concerned both French language quality and variable latency. ChatGPT’s responsiveness reinforced the goal of a more immediate response, without providing a comparison under equivalent conditions. The work then shifted towards context construction and reuse between requests.

02 / Stack and observability

LangGraph for execution, Langfuse for diagnosis.

Ariath uses LangGraph to orchestrate the agent, its model requests and its tools. Execution state is persisted in PostgreSQL through checkpoints so a conversation can resume from its recorded state. These checkpoints belong to the application; prefix caching is handled by the inference provider. Preserving the former therefore does not guarantee reuse of the latter. LangGraph persistence

Langfuse is used to inspect executions and understand what is actually sent to the model. In this work, its role goes beyond aggregate metrics: I use it to investigate cache breaks, context construction and tool use. Langfuse observability

  • Investigating prefix changes. Compare the context of successive requests to identify rewritten information, a moved block or a change introduced when conversation history is restored.
  • Checking the transmitted context. Inspect which information enters the prompt, where it is inserted and how it evolves between turns: history, technical footers and information retrieved from memory.
  • Inspecting tools. Check calls, arguments and results, as well as the size of the catalogue exposed to the model. Tool definitions contribute to input context; their changes need to be examined alongside those of the messages.
  • Connecting context to execution. Examine durations, errors and available token counters to distinguish a prompt construction issue from time spent on a model request, RAG retrieval or tool execution.

A drop in the cache rate therefore triggers request inspection rather than immediate attribution to the model. A prefix difference provides a verifiable lead; if none is found, cache conditions and the provider used also need investigation. The charts in this article are calculated from the OpenRouter export, separately from Langfuse diagnostic traces.

03 / Inference performance

Separating TTFT from total response time.

TTFT measures the delay before the first token. For an individual request, it depends on factors including network transit, provider queueing and initial prompt processing, known as prefill. Generating subsequent tokens is a separate phase. OpenRouter latency documentation

Simplified processing sequence

Context preparation → request transmission → queueing and prefill → first token → subsequent generation.

Perceived latency in the application also includes its own processing. An agent turn can involve several model requests and tool executions.

Prompt caching reuses a prefix during input processing. It can reduce prefill work and therefore contribute to a lower TTFT. On its own, it does not guarantee faster output generation or stable end-to-end latency.

This distinction matters for optimisation: improving caching removes neither the wait at a busy provider nor the time spent executing a tool. The same cache rate can therefore coexist with very different response times.

04 / Cost model

Accounting for cache reads and writes.

Prefix caching reuses the processing of an initial portion of a prompt previously sent to the model. It does not return a stored answer. Reuse requires an identical prefix, subject to provider conditions such as minimum length and retention time. Anthropic documentation

An economic comparison needs to distinguish four components: uncached inputs, cache reads, cache writes and output tokens. Read and write rates depend on the model and provider; a cache write can cost more than an ordinary input token. OpenRouter caching documentation

Inference cost per request

C = Ninput × Pinput + Nread × Pread + Nwrite × Pwrite + Noutput × Poutput

N is a token count and P its unit price. Input categories must be disjoint under the provider’s billing rules: a cache-write token must not also be counted as an ordinary input token. Any ancillary charges are outside this breakdown.

A low input price is therefore insufficient to determine conversation cost. Actual reuse, write frequency and the number of requests needed to complete a task also matter. The cache rate remains a technical metric, distinct from the percentage saved on the bill.

05 / Request construction

Preserving a stable prefix between requests.

The first change to Ariath was to stabilise the system prompt and tool definitions. Reusable elements need a consistent representation across requests. A change near the start of the context can prevent reuse of the prefix that follows it.

This constraint applies to the content actually presented to the model: instructions, tool definitions and history. Two requests that appear semantically equivalent can still differ enough to lose reuse opportunities. Stability needs to be checked after the application’s transformations, not only in the stored data model.

I introduced a technical footer at the end of user messages to enrich the new turn’s context. New information is appended to the conversation without changing technical blocks already attached to previous turns.

Context organisation between two requests
Request N
Stable instructions and tools
Previously sent history
New turn + context
Request N+1
Same conversation prefix
Turn N kept unchanged
Continuation + new context

Solid blue: retained content. Dashed border: appended content. A logical diagram, not token proportions; exact block order depends on the API and cache eligibility on the provider.

The footer is a context construction convention. Its benefit depends on applying the same rule to subsequent requests: do not regenerate previous footers using the latest information.

06 / Persistence and context freshness

Preserving history while keeping knowledge current.

I also revised conversation storage and decoding to preserve context when a conversation resumes. Persistence needs to reconstruct previously transmitted messages without implicitly rewriting their technical information.

During LLM-assisted development, coding assistants regularly proposed updating context inside the history. This prioritised information freshness but changed the replayed prefix. I made the constraint explicit: refreshing context must not silently mutate previous exchanges.

Ariath handles freshness through complementary mechanisms:

  • Prioritising recent information. The prompt asks the model to give more weight to fresh information when the context contains several states of the same information.
  • Searching user memory. RAG searches run when relevant to retrieve context useful to the request.
  • Context compaction. Once the context reaches a certain volume, it is compacted to limit its size.

Compaction introduces a trade-off: it changes the context presented to the model and can therefore require part of the cache to be rebuilt. The objective is to keep a stable prefix between these transitions while controlling total volume and information relevance. Maximising the cache rate is not sufficient reason to accumulate context indefinitely.

07 / Measurements and results

73% caching in October, with daily peaks above 85%.

Early in this work, my recollection was that roughly 25% of input tokens were served from cache under favourable conditions. After the changes, I regularly observed rates around 70–75%. Analysing an OpenRouter export provides more precise evidence.

Within the selected scope, the token-weighted rate rose from 20.6% in July to 73.0% in October. It was 67.8% for 1–15 November. The full series shows monthly variations and the two days above 85%.

Two days above 85%

1 October 2026

85.4%

629 included requests
36,745,694 input tokens

27 October 2026

85.7%

548 included requests
10,707,605 input tokens

100 %85 %0 % Share of input tokens served from cacheDaily rate weighted by tokens · 26 April to 15 November 2026. Ringed points mark days at or above 85%; gaps indicate days with no included requests.1 October 2026: 85.4 %27 October 2026: 85.7 %

Daily rate weighted by tokens · 26 April to 15 November 2026. Ringed points mark days at or above 85%; gaps indicate days with no included requests. These daily peaks are not monthly averages. They use the same filters and the same ratio of token sums.

View daily values
Share of input tokens served from cache
Period in 2026Weighted rateIncluded requests
2026-04-290.0 %2
2026-04-300.5 %2
2026-05-050.0 %1
2026-05-110.0 %9
2026-05-127.0 %181
2026-05-1317.1 %256
2026-05-1435.8 %10
2026-05-1528.8 %70
2026-05-1638.5 %93
2026-05-1811.8 %179
2026-05-1918.3 %294
2026-05-2018.3 %144
2026-05-2124.1 %162
2026-05-2220.1 %39
2026-05-236.4 %19
2026-05-249.1 %6
2026-05-255.1 %94
2026-05-2636.2 %258
2026-05-2758.8 %158
2026-05-2862.3 %378
2026-05-2964.1 %283
2026-05-3044.1 %47
2026-05-3148.2 %129
2026-06-0146.9 %1,341
2026-06-0277.8 %347
2026-06-0356.5 %275
2026-06-0452.6 %113
2026-06-0528.2 %559
2026-06-0623.3 %78
2026-06-0732.5 %225
2026-06-0825.4 %339
2026-06-0942.9 %243
2026-06-1024.6 %118
2026-06-1140.2 %83
2026-06-1220.8 %198
2026-06-1325.8 %140
2026-06-1416.0 %288
2026-06-1515.6 %284
2026-06-1635.9 %368
2026-06-1713.4 %93
2026-06-1819.7 %351
2026-06-1914.8 %342
2026-06-2036.0 %211
2026-06-2137.6 %139
2026-06-2223.6 %113
2026-06-2323.2 %75
2026-06-2423.8 %113
2026-06-2521.8 %181
2026-06-2627.6 %329
2026-06-2721.2 %349
2026-06-2828.7 %398
2026-06-2923.7 %307
2026-06-3022.9 %282
2026-07-0134.6 %156
2026-07-0227.2 %662
2026-07-0316.2 %532
2026-07-0437.5 %229
2026-07-0518.6 %97
2026-07-0617.8 %465
2026-07-0732.5 %213
2026-07-0816.9 %322
2026-07-0913.6 %310
2026-07-1023.1 %296
2026-07-1112.3 %172
2026-07-1216.6 %195
2026-07-1343.6 %196
2026-07-145.6 %126
2026-07-1516.4 %67
2026-07-167.2 %211
2026-07-176.3 %147
2026-07-183.9 %127
2026-07-1915.2 %93
2026-07-2015.0 %207
2026-07-2112.9 %170
2026-07-2217.5 %130
2026-07-2311.5 %174
2026-07-2423.7 %484
2026-07-2510.9 %142
2026-07-267.1 %71
2026-07-2723.7 %112
2026-07-2814.9 %161
2026-07-2915.6 %113
2026-07-3039.6 %177
2026-07-3131.3 %241
2026-08-0132.1 %333
2026-08-0247.5 %132
2026-08-0317.4 %203
2026-08-0438.0 %451
2026-08-0519.3 %275
2026-08-0629.5 %322
2026-08-0736.4 %300
2026-08-0843.5 %127
2026-08-0924.4 %106
2026-08-1029.0 %149
2026-08-1121.7 %212
2026-08-1241.7 %619
2026-08-1341.6 %687
2026-08-1449.3 %776
2026-08-1553.1 %736
2026-08-1649.4 %420
2026-08-1749.6 %477
2026-08-1842.7 %579
2026-08-1944.0 %469
2026-08-2031.9 %105
2026-08-2148.4 %136
2026-08-2241.2 %137
2026-08-2347.9 %159
2026-08-2448.5 %332
2026-08-2548.6 %345
2026-08-2650.1 %236
2026-08-2739.5 %270
2026-08-2853.2 %254
2026-08-2942.3 %90
2026-08-3038.3 %219
2026-08-3138.5 %237
2026-09-0129.9 %350
2026-09-0238.2 %757
2026-09-0345.1 %214
2026-09-0446.0 %389
2026-09-0540.1 %199
2026-09-0653.7 %342
2026-09-0744.8 %206
2026-09-0847.8 %281
2026-09-0953.7 %364
2026-09-1059.5 %392
2026-09-1181.9 %326
2026-09-1264.3 %134
2026-09-1367.2 %215
2026-09-1461.5 %259
2026-09-1565.7 %241
2026-09-1663.0 %111
2026-09-1764.6 %204
2026-09-1870.5 %390
2026-09-1954.7 %119
2026-09-2056.6 %134
2026-09-2162.8 %544
2026-09-2263.8 %626
2026-09-2368.4 %520
2026-09-2469.3 %313
2026-09-2560.6 %313
2026-09-2669.3 %567
2026-09-2753.0 %158
2026-09-2855.8 %171
2026-09-2969.4 %602
2026-09-3072.5 %642
2026-10-0185.4 %629
2026-10-0274.8 %639
2026-10-0368.1 %494
2026-10-0463.4 %305
2026-10-0568.2 %414
2026-10-0670.3 %493
2026-10-0773.6 %429
2026-10-0851.1 %494
2026-10-0977.0 %436
2026-10-1074.8 %242
2026-10-1162.0 %346
2026-10-1240.8 %2,625
2026-10-1362.5 %496
2026-10-1475.5 %290
2026-10-1585.0 %359
2026-10-1681.0 %558
2026-10-1767.9 %193
2026-10-1867.9 %191
2026-10-1980.1 %405
2026-10-2070.0 %276
2026-10-2177.4 %130
2026-10-2277.2 %171
2026-10-2380.1 %360
2026-10-2461.9 %94
2026-10-2565.6 %88
2026-10-2674.5 %257
2026-10-2785.7 %548
2026-10-2883.9 %651
2026-10-2974.8 %245
2026-10-3075.6 %432
2026-10-3158.3 %237
2026-11-0169.7 %844
2026-11-0269.8 %667
2026-11-0365.7 %150
2026-11-0463.0 %370
2026-11-0561.0 %542
2026-11-0663.6 %600
2026-11-0762.4 %313
2026-11-0851.2 %204
2026-11-0972.6 %852
2026-11-1072.7 %453
2026-11-1165.1 %175
2026-11-1258.3 %582
2026-11-1370.2 %448
2026-11-1482.7 %272
2026-11-1577.1 %334
Share of input tokens served from cache
Period in 2026Weighted rateIncluded requestsMedian TTFT (s)
26–30 April0.2 %46.52
May38.3 %2,8101.20
June37.4 %8,2821.58
July20.6 %6,7982.19
August42.1 %9,8931.77
September60.1 %10,0831.55
October73.0 %13,5271.66
1–15 November67.8 %6,8061.90
Source: personal activity export, aggregated calculations. Median TTFT per request as reported by OpenRouter, across all included models and providers. April has only four requests and November is incomplete. Raw logs and identifiers are not published.
Method and scope

OpenRouter export dated 15 November 2026: 82,970 rows, with 58,203 requests retained from Prisma and Mentat keys. OpenClaw, OpenRouter Chatroom, embeddings, cancelled or failed requests, nonpositive inputs and missing cache counters are excluded. Periods follow the exported dates, which do not specify a timezone.

Rate = sum of tokens read from cache ÷ sum of input tokens. Requests with zero cached tokens remain included; a missing counter is not treated as zero.

Improving the cache rate does not automatically produce a continuous fall in TTFT. In this export, median TTFT per request was 2.19 s in July, 1.55 s in September, 1.66 s in October and 1.90 s in the first half of November. Cache rate and latency describe different dimensions of the system.

These are observational results. Models, providers, context sizes and workloads change during the period; the export cannot isolate the causal effect of each modification. Reported TTFT concerns requests recorded by OpenRouter, not full latency measured in Ariath’s interface. No percentage reduction in cost or response time is inferred from the cache rate.

08 / Architectural lessons

Measuring reuse, cost and latency separately.

This optimisation covered the entire context construction chain: system prompts, tool definitions, message enrichment, storage and conversation replay. Caching became an explicit design constraint, with deliberate transitions when context needs compaction.

To attribute gains precisely, the validation step is to compare representative requests with a fixed model, provider and configuration, first with a cold cache and then with reuse. Relevant measures include tokens read and written, billed cost, TTFT distributions — especially median and p90 — and the full latency of an agent execution.

The technical decision needs to keep three objectives together: limiting cost per task, improving responsiveness and preserving response quality. A higher cache rate is useful when it contributes to that balance.

This study continues with an analysis of the full request path, from frontend requests and backend preparation to the first visible token. Reducing latency between the user and the model

Further work

Tool exposure and context management.

Reducing the tool set, dynamic discovery, transient context and sub-agent delegation affect both request volume and request count. These trade-offs warrant a separate study so their cost and performance effects can be examined with their own measurements.

Gautier Billard

Gautier Billard

Mobile Tech Lead, AI specialist and creator of Ariath.

Discuss this case study