Cache hit rate

The cache hit rate is the share of input tokens on a request that were served from a prompt cache rather than processed fresh.

It is the number that turns a token count into a cost. Two requests with the same raw input can differ severalfold in price depending on how much of that input was already cached.

In practice the rate is driven by stability at the front of the request. A stable prefix that grows only at the end caches well; anything that rewrites the beginning of the sequence does not.

When comparing two ways of running the same task, weighted input, meaning fresh tokens at full weight plus cached tokens at their reduced weight, is the measurement that corresponds to what you pay. Raw input is the measurement that is easier to quote.

All terms