shelf.cafe
top new
submit sign in

Why your prompt cache hit rate is zero: the silent invalidators

0

Mike Bifulco · 2mo · discuss

Prompt caching is a prefix match: one changing byte near the top of your prompt drops your hit rate to zero. A timestamp is the obvious culprit - here are the rest.

Sign in
shelf.cafe - just a place to share interesting blog posts and articles.
about

We can't find the internet

Attempting to reconnect

Something went wrong!

Attempting to reconnect