Cost-Per-Token: The Physics Behind the Number New

I was building an inference GPU recommender — a tool that takes a workload description and ranks GPUs by cost per million output tokens (CPT). I thought I understood the formula, and then I had to rewrite it twice. The first rewrite was a physics gap: the formula ignored the KV cache, the stored attention keys and values the GPU re-reads for every token it generates. The second was a label problem. What I ended up with: a CPT number means very little until you know the context length, batch size and attention type behind it. ...

November 1, 2026 · 14 min · Srikanth Samudrla