On inference cost and attention
ideasSmall Tools, Long Horizonsaisystems
Summary
When inference is cheap, the scarce resource stops being compute and becomes the reader's willingness to read the output.
The shifted bottleneck
Generation is nearly free, so most interfaces overproduce. The cost lands on the reader, who has no way to tell which paragraph mattered.
Design consequence
Systems should be judged on what they decline to output. Restraint is the feature that does not benchmark well.