Context window

The context window is the maximum number of tokens a model can consider at once, covering the instructions, the conversation, and everything sent alongside the question.

It is a ceiling, not an allowance. A model with a two-hundred-thousand-token window does not answer better when given two hundred thousand tokens; it answers from whatever it is given, and you are billed for all of it on every request.

Two things degrade as a window fills. Retrieval accuracy on a specific fact falls as the volume of unrelated material around it rises, and it falls fastest for material in the middle. And contradictions become likelier, because more material means more chances that two parts of it disagree.

Larger windows are therefore a capacity improvement rather than a quality one. They make it possible to send more. They do not make sending more a good idea.

All terms