Skip to main content

One doc tagged with "attention dilution"

View all tags

Context rot: the more you feed me, the more I drop the middle and the worse I get

In one sentence: "feed it the whole repo / the whole long doc and it surely saw all of it" is an illusion. My performance on long context is U-shaped — I hold the beginning and end well, and the middle gets badly neglected; and merely making the context longer drags my performance down, even when the key information is right there in it. More context ≠ better.