The Memory Paradox of Large Models: Why AI Can Remember All Human Knowledge But Never "Yesterday"?

Author: Hugo, Robonaissance

Compiled by: TechFlow

TechFlow Summary: From early Atari game agents to large language models with context windows spanning millions of tokens, artificial intelligence has always faced a fundamental physical contradiction when making continuous decisions across time: while models can effortlessly ingest humanity's massive existing body of static knowledge, they cannot carry every past detail forward without loss in dynamic interactive environments.

Under the hard constraints of limited compute and context windows, accurately distilling the core states that dictate the future while compressing the past is not merely a technical breakthrough for resolving "perceptual aliasing," but also the foundational logic driving today's autonomous agents toward genuine persistent memory.

The State Paradox in Stilled Frames

This is Part 3 of the "Memory, Forgetting, and Self in Machines" series: When an agent cannot carry its entire past with it, how does intelligence actually unfold across time?

Try freezing a classic game of Pong at any single frame.

The screen shows only two paddles, a ball, and the current score. At this moment, you would be completely unable to state which direction the ball is traveling; even if shown this exact static image, a machine would be equally incapable of determining it.

In a 2015 paper on agents learning to play Atari games, scholars Matthew Hausknecht and Peter Stone pointed out bluntly: "A current single-frame screen can only reveal the positions of the paddle and the ball, but it cannot reveal the speed of the ball."

They then explained why this matters: "Knowing the direction of the ball's movement is the crucial element for determining the optimal position of the paddle."

This frozen frame is precise; every detail on the screen sits exactly where it should be. Yet this single frame alone is still insufficient to play the game.

What a sensor captures in this instant is merely an "Observation".

The true global context upon which the agent's next action depends is the "System State".

In Pong, the observation and the system state are completely disjointed across every single-frame slice. Because the ball's direction of motion is a fact derived from several past moments, no isolated instant contains it alone.

When immediate input fails to satisfy decision-making requirements, a well-performing system must carry certain information from its past into the present.

However, what it carries can never be the complete past. It carries only an abstraction, and the selection within this abstraction depends entirely on what needs to be predicted next.

This selection determines the informational foundation available in subsequent moments. Any details omitted from the abstraction will permanently be unavailable for reaction in later stages.

A Trade-off for Four-Frame Memory Depth

The system Hausknecht and Stone were responding to at that time was precisely the DQN agent developed by Volodymyr Mnih and others at DeepMind, published in Nature in 2015.

DQN learned to play dozens of Atari games purely through screen images, yet the paper's authors encountered the stilled-frame dilemma from the very beginning of development.

They wrote in the methodology section of the paper: "Since the agent can only observe the current screen, the task belongs to partially observable states, and many simulator states are perceptually aliased."

In other words, relying solely on the current screen image absolutely cannot inform the agent of the full situation.

The term "perceptual aliasing" provides a precise academic description for a common everyday frustration. Two radically different situations (e.g., the ball is flying toward your paddle, or away from it) produce identical images in a static frame.

A system that relies only on images is forced to treat them as the same situation; if these two situations require completely opposite actions, the system will inevitably make mistakes in one of them.

Their remedy was highly direct. Instead of receiving just one screen image, the network stacks recent frames together as input.

It defaults to 4 frames, though the algorithm remains adaptable to values of 3 or 5 frames.

By placing four frames side by side, the ball's motion directly manifests as positional change. Motion direction ceases to be a concept requiring abstract memory and becomes a visual attribute that can be seen directly.

The paper does not refer to this set of stacked frames as memory. It is essentially nothing more than an input channel that happens to be slightly wider than a single moment.

Perceptual Aliasing and Hard Physical Boundaries

This input width possesses an extremely rigid physical boundary, which Hausknecht and Stone precisely delineated.

They wrote: because DQN is trained using inputs composed of the four states the agent recently experienced, "if a game requires the player to remember events further back than the last four screens, DQN will be unable to master that game."

For DQN, history prior to the fifth-to-last screen simply does not exist within its input scope.

This mechanism based on a fixed sliding window directly exposes the core contradiction in machine intelligence's handling of time series: regardless of how wide the window is expanded, information beyond the boundary is physically erased.