Right name, wrong rules

In one paragraph

The main experiment shows gemma-2-2b reliably returning the right local variable (71 of 72 name-and-shape combinations) through a mechanism of per-name pathways. This page asks the harder question that mechanism invites: does the model hold the actual rules of Python binding? Ten classic traps say no. Each is cut at a single token where the correct semantics and the natural misreading predict different continuations. Scored per case it gets 4 of 10, and even that number is misleading. The cases come in pairs of near-identical surface form with opposite answers, and the pairs settle it: 0 of 4 pairs passed. Where the model scores a hit, it gives the same answer to both members of the pair. That is surface-form matching, not binding semantics.

The sharpest case: the for-loop variable does leak into enclosing scope, and the model says it does not (35.6% on Traceback). A comprehension variable does not leak, and the model says it does (33.6% on 2). It holds both rules, swapped, with matching confidence.

The design

Three decisions carry this probe, and each one closes a way of fooling yourself.

• Ground truth is executed, not remembered. Every case carries its runnable source and an expected output. The script runs the source and refuses to start if Python disagrees. Binding traps are famous precisely because people get them wrong. Asserting the answers from memory would put the scorer's memory on the same footing as the model's.
• Every case cuts at a single-token divergence point. The prompt is a transcript prefix ending exactly where the correct semantics and the natural misreading predict different next tokens. One forward pass decides it: no generation, no parsing of free-form output.
• Cases come in pairs. Each pair differs by a few characters (lambda: i against lambda i=i: i) and has opposite correct answers. A model matching on surface form will give the same answer to both, scoring one hit by luck. Only passing both members counts.

How a case is scored

One forward pass per case. The model sees the transcript up to the cut and nothing else. From the softmax at that position the probe reads the probability of two tokens, the correct continuation and the natural misreading, and ignores every other token. The verdict follows three rules, applied in this order:

• noise: both candidates below 2%. The model's answer is neither token, so comparing them compares scraps. class-body-scope is the example: Traceback 0.54% against > 0.07%, while the actual top-1 was >>> at 70.7%. Calling the larger of two negligible numbers "correct" would be free points, so it scores nothing.
• ok: P(correct) > P(wrong). The correct token beats the misreading, by any margin. There is no threshold on the margin. The top-1 column in the table below is there so a narrow or accidental "ok" can be seen for what it is.
• MISS: otherwise. The misreading is at least as likely as the correct token.

How a pair is scored

A pair passes only if both members score ok. Anything else (ok+MISS, ok+noise, MISS+MISS) is a fail. The reason is in what the members share. The two prompts are near-identical in form and opposite in answer. A model that answers from the form gives the same token to both, and that token is correct for exactly one of them. An ok sitting next to a MISS in the same pair therefore earns no half credit. It is the signature of form-matching. The two cases without a natural twin (tuple-augmented-assign, chained-comparison) are scored on their own and do not enter the pair count. Per-case totals are reported only to show how much they flatter the model.

What the model answers

The top-1 column is worth reading on its own. In two cases (class-body-scope, chained-comparison) the model's actual answer is a REPL prompt marker. It is continuing the transcript's shape rather than choosing either semantics. That is what the noise floor exists to catch.

The four pairs

Closure capture: same answer to both

>>> fs = [lambda: i for i in range(3)]
>>> print([f() for f in fs])
[█   ← model: '0' at 79.3%   (wrong)

>>> fs = [lambda i=i: i for i in range(3)]
>>> print([f() for f in fs])
[█   ← model: '0' at 66.8%   (right)

The i=i is the whole difference between the two prompts, and the model's answer does not move: 0 at 79.3%, 0 at 66.8%. It has learned that this shape of snippet prints [0, 1, 2]. That is the famous expectation the first trap exists to break. One of the two scores "correct". It should get no credit.

Mutable default: same answer to both

>>> def add(x, acc=[]):
...     acc.append(x)
...     return acc
...
>>> add(1)
[1]
>>> add(2)
[█   ← model: '1' at 89.1%   (right)

>>> def add(x, acc=None):
...     acc = acc or []
...     acc.append(x)
...     return acc
...
>>> add(1)
[1]
>>> add(2)
[█   ← model: '1' at 93.4%   (wrong)

Here the model's fixed answer happens to match the trap side, so the famous case scores "correct". The fixed version, the one every Python tutorial tells you to write, is answered as if the fix did nothing. Its confidence is higher on the wrong member (93.4%) than on the right one (89.1%).

Loop-variable leak: exactly inverted

>>> for i in range(3):
...     pass
...
>>> print(i)
█   ← model: 'Traceback' at 35.6%   (wrong)

>>> [j for j in range(3)]
[0, 1, 2]
>>> print(j)
█   ← model: '2' at 33.6%   (wrong)

The sharpest result in the set. This is not ignorance; ignorance would look like noise. The model holds a definite rule for each construct. Both rules are wrong, and they are wrong in opposite directions, at matching confidence. It has the two scoping behaviours swapped.

Class-body comprehension: the illegal one vanishes

>>> class C:
...     xs = [1, 2, 3]
...     ys = [x for x in range(5) if x in xs]
...
█   ← model: '>>>' at 70.7%   (neither)

>>> class C:
...     xs = [1, 2, 3]
...     ys = [x * 2 for x in xs]
...
>>> C.ys
[█   ← model: '2' at 93.8%   (right)

The legal member is answered confidently and correctly. The illegal member gets a fresh REPL prompt at 70.7%. The model continues the session as if the class defined cleanly, with Traceback at 0.54%. Both candidates are under the noise floor, so this scores noise rather than a hit for either side. The model was not weighing the two semantics at all.

The two unpaired cases

>>> t = ([1],)
>>> t[0] += [2]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment
>>> t
([1█   ← model: ',' at 32.8%   (right; top-1 was '],' at 37.1%)

>>> 1 in [1, 2] == True
█   ← model: 'True' 11.2% vs 'False' 5.3%   (wrong; top-1 was '>>>' at 64.5%)

Neither has a natural twin, so they are scored on their own and carry no weight in the pair count. The augmented-assignment case is the one "ok" outside the pairs; the chained comparison is mostly not answered at all.

Why per-case scoring lies

This is a measurement point as much as a model point. A benchmark reporting "gemma-2-2b: 40% on Python binding traps" would be technically true and completely misleading. The paired design shows the number does not measure the semantics. It measures one shape-prior scoring lucky hits. Two scoring rules make the result honest: only pairs count, and sub-noise comparisons score nothing. Both exist to take away points the model did not earn.

The unpaired cases carry the same warning in miniature. tuple-augmented-assign scores "ok", but the model's actual top choice was ], (a token that starts the right answer for a plausibly different reason). And chained-comparison's miss comes with the model mostly answering >>>: it would rather end the transcript than evaluate the expression.

What this bounds

The main page shows a mechanism that reliably returns the right variable in accumulator functions: 71 of 72 name-and-shape combinations. That mechanism is per-name pathways, volume set by binding, winner-take-all at the logit layer. This page is the boundary of that result. The mechanism tracks which name carries how much drive; it is not backed by the rules of Python binding. Step outside the prompt family, into cases where the question is when a name is resolved rather than which name is strongest. There the model answers from surface shape.

Both facts matter for the same audience. An AI coding tool built on a model this size can name the right variable essentially always. It will also confidently misjudge closure capture, mutable defaults and comprehension scope. Those are the exact places where human intuition also fails and where a debugger is most needed. The failure is not "the model is weak". It holds definite, confident, wrong rules, which is worse than noise. Per-case benchmarks will score much of it as competence.

Back to the main experiment →

Reproducing this

This probe needs no transcoders and no attribution run. It is plain transformers forward passes on google/gemma-2-2b in bfloat16, one per case. The script (probe_binding_traps.py, in the same currently-private fork as the main experiment's code) carries each case's runnable source and expected output. It executes every snippet first and refuses to start if Python's actual behaviour disagrees with the recorded ground truth. The full output behind every number on this page is published here as probe_binding_traps_gemma-2-2b.txt.

Related, in the same spirit and now with its own page: generator-expression late binding (PEP 289). There the model handles a no-reassignment control at 98.4% but gives the correct continuation 0.00% on the one cut that separates eager from lazy capture. It is worse on the generator expression than on the desugared function PEP 289 defines it to mean. Four further chapters (floats, set order, bound-method identity, interpreter versions) pass 1 of 13 pairs.

Credits

Companion page to Which variable does the model return?. Model google/gemma-2-2b, probed with plain transformers. The trap cases are the community's classics; ground truth is re-executed on every run rather than taken on authority.