The expression is not enough

In one paragraph

The main experiment shows gemma-2-2b tracking what a name is bound to. The trap pairs show it does not hold the rules of when a binding happens. This page pushes on two more fronts, with 35 scored single-token cases across two probes, 30 of them new. First, PEP 289 defines a generator expression to mean its desugared generator function: one program, written twice. The model is worse on the form a human would actually write, on both cuts probed. Second, four chapters of behaviours where the printed expression underdetermines the answer (float width, hash-table order, bound-method identity, interpreter version): 1 of 13 pairs pass. The model answers float comparisons by literal identity rather than arithmetic. It assumes sets preserve insertion order at 97.3% confidence. It gives the same locals() answer for Python 3.11, 3.12 and 3.13, versions across which the answer changed.

The design, and three new honesty mechanisms

Everything from the trap probe's design carries over. Ground truth is executed, not remembered, here under the exact interpreter (python3.11/3.12/3.13 where a case names one) and environment (PYTHONHASHSEED=0 where it matters). Every case cuts at a single-token divergence point. Cases come in pairs of near-identical form with opposite answers. Three new mechanisms were each forced by a failure observed while building this probe:

• Coverage, because the model may not be answering. On bare >>> expr prompts the model's top prediction was >>> itself, at 37-45%. It declined to answer and moved to the next prompt, leaving both candidates in the noise. The summary now prints P(correct) + P(wrong) for every case, so a verdict resting on scraps is visible as such.
• Counterbalanced priming, because the context can answer for the model. The fix for the non-answering failure is two worked True/False exchanges before the prompt. But a primer can bias the answer, so every primed case runs under both primer orders. A verdict that flips with the order scores prime-dep, not a win. Three cases flipped, including 0.1 + 0.2 == 0.3 itself: an exact 48.83/48.83 split under one primer, 74.2% True under the other. That verdict belongs to the primer.
• Exact ties score tie, because argmax invents a preference. bfloat16 makes exact ties common (40.43/40.43, 44.14/44.14 in this work). Argmax resolves them by candidate order. The correct candidate is always listed first, so every such coin flip would have been recorded as a hit.

How a case is scored

One forward pass per prompt. From the softmax at the cut the probe reads two numbers: P(correct) and P(wrong), the probability of the correct continuation and of the natural misreading. It ignores every other token. From those two numbers:

• coverage = P(correct) + P(wrong). How much of the model's answer the two candidates account for. It is printed for every case. It is not a verdict, but it says how much the verdict is worth. At 97% the model chose between the two tokens. At 19% (dict-reversed-keeps-order) it mostly predicted something else, and the verdict rests on scraps.
• margin = |P(correct) − P(wrong)|. Reported, not thresholded: no cutoff below which a win stops counting is claimed.
• noise: the larger of the two candidates is below 2%. The answer is neither token; the case scores nothing.
• tie: margin is exactly 0. The two candidates have identical probability, which in bfloat16 happens often. Scored as neither ok nor MISS, and it fails its pair.
• ok: P(correct) > P(wrong), by any margin.
• MISS: P(wrong) > P(correct).
• prime-dep: ch.1 only, where each case runs under two primer orders. If the two runs give different verdicts, the case is prime-dep regardless of what they were; only two matching verdicts carry through. The two primers are shown at the top of ch.1.

How a pair is scored

Same rule as the trap probe: a pair passes only if both members are ok. A tie, noise or prime-dep on either side fails it just as a MISS does. Pairs are counted per chapter and never pooled across chapters, because ch.1 is float arithmetic and ch.4 is a fact about interpreter releases. One number over both measures nothing. Cases marked unscored (the unseeded string set, whose output changes with the process) are executed and shown but never put to the model. The seeded string set is scored but unpaired.

One program, written twice (PEP 289)

The material is real. The snippets come from a teaching video, kept precisely because synthetic prompts invite "you designed that":

>>> lst = [1, 2, 3, 4, 5]
>>> g = (n for n in lst if n in lst)
>>> lst = [0, 1, 2]
>>> print(list(g))
[1, 2]

The iterable is captured eagerly (old [1..5]). The filter resolves the global lazily (new [0,1,2]). The output is their intersection, neither list on the page. PEP 289 defines this expression to mean its desugaring: an explicit generator function taking iter(lst) as an argument and filtering on the global. That desugared form is the second spelling the probe uses:

>>> lst = [1, 2, 3, 4, 5]
>>> def __g(it):
...     for n in it:
...         if n in lst:
...             yield n
...
>>> g = __g(iter(lst))
>>> lst = [0, 1, 2]
>>> print(list(g))
[1, 2]

The probe cuts both spellings at the same two points. A disagreement between them rules out reading of the semantics outright; agreement proves little, since two matching wrong answers also agree.

Both cuts move the same way: the sugar itself costs it. Controls rule out the cheap explanations. With no reassignment it is correct at 99.61% at the same list length, so closing the list early is not a taste for short lists. Merely adding the vacuous filter if n in lst, before any rebinding, costs 10 pp (89.45%). On the one cut that cleanly separates eager from lazy capture (whether (3, …) appears after the inner list is exhausted), the correct continuation receives 0.00%.

All nine cuts

Every case is a REPL transcript cut just before a single token that the correct semantics and the natural misreading predict differently. The model sees exactly the text above the cursor and nothing else. Nine cuts, shown in full.

Controls: no reassignment, so nothing is captured early or late

>>> lst = [1, 2, 3]
>>> g = (a for a in lst)
>>> print(list(g))
[1, 2█   ← model: ',' at 98.44%   (right)

>>> lst = [1, 2, 3, 4, 5]
>>> g = (n for n in lst)
>>> print(list(g))
[1, 2█   ← model: ',' at 99.61%   (right)

>>> lst = [1, 2, 3, 4, 5]
>>> g = (n for n in lst if n in lst)
>>> print(list(g))
[1, 2█   ← model: ',' at 89.45%   (right)

Which list is iterated: the outermost iterable is captured eagerly, so the first element is 1

>>> lst = [1, 2, 3, 4, 5]
>>> g = (n for n in lst if n in lst)
>>> lst = [0, 1, 2]
>>> print(list(g))
[█   ← model: '0' at 55.08%, '1' at 37.70%   (wrong)

>>> lst = [1, 2, 3, 4, 5]
>>> def __g(it):
...     for n in it:
...         if n in lst:
...             yield n
...
>>> g = __g(iter(lst))
>>> lst = [0, 1, 2]
>>> print(list(g))
[█   ← model: '1' 40.43%, '0' 40.43%   (exact tie)

Which list the filter uses: n in lst is resolved lazily against the new list, so the output stops at [1, 2]

>>> lst = [1, 2, 3, 4, 5]
>>> g = (n for n in lst if n in lst)
>>> lst = [0, 1, 2]
>>> print(list(g))
[1, 2█   ← model: ']' at 53.12%, ',' at 46.88%   (right, by 6.25 pp)

>>> lst = [1, 2, 3, 4, 5]
>>> def __g(it):
...     for n in it:
...         if n in lst:
...             yield n
...
>>> g = __g(iter(lst))
>>> lst = [0, 1, 2]
>>> print(list(g))
[1, 2█   ← model: ']' at 73.05%, ',' at 26.95%   (right, by 46.09 pp)

Nested loops: outer for a in lst is eager (old [1, 2, 3]), inner for b in lst is lazy (new [1, 2]); Python prints [(1, 1), (1, 2), (2, 1), (2, 2), (3, 1), (3, 2)]

>>> lst = [1, 2, 3]
>>> g = ((a, b) for a in lst for b in lst)
>>> lst = [1, 2]
>>> print(list(g))
[(1, 1), (1, 2), (█   ← model: '2' at 99.22%   (right, but recency predicts it too)

>>> lst = [1, 2, 3]
>>> g = ((a, b) for a in lst for b in lst)
>>> lst = [1, 2]
>>> print(list(g))
[(1, 1), (1, 2), (2, 1), (2, 2)█   ← model: ',' 0.00%, ']' 0.00%; top-1 is a newline 53.9%   (noise)

test1-third-pair passes because the inner loop being exhausted at 2 is also what a recency heuristic predicts. test1-does-3-appear is the only cut where eager and lazy capture of the outer iterable give different tokens. There the model's top prediction is a newline: it thinks the output ends at (2, 2).

Four chapters, thirteen pairs, one pass

Twenty-six scored cases in four separately scored chapters. Separately, because pooled accuracy across a float-arithmetic chapter and an interpreter-versioning chapter measures nothing. The single passing pair is list-vs-set order preservation, the least surprising in the set.

ch.1, floats: literal identity, not arithmetic (0/6 pairs)

1e500 is not a big number. The literal overflows to inf at parse time, and 1e500/1e500 is nan. Twelve cases, six pairs, and all twelve verdicts fit a single heuristic: same expression written twice → True, different literals → False.

Every prompt in this chapter is one bare expression. On its own the model tends to answer with another >>> rather than a value, so two worked exchanges are placed in front of it. Because the last exchange might bias the answer, each case is run twice, once under each order. False-first and True-first are the names of those two prompts. A case only counts if both runs agree:

>>> 1 == 2
False
>>> 1 == 1
True
>>> <expr>
█

>>> 1 == 1
True
>>> 1 == 2
False
>>> <expr>
█

Each block below reports the answer under both. prime-dep means the two runs gave different verdicts, so the verdict belongs to the primer, not the model.

a + b == c: inexact vs exact

>>> 0.1 + 0.2 == 0.3
█   ← False-first: False 48.83% / True 48.83% (tie)
    True-first:  True 74.22%   (prime-dep)

>>> 0.25 + 0.5 == 0.75
█   ← False-first: True 60.55%
    True-first:  True 85.16%   (right both ways)

float == int: inexact vs exact

>>> 1e50 == 10 ** 50
█   ← False-first: False 46.09% / True 46.09% (tie)
    True-first:  True 58.59%   (prime-dep)

>>> 2.0 ** 50 == 2 ** 50
█   ← False-first: True 57.81%
    True-first:  True 80.86%   (right both ways)

1eX == 1eY: overflowed vs not

>>> 1e500 == 1e600
█   ← False-first: False 66.41%
    True-first:  False 67.58%   (wrong both ways)

>>> 1e50 == 1e60
█   ← False-first: False 69.14%
    True-first:  False 74.61%   (right both ways)

float > 10**1000: inf vs finite

>>> 1e500 > 10 ** 1000
█   ← False-first: True 47.66% / False 47.66% (tie)
    True-first:  True 62.50%   (prime-dep)

>>> 1e300 > 10 ** 1000
█   ← False-first: True 50.78%
    True-first:  True 62.89%   (wrong both ways)

x == x: inf vs nan

>>> 1e500 == 1e500
█   ← False-first: True 78.91%
    True-first:  True 91.41%   (right both ways)

>>> 1e500 / 1e500 == 1e500 / 1e500
█   ← False-first: True 83.20%
    True-first:  True 91.02%   (wrong both ways)

> 0 after an inf operation

>>> 1e500 * 1e500 > 0
█   ← False-first: True 58.20%
    True-first:  True 64.06%   (right both ways)

>>> 1e500 / 1e500 > 0
█   ← False-first: True 58.98%
    True-first:  True 72.27%   (wrong both ways)

The four identity-shaped cases (1e500 == 1e500, 1e500/1e500 == 1e500/1e500, 1e50 == 1e60, 1e500 == 1e600) are the tell. The verdict tracks whether the two sides are the same string, never what they evaluate to. Nothing was computed. The three prime-dep cases are exact ties under one primer and a confident answer under the other, so the verdict belongs to the primer. The exact/inexact pairs were answered True both ways, consistent with a general lean toward True on arithmetic that is not an identity comparison.

ch.2, sets: the order you see is the hash table (1/3 pairs)

dicts preserve insertion order; sets never promised to. The sharpest minimal pairs in the whole probe: a dict comprehension and a set built from the same elements, both printed with braces, opposite first element.

reversed() into [] vs {}: the one passing pair, plus the dict twin

>>> list(reversed([x for x in range(5)]))
[█   ← model: '4' at 48.24%, '0' at 6.54%   (right)

>>> set(reversed([x for x in range(5)]))
{█   ← model: '0' at 35.16%, '4' at 11.43%   (right)

>>> {k: None for k in reversed(range(5))}
{█   ← model: '0' at 11.47%, '4' at 7.91%; coverage 19%   (wrong)

range(1022, 1027) into {}: dict vs set

>>> {x: None for x in range(1022, 1027)}
{102█   ← model: '2' at 81.64%, '4' at 2.80%   (right)

>>> set([x for x in range(1022, 1027)])
{102█   ← model: '2' at 97.27%, '4' at 0.45%   (wrong)

Strings: a floor, not a fair test

$ PYTHONHASHSEED=0 python -c "print(set([str(x) for x in range(5)]))"
{'█   ← model: '0' at 80.08%, '2' at 1.47%   (wrong, as expected)

Pairs: dict-reversed/set-reversed fails on the dict side, where the model mostly predicted neither candidate. dict-int/set-int fails too. The wrap case is the most confident wrong answer in this study: the model assumes sets keep insertion order the way dicts do. list-reversed/set-reversed is the one pass. The string-set case under PYTHONHASHSEED=0 is included only as a floor: deterministic but underivable, since passing it would mean having memorised CPython's siphash. The unseeded variant is executed under two seeds to prove its output differs, then never shown to the model. There is no correct token to score.

ch.3, bound methods: CPython behaviour, stated as such (0/3 pairs)

o.f constructs a fresh bound method on every attribute access, so o.f is o.f is False while C.f is C.f and o.f == o.f are True. id(o.f) == id(o.f) is True only because the allocator hands the freed address straight back. A single interleaved allocation flips it to False (both stable 5/5 on this build). This chapter is marked implementation-defined in the harness. A ground-truth mismatch on another build warns instead of aborting, and none of it is evidence about "Python".

Every prompt in this chapter starts from the same prelude, shown once here and elided below:

>>> class C:
...     def f(self):
...         pass
...
>>> o = C()

is vs == on o.f, and C.f vs o.f under is

…
>>> C.f is C.f
█   ← model: 'True' at 52.73%   (right)

…
>>> o.f is o.f
█   ← model: 'True' at 53.52%   (wrong)

…
>>> o.f == o.f
█   ← model: 'True' 44.14%, 'False' 44.14%   (exact tie)

The same two id() calls, one allocation apart

…
>>> id(o.f) == id(o.f)
█   ← model: 'True' at 55.47%   (right, for the wrong reason)

…
>>> a = id(o.f)
>>> junk = str(a)
>>> a == id(o.f)
█   ← model: 'True' at 48.83%   (wrong)

Pairs: method-is-fresh-object/method-compares-equal (is vs ==), classfunc-is-itself/method-is-fresh-object (C.f vs o.f under is), and the two id() cases one allocation apart.

Read the last three cases together: the model says True to every id() question, which happens to be right exactly once. The is/== pair fails on a miss and an exact tie.

ch.4: the same code, three interpreters (0/1 pair)

A comprehension with a cell variable, asking for locals() inside it, prints {'.0': <iterator>, 'k': 0} on Python 3.11 and {'k': 0} on 3.12/3.13. PEP 709 inlined comprehensions. Ground truth was executed on all three interpreters. The prompt is a shell transcript naming the interpreter, and only that token differs between cases.

The same snippet under three interpreters; only the python3.1x token differs

$ python3.11 -c "def f():
    lambda: k
    k = 1
    print([locals() for k in [0]])
f()"
[{'█   ← model: 'k' at 22.46%, '.' at 0.00%   (wrong)

$ python3.12 -c "def f():
    lambda: k
    k = 1
    print([locals() for k in [0]])
f()"
[{'█   ← model: 'k' at 21.09%, '.' at 0.00%   (right)

$ python3.13 -c "def f():
    lambda: k
    k = 1
    print([locals() for k in [0]])
f()"
[{'█   ← model: 'k' at 22.85%, '.' at 0.00%   (right)

The model answers 'k' for every version: right for 3.12/3.13, and structurally incapable of the 3.11 answer, at 0.00%. It has one answer per snippet, not a model of interpreter versions.

What this bounds

With the trap pairs, the picture is consistent across three probes and 45 scored cases. The mechanism the main experiment verified tracks which name carries how much drive, and essentially nothing beyond it. When the answer requires evaluation (of time, of scope, of float width, of a hash table, of an interpreter version), the model substitutes the nearest surface regularity, often at high confidence. For an AI debugging tool this is the exact failure profile that matters. The places where the expression on the page underdetermines the behaviour are the places debugging happens.

One model, behavioural probes only, single-token cuts; no cross-model claims.

Back to the main experiment →

Reproducing this

Both probes are plain transformers forward passes on google/gemma-2-2b in bfloat16: no transcoders, no attribution run. The scripts (probe_late_binding.py, probe_gotchas.py, in the same currently-private fork as the main experiment's code) execute every snippet before loading the model, under the interpreter and environment each case names. They refuse to start unless the prompt is a true prefix of what Python prints, the candidate labelled correct is the actual next character, and the distractor is not. The complete per-token outputs behind every number on this page: probe_late_binding_gemma-2-2b.txt and probe_gotchas_gemma-2-2b.txt.

Credits

Companion page to Which variable does the model return? and Right name, wrong rules. Model google/gemma-2-2b, probed with plain transformers. The generator-expression material follows a teaching video on PEP 289; the gotcha cases are the community's classics. Ground truth is re-executed on every run rather than taken on authority.