10/12/2010

10-12-10 - Image Comparison Part 1

First of all, let's talk about how we graph these things. For this comparo, I'm measuring RGB L2 error (aka MSE) using IJG JPEG. I'm comparing :

    jpeg : default settings (with -optimize)
    jpegflat : jpeg + flat quantization matrix
    jpegflatnosub : jpegflat + no subsample for chroma

what people usually plot is PSNR vs. bpp, which looks like this :

jpegs psnr vs bpp :

these psnr vs bpp graphs are total shit. I can't see anything. The area that you actually care about is around 0.5 - 1.5 bpp, and it's all crammed together. In particular it's impossible to tell if jpeg vs jpegflat is better, and I have no intuition for what the numbers mean. Please stop making these graphs right now.

(NOTE : bpp means bits per *pixel* not bits per byte; eg. uncompressed is 24 bpp)

IMO rmse is better than PSNR because it's more intuitive. 1 level of rmse is one pixel value, it's intuitive. Unfortunately, the plot is only slightly better :

jpegs rmse vs bpp :

What we want obviously is to expand the area around 1 bpp. The obvious thing should occur to - our bpp scale is wrong. In particular, what we really care about are doublings of bpp - eg. 0.25 bpp, 0.5 bpp, 1.0 bpp, 2.0 bpp - the step from 0.25 to 0.50 is about the same importance as the step from 4.0 bpp to 8.0 bpp. Obviously we should use a log scale. A similar argument applies to rmse, so we have :

jpegs log rmse vs log bpp :

which is much clearer. It also is amazingly linear through the "money zone" of -1 to 1 (0.5 to 2.0 bpp) which is where JPEG performs well.

BTW note of course PSNR is a log scale as well, it's just log rmse flipped upside down and then offset all weird by some constant. I don't like it as much, but PSNR vs. log bpp is okay :

PSNR vs log bpp :

Conclusion :

Plots vs log bpp are the way to go.

If you are showing L2 errors for JPEG you need to be using a flat quantization matrix with no subsampling of chroma. (note that I didn't optimize the relative scaling of the planes, which might change the results or improve them further).

Next post I'll move on to some perceptual measurements.

(BTW the JPEG numbers posted here are all with PAQ ; see later posts for full details)

No comments:

old rants