Created attachment 27928 [details] [review]
patch to fix YUV scaling and use Rec709 coefficients

radeonhd uses certain constants in the conversion from YUV to RGB in the Xv code.
However, there's a minor scaling bug where 256 is used instead of 255 as the maximum value of an 8-bit unsigned.

In addition, the entries of the matrix A used to perform the following computation
[Y; Cb; Cr] = A * [R; G; B]
should probably be those given in Rec 709 (see http://en.wikipedia.org/wiki/Rec._709).

I've attached a patch which does this.

I determined the Y_mul, Y_shift, C_mul, and C_shift values using the equation given in the code comments, N' = N * N_mul + N_shift. For Y, N' is in the range [0,1], but for C, it's in the range [-0.5,0.5].
Here's an example for Y:
When N = 16/255, we should have N' = 0. When N = 235/255, we should have N' = 1. The constants Y_mul and Y_shift are easily obtained by solving the system of two equations. Here's a script for Maxima (http://maxima.sourceforge.net/) that computes the coefficients:

---

range_min: 0;
range_max: 1;

value_min: 16;
value_max: 235;

A: matrix([value_min/255,1],[value_max/255,1]);
invert(A) . matrix([range_min],[range_max]);

----

Running this script (maxima -b path-to-script-file) gives 255/219 and -16/219, which are Y_mul and Y_shift. To obtain the values for C_mul and C_shift, we just need to change range_min to -0.5, range_max to 0.5, and value_max to 240.

In order to determine the entries of the matrix A used for the YUV to RGB conversion, I used the following equations (from http://en.wikipedia.org/wiki/YCbCr):

Y' = Kr * R'    + (1 - Kr - Kb) * G' + Kb * B'
Pb = 0.5 * (B' - Y') / (1 - Kb)
Pr = 0.5 * (R' - Y') / (1 - Kr)
....................................................
R', G', B' in [0; 1]
Y' in [0; 1]
Pb in [-0.5; 0.5]
Pr in [-0.5; 0.5]

and http://en.wikipedia.org/wiki/Rec._709 gives Kb = 0.0722, Kr = 0.2126. We simply invert these equations to obtain the coefficients we want.

Without this patch, the white level (contrast control) test pattern from AVS HD 709 (explained at http://www.avsforum.com/avs-vb/showthread.php?t=948496 , MP4 version available at http://www.sendspace.com/file/kaotq5) was rather different in mplayer -vo x11 (which does software YUV->RGB conversion) compared to mplayer -vo xv. After the patch, they're pretty much indistinguishable (and it looks just like it does on my PS3).

I should probably mention that a lot of the bars from the test patterns don't flash on my PC monitors. I have my radeonhd box hooked up to an HDTV, where they do flash (at least after applying the patch.)