Created attachment 17547 [details] [review]
proposed patch for the issue

In nv_output:

error = RRChangeOutputProperty(output->randr_output, dithering_atom,
                                        XA_INTEGER, 32, PropModeReplace, 1, &nv_output->dithering,FALSE, FALSE);

&nv_output->dithering is a Boolean, but the property is set up as int32. On big endian systems, this results in the property set being 0x1000000 instead of 0x1. This currently fails the range check (it can only be 1 or 0). Leading to an error returned above which made me notice the issue.

Actually I think it may be an idea if X would check the type of the pointer passed to RRChangeOutputProperty and return a better error if it is not equal to the type (this problem would go unnoticed without the range check in nv_output's set_property function), thus I add a cc to keithp about that.

To fix the problem in nouveau, either the property should be set up as Bool, or we need to do an explicit cast. I provide a patch for the latter (because I am not sure if we want more than 2 states for this property in the future). Let me know if the former is preferred.


Cheers,

danny