The overlay does not work when the output is rotated (under randr12)
1) Start nouveau, using randr12
2) xrandr -o 2
3) mplayer foo.avi
Since rotation is commonly used for projectors (where there's a reasonable chance of Xv being used), this is not a made up problem :)
Someone fixing the coordinate-fu would be awesome, but falling back to the blitter is probably far far easier.
Very superficial investigation confirms the bug and suggests the overlay cannot do rotation.
Blitter fallback would appear as the only solution.
If you have time to test with the binary blob and see how it deals with rotation and overlay, the data would be greatly appreciated (mmiotrace).
From the nvidia README:
" Option "RandRRotation" "boolean"
Enable rotation support for the XRandR extension. This
allows use of the XRandR X server extension for
configuring the screen orientation through rotation.
This feature is supported on GeForce2 or better hardware
using depth 24. This requires an XOrg 6.8.1 or newer
X server. This feature does not work with hardware overlays,
emulated overlays will be used instead at a substantial
performance penalty. See APPENDIX W for details.
Testing with the nvidia driver confirms that even for the non-rotated case the overlay is not used (when the above option is set), so it would seem falling back to blitter is the answer.
Fixed in commit c8cec9b2 "randr12, xv: no overlay when the crtc is transformed"