While investigating the Xv overlay & blitter issue with the X-server backing store, I found the real overlay jerky and tearing.
I have to disable the Composite extension of the X-server to get the real overlay adapter. Otherwise, due to the backing store issue, it falls back to blitter (which I hear is then still drawing the colorkey when it should not, but that's another issue). If Composite extension is enabled, blitter and overlay adapters are identical with mplayer, producing a static horizontal tear line (bug #16231).
Having Composite disabled, the blitter seems to work fine. Overlay looks jerky and occasionally tears at random places, it looks mostly horizontal tear lines, but they blink too fast to see clearly. It might also be blocky instead of just a line.
Practically my whole graphics stack is from git, around 20th-22nd June.
The card is nv20, a Geforce 3.
Just adding that I do not experience this tearing without composite on my nv34.
Not all videos exhibit the problem. Several 768x576 sized videos do not have problem, but 1024x576 does. I thought the scaling factor would also affect, but it might be an illusion: problems are better visible in a bigger view. Enlarged, native size, or shrunk, makes probably no difference. I'm not so sure it's random, either, but I'm not near the computer to test.
Closing as a very old report. If tearing somewhere still appears, I'll file a new bug or reopen this one. Current status is unknown.