This is moved over from https://trac.ffmpeg.org/ticket/5732, which contains somewhat more discussions and info. Long story short, when encoding H.264 video on a Haswell (with a specific, very challenging input), the hardware or driver produces a bitstream that's not spec-compliant and does not play well in libavcodec. However, it is somehow consistently wrong, because libva itself decodes it right.
You can find an example stream (big; 5.1 GB) at http://storage.sesse.net/through-the-cracks.mp4; the first instance corruption happens at 13:50. This is encoded at constant qp 15. There is a smaller, cut-out section (with audio removed) at http://storage.sesse.net/through-the-cracks-cut.mp4 if you just want to see that part.
The original encode was done using my live video mixer, Nageru, which contains a VA-API-based encoder derived from Intel's example H.264 code, running against version 0.7.0 of the Intel VA driver on a desktop Haswell. I can also reproduce this reliably with ffmpeg(1) against version 0.7.1, on my Haswell laptop, so I don't think it's an error of the driving application. I don't have access to Broadwell or Skylake to test.
To reproduce, you can use the uncut .mp4 and reencode it with ffmpeg(1) as follows:
ffmpeg -hwaccel vaapi -ss 12:00 -vaapi_device :0 -i through-the-cracks.mp4 -vf 'format=nv12,hwupload' -c:v h264_vaapi test.mp4 -qp 20
Note the use of -hwaccel vaapi to get VA-API _decode_ instead of libavcodec, so that the encoder has a correct bitstream to work with (as VA-API decodes its own stream correctly).