You need -profile:v 578.

x11grab on my old CPU at least is not the best way to go.
ffmpeg is slow doing the BGR0 -> nv12 conversion, and by default it will be 601 rather than 709.
There are ways around that, and to be a bit faster with some convoluted commands.

I don't know what versions of everything you need, but you can "now" get the GPU to do the conversion and zero copy input, which is way faster and does BGR0 -> rec 709 CSC.

The downside is you need to be root or suid the ffmpeg binary.

1920x1080 is also slightly problematic but using mp4 or mkv will hide it (I guess depending on player). mkv seems to be broken with current ffmpeg git, bah. The issue being that eg. .ts would be seen as 1088 rather than 1080.

To get GPU accel using the newer/faster way would be something like -

ffmpeg -framerate 60 -device /dev/dri/card0 -f kmsgrab -i - -vsync 0 -init_hw_device vaapi=v:/dev/dri/renderD128 -filter_hw_device v -vf 'hwmap,scale_vaapi=format=nv12' -c:v h264_vaapi -profile:v 578 -bf 0 out.mp4

Thank you, -profile:v 578 works fine.

Looks like I had everything needed for the new way with kmsgrab too, which is even better!

Thank you so much for the tip and explanation, much appreciated!

Resolving as NOTABUG as this was a configuration problem/ignorance on my part.

it's -profile v: constrained_baseline --level 3.1 or 3.0 on newer ffmpeg versions.

Even with that I'm having corrupted encoding so I'll open a new ticket.

As a favour can you test your hardware encoded videos with totem (it uses gstreamer) and see if they load. vlc says the first 2 frames don't work but plays, and mpv can play it fine. Hardware players also bork at the encoded content. This is the ffmeg command line I'm using...

GST_DEBUG=3 reports this:
qtdemux qtdemux.c:9719:qtdemux_parse_trak:<qtdemux0> Track shorter than 20% (2881879/24000 vs. 5834998/1000) of the stream found, assuming preview image or something; skipping track

ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128 -hwaccel_output_format vaapi -ss 120 -t 120 -i a-movie.mkv -map 0:0 -map 0:1 -c:v h264_vaapi -b:v 4000k -qp 20 -bf 0 -profile:v 578 (or constrained_baseline in ffmpeg-git) -movflags +faststart -quality:v 0 -level:v 3.1 -coder:v cavlc -c:a aac -ab 128k -ar 48000 -ac 2 vaapitest.mp4

Here's the full specs on the created ffmpeg file using ffprobe:

https://pastebin.com/TRVM9XiH

See #105277 for more info. People had trouble getting encoding to start working on this ticket - not a bug - but when it works the data may be corrupt and not decodable by qtdemux in gstreamer and other hardware players.

Also vaapi may have trouble decoding what it creates itself - at least in vlc it does not mpv ;)

I'm still using ffmpeg 3.4.1 with -profile:v 578 and Totem plays the videos encoded fine here.

(In reply to Luke McKee from comment #3)

> ffmpeg -hwaccel vaapi -hwaccel_device /dev/dri/renderD128
> -hwaccel_output_format vaapi -ss 120 -t 120 -i a-movie.mkv -map 0:0 -map 0:1
> -c:v h264_vaapi -b:v 4000k -qp 20 -bf 0 -profile:v 578 (or
> constrained_baseline in ffmpeg-git) -movflags +faststart -quality:v 0
> -level:v 3.1 -coder:v cavlc -c:a aac -ab 128k -ar 48000 -ac 2 vaapitest.mp4

Not related to any issue as such - and I don't have the same h/w to test, but some thoughts related to the command line.

-b:v 4000k -qp 20

I would think one or the other, not both ie. you want constant bitrate or you want constant qp (= highly variable bitrate). I didn't test what ffmpeg takes this to mean.

-coder:v cavlc

I don't think this is hooked up - you will probably get cabac or calvlc depending on h/w.

-quality:v 0

I don't thing this will do anything either on VCE (I don't know anything about the newer engines)

NOTABUG?

IMHO, it's a documentation issue, *at least*. Please add the info to https://wiki.freedesktop.org/xorg/RadeonFeature/ or somewhere else.

Also, why is it possible to use radeonsi/vaapi for encoding that isn't actually supported? Isn't it possible to return a fail value? It shouldn't be the software *using* vaapi being responsible to check if the hardware supports stuff, or am I wrong?

(In reply to grmat from comment #7)
> NOTABUG?
> 
> IMHO, it's a documentation issue, *at least*. Please add the info to
> https://wiki.freedesktop.org/xorg/RadeonFeature/ or somewhere else.
> 
> Also, why is it possible to use radeonsi/vaapi for encoding that isn't
> actually supported? Isn't it possible to return a fail value? It shouldn't
> be the software *using* vaapi being responsible to check if the hardware
> supports stuff, or am I wrong?

No, that is perfectly correct. VA-API advises to the application what the hardware can do or can't do.

When the application choose to ignore that it is a problem in the application and not the driver.

Thanks, that's what I was (In reply to Christian König from comment #8)
> 
> No, that is perfectly correct. VA-API advises to the application what the
> hardware can do or can't do.
> 
> When the application choose to ignore that it is a problem in the
> application and not the driver.

Thanks, that's what I was asking. So it's a ffmpeg issue, they should automatically select the correct profile for the vaapi_device.

It would still be good to have it documented on RadeonFeature, IMHO.