I have been facing an issue with my 5K display (iiyama ProLite XB2779QQS-S1).
It works fine as long as it is the only active monitor, as soon as I activate another monitor the main one (5k) can't display more than 4k.
Debug using "echo 0x4 > /sys/module/drm/parameters/debug" show this :
mai 23 09:01:22 bureau-gael /usr/lib/gdm3/gdm-x-session: (EE) AMDGPU(0): failed to set mode: Invalid argument
mai 23 09:01:22 bureau-gael kernel: [drm:dce112_validate_bandwidth [amdgpu]] dce112_validate_bandwidth: Bandwidth validation failed!
I disabled the check by forcing is_display_configuration_supported to return true in dce_calcs.c and it works fine.
I tried a lot of kernel (includig drm-tip and 5.2-rc1) and they all have the same issue.
My custom kernel with no validation is based on Ubuntu's 5.0.0-15-generic.
If I can help by providing more logs or test something ask and I will do my best.