I noticed some console output:
% fretsonfire
radeon: The kernel rejected CS, see dmesg for more information.

% dmesg|grep radeon
[    1.201137] [drm] radeon defaulting to kernel modesetting.
[    1.201140] [drm] radeon kernel modesetting enabled.
[    1.201193] radeon 0000:01:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
[    1.201198] radeon 0000:01:00.0: setting latency timer to 64
[    1.201570] radeon 0000:01:00.0: VRAM: 512M 0x0000000000000000 - 0x000000001FFFFFFF (512M used)
[    1.201573] radeon 0000:01:00.0: GTT: 512M 0x0000000020000000 - 0x000000003FFFFFFF
[    1.201670] [drm] radeon: 512M of VRAM memory ready
[    1.201672] [drm] radeon: 512M of GTT memory ready.
[    1.201728] radeon 0000:01:00.0: irq 44 for MSI/MSI-X
[    1.201733] radeon 0000:01:00.0: radeon: using MSI.
[    1.201757] [drm] radeon: irq initialized.
[    1.207978] radeon 0000:01:00.0: WB enabled
[    1.254326] [drm] radeon: ib pool ready.
[    1.326794] [drm] radeon: power management initialized
[    1.421474] fbcon: radeondrmfb (fb0) is primary device
[    1.848199] fb0: radeondrmfb frame buffer device
[    1.848205] [drm] Initialized radeon 2.10.0 20080528 for 0000:01:00.0 on minor 0
[ 3447.796703] [drm:radeon_cs_ioctl] *ERROR* Failed to parse relocation -12!

Created attachment 50816 [details] [review]
game patch

The game tries to create the texture of maximum size for the glyph cache. I think it's not a good idea. It doesn't even start on my evergreen due to this. You might want to try attached patch for the game (for the "src/Font.py" in the game folder), it should fix this issue.

Though, it seems the driver doesn't handle this situation correctly too. I'll look into it.

Probably the driver can't do anything other than report the "out of memory" error (error code -12 in your dmesg log). The game just shouldn't use such huge textures, it shouldn't use MAX_TEXTURE_SIZE as the size for texture (this result in 0.4 GB on r6xx/r7xx, 1.4 GB on evergreen, next step will be >5 GB with future GPUs). I think it makes sense to tell the game developers about this issue. With my patch it uses 1024x1024 texture (~5 MB) and runs fine without artifacts or error messages in the logs (though I haven't tested it much). I don't know exact requirements for this, the game developers should know what size is really required.

the patch worked for me :)

But it's strange that fretsonfire work without problems with catalyst... maybe they set MAX_TEXTURE_SIZE to a different value.

(In reply to comment #4)
> the patch worked for me :)
> 
> But it's strange that fretsonfire work without problems with catalyst... maybe
> they set MAX_TEXTURE_SIZE to a different value.

I think the value of MAX_TEXTURE_SIZE is the same with catalyst, but probably it's using compression or some other tricks to reduce the size of the texture.

(In reply to comment #3)
> Probably the driver can't do anything other than report the "out of memory"
> error (error code -12 in your dmesg log). The game just shouldn't use such
> huge textures, it shouldn't use MAX_TEXTURE_SIZE as the size for texture
> (this result in 0.4 GB on r6xx/r7xx, 1.4 GB on evergreen, next step will be
> >5 GB with future GPUs). I think it makes sense to tell the game developers
> about this issue. With my patch it uses 1024x1024 texture (~5 MB) and runs
> fine without artifacts or error messages in the logs (though I haven't
> tested it much). I don't know exact requirements for this, the game
> developers should know what size is really required.

This explains it. Closing.