When calling drmModeDirtyFB on linux, it fails when num_clips are more than DRM_MODE_FB_DIRTY_MAX_CLIPS (value: 256). But there is no way to check this for xf86-video-* drivers, because when including drm_mode.h, you get the drm_mode.h from /usr/include/libdrm/drm_mode.h and not /usr/include/drm/drm_mode.h.

Please add

#define DRM_MODE_FB_DIRTY_MAX_CLIPS 256 

to your drm_mode.h, to allow xf86-video-* drivers to check parameters before calling drmModeDirtyFB(...).