Created attachment 136727 [details]
--scale --transform etc fail when changed in a specific way

Whenever scale value is increased when it is between 0.0 - 1.0
and is kept within 0.0 - 1.0 range, the display does not expand
to fullscreen. It rather becomes smaller and leaves blank spaces on
the right and bottom sides. And the following error message shows up:

X Error of failed request: BadMatch (invalid parameter attributes)
  Major opcode of failed request: 139 (RANDR)
  Minor opcode of failed request: 7 (RRSetScreenSize)
  Serial number of failed request: 36
  Current serial number in output stream: 38

Before xrandr attempts to apply the transformation matrix requested
by "scale", "transform" or "scale-from" options, it disables the
CRTCs for which the transformed image's dimensions are larger than the
target display's resolution/mode. But, when the image is smaller, it
does not disable the CRTCs, and proceeds with applying the
transformation (involves framebuffer dimension changes). This works well
for all the scale values and transitions except for the above-mentioned
transitions.

Also, see the attached png where I have represented the problem in a state diagram. Green represents successful, whereas Red represents failed state transition.

I have submitted a patch for this issue as well (https://lists.x.org/archives/xorg-devel/2017-December/055482.html), but couldn't hear from anyone for a while, therefore reported it here.

I have described in that patch on how to reproduce the issue.

Build 2017-12-21 on Yocto Linux & on Ubuntu 16.04.3 LTS