Bug 101372

Last known good version: 4.9.8
Software: Arch Linux, XFCE, compton, nouveau (using DRI3)
Hardware: NVIDIA Corporation GK107 [GeForce GT 740] (rev a1)

The behavior of this bug is identical to #97916 (https://bugs.freedesktop.org/show_bug.cgi?id=97916), but none of the workarounds work. Usually.

Quoting its behavior because they explained it way better than I could:
With default settings (no xorg.conf), dual monitors can only be used in "clone" mode.  When trying to switch to "extend" mode, the right-hand display freezes and continues to display a snapshot of the cloned display contents at the time of the switch.  Using Option "DRI" "2" in xorg.conf is a workaround.

And later on in the bug report, they discovered another workaround is making sure your compositor does what the --paint-on-overlay option makes compton do.

In my case, a few things are different:
1. I'm not running Ivy Bridge.
2. Switching to DRI2 does NOT fix it.
3. Enabling --paint-on-overlay fixed it the first time I tried it, even across multiple boots and tests, and then randomly stopped working for seemingly no reason. I had changed nothing about my system.
4. I started having this issue when I upgraded the kernel to 4.9.10 from 4.9.8, rather than when it started for them: 4.7.4. And yes, I already tested to confirm that this behavior changes between those two versions of Linux, and not with any other upgrade or downgrade to any of my other packages (not even nouveau, mesa, or xorg). You might ask why I'm making this bug report here, then, instead of at bugzilla.kernel.org. That's because of https://bugzilla.kernel.org/show_bug.cgi?id=195321#c5
5. When X and XFCE first run, this behavior doesn't  happen at all (assuming it wasn't happening before I shut it down). My dual-monitor setup still works properly. However, once I disconnect that monitor--or if it was disconnected when I rebooted, causing my system to remember that on boot--this behavior starts, and does not go away until I disable and enable the monitor in XFCE's Display preferences, which then restores the proper behavior until the next disconnect. It specifically has to be XFCE's Display preferences. Adding a regular, simple (using nothing but two Monitor sections and Identifiers matching the monitor names) dual-monitor .conf file to xorg.conf.d/ actually made it worse by causing this behavior to start immediately at boot no matter what, and for some reason xrandr couldn't turn the monitor back on after running xrandr --output <output> --off (though XFCE's preferences COULD turn it back on).
6. I still have this behavior when compton is disabled and I'm not running any compositor at all. I specifically have to either downgrade the kernel or run compton with the --paint-on-overlay option to fix it (when that even manages to fix it).

I have not yet tested a newer kernel version than 4.9.10, but I plan to when I have more time, probably in a week, but possibly in a month.