Just tried running modetest, quitting it, and restarting it - that immediately jams:
# ./modetest -s 12:1920x1200 -v
trying to load module i915...success.
setting mode 1920x1200 on connector 12, crtc 4
select timed out or error (ret 0)
select timed out or error (ret 0)

and display stuck on the colourful screen.

Does the modetest trace look the same as the earlier X trace?  These messages:

[176995.150] (WW) intel(0): get vblank counter failed: Invalid argument
[176995.150] (WW) intel(0): first get vblank counter failed: Invalid argument

look like the kernel is rejecting vblank event requests for some reason.  A lack of space (i.e. unconsumed events) should result in an -ENOMEM return though; are there any messages in dmesg indicating why the call failed?

modetest never gives me a helpful trace - it's always stuck in select when it dies.

There's no messages in dmesg when things go wrong, to suggest why it's failing. It's just dead. I've also found a way to kill the system (no network, no local console) - run "./modetest -s 12:1920x1200 -v", leave it for a few seconds, and press enter to have it shut down nicely. Run it again, getting output like:
# ./modetest -s 12:1920x1200 -v
trying to load module i915...success.
setting mode 1920x1200 on connector 12, crtc 4
select timed out or error (ret 0)
select timed out or error (ret 0)
select timed out or error (ret 0)
select timed out or error (ret 0)
select timed out or error (ret 0)
select timed out or error (ret 0)
select timed out or error (ret 0)
select timed out or error (ret 0)
select timed out or error (ret 0)

With modetest running in one SSH session, run vbltest in another session. Watch the machine disappear out from under you - even magic SysRq is gone.

Some tidbits that might help:
 * To get identical frequency outputs from vbltest and modetest, I need to run vbltest -s.
 * If I run vbltest -s instead of vbltest in my "kill the world" setup, it doesn't die.
 * If I run vbltest -s while the first modetest is running, the second one succeeds.

I wonder if we're not requesting the right IRQs...

With the fixes in bug #28788 applied (apply all three patches, reverse the order of finish and prepare as in comment 34 on that bug), I no longer get hangs.

The patches are:
https://bugs.freedesktop.org/attachment.cgi?id=36463
https://bugs.freedesktop.org/attachment.cgi?id=36464
https://bugs.freedesktop.org/attachment.cgi?id=35551

And then, in https://bugs.freedesktop.org/attachment.cgi?id=36464 change the following hunk:
diff --git a/drivers/gpu/drm/i915/i915_irq.c b/drivers/gpu/drm/i915/i915_irq.c
index 2479be0..a846cd8 100644
--- a/drivers/gpu/drm/i915/i915_irq.c
+++ b/drivers/gpu/drm/i915/i915_irq.c
@@ -940,22 +940,30 @@ irqreturn_t i915_driver_irq_handler(DRM_IRQ_ARGS)
         if (HAS_BSD(dev) && (iir & I915_BSD_USER_INTERRUPT))
             DRM_WAKEUP(&dev_priv->bsd_ring.irq_queue);

-        if (iir & I915_DISPLAY_PLANE_A_FLIP_PENDING_INTERRUPT)
+        if (iir & I915_DISPLAY_PLANE_A_FLIP_PENDING_INTERRUPT) {
             intel_prepare_page_flip(dev, 0);
+            if (dev_priv->flip_pending_is_done)
+                intel_finish_page_flip_plane(dev, 0);
+        }

-        if (iir & I915_DISPLAY_PLANE_B_FLIP_PENDING_INTERRUPT)
+        if (iir & I915_DISPLAY_PLANE_B_FLIP_PENDING_INTERRUPT) {
+            if (dev_priv->flip_pending_is_done)
+                intel_finish_page_flip_plane(dev, 1);
             intel_prepare_page_flip(dev, 1);
+        }

         if (pipea_stats & vblank_status) {
             vblank++;
             drm_handle_vblank(dev, 0);
-            intel_finish_page_flip(dev, 0);
+            if (!dev_priv->flip_pending_is_done)
+                intel_finish_page_flip(dev, 0);
         }

         if (pipeb_stats & vblank_status) {
             vblank++;
             drm_handle_vblank(dev, 1);
-            intel_finish_page_flip(dev, 1);
+            if (!dev_priv->flip_pending_is_done)
+                intel_finish_page_flip(dev, 1);
         }

         if ((pipea_stats & I915_LEGACY_BLC_EVENT_STATUS) ||


to

diff --git a/drivers/gpu/drm/i915/i915_irq.c b/drivers/gpu/drm/i915/i915_irq.c
index 2479be0..a846cd8 100644
--- a/drivers/gpu/drm/i915/i915_irq.c
+++ b/drivers/gpu/drm/i915/i915_irq.c
@@ -940,22 +940,30 @@ irqreturn_t i915_driver_irq_handler(DRM_IRQ_ARGS)
         if (HAS_BSD(dev) && (iir & I915_BSD_USER_INTERRUPT))
             DRM_WAKEUP(&dev_priv->bsd_ring.irq_queue);

-        if (iir & I915_DISPLAY_PLANE_A_FLIP_PENDING_INTERRUPT)
+        if (iir & I915_DISPLAY_PLANE_A_FLIP_PENDING_INTERRUPT) {
             intel_prepare_page_flip(dev, 0);
+            if (dev_priv->flip_pending_is_done)
+                intel_finish_page_flip_plane(dev, 0);
+        }

-        if (iir & I915_DISPLAY_PLANE_B_FLIP_PENDING_INTERRUPT)
+        if (iir & I915_DISPLAY_PLANE_B_FLIP_PENDING_INTERRUPT) {
             intel_prepare_page_flip(dev, 1);
+            if (dev_priv->flip_pending_is_done)
+                intel_finish_page_flip_plane(dev, 1);
+        }

         if (pipea_stats & vblank_status) {
             vblank++;
             drm_handle_vblank(dev, 0);
-            intel_finish_page_flip(dev, 0);
+            if (!dev_priv->flip_pending_is_done)
+                intel_finish_page_flip(dev, 0);
         }

         if (pipeb_stats & vblank_status) {
             vblank++;
             drm_handle_vblank(dev, 1);
-            intel_finish_page_flip(dev, 1);
+            if (!dev_priv->flip_pending_is_done)
+                intel_finish_page_flip(dev, 1);
         }

         if ((pipea_stats & I915_LEGACY_BLC_EVENT_STATUS) ||

For reference, the first two patches and the correction to the second patch are included in 2.6.35-rc4 under the following commits:

83f7fd0 drm/i915: don't queue flips during a flip pending event
1afe3e9 drm/i915: gen3 page flipping fixes
70565d0 drm/i915: fix page flip finish vs. prepare on plane B

And the final patch is currently committed to drm-intel-next as the following:

f602afd drm/i915: Include instdone[1] in hangcheck