Created attachment 15094 [details]
xlog randr12 modeset_trace

I assume the monitor is correctly detected as being attached on DVI-D? Assuming this is so, this is not expected to work currently on your particular card as a separate TMDS chip needs programming up, so there's going to be some work in getting it going.

Do the VGA and DVI-A connections work as expected?

Yes VGA and DVI-A works well. I tried only booting with the screen connected at boot, but no "hot plugging" and detection while xserver is runing.
Connected with DVI-D the screen stay on but shows only black. The driver detects the right mode.

If only the DVI output is used at boot, does it work with nouveau in non-randr12 mode or the nv driver? If it does:
git clone git://people.freedesktop.org/~airlied/radeontool
git checkout origin/nvidia
make

then run "./radeontool regs > logfile", both when in the working non-randr12 state, and (if possible) remotely over ssh when in the broken randr12 state, and attach the logs to this bug.

If nv / non-randr12 doesn't work, but console mode does, comparative working/non-working dumps would still be useful.

Created attachment 15481 [details]
radeontool regs output - different configs

dvi-d works with nv and nouveau no randr12 when the screen is connected from boot.
this is the dvi-d chip on the card. I'm not sure if I2C is connected but in the datasheet is a description of registers.
http://www.siliconimage.com/docs/SiI-DS-0021-E-164.pdf

I think something went wrong, that log is filled mostly with zeros and some random stuff.

While bringing up the SIL may be eventually interesting, we first ought to be able to bring up a display by not breaking the existing setup (as in the non-randr12 case) -- radeontool dumps should let us do this.

Unfortunately the radeontool dumps you provided are broken, as you have many nvidia pci devices, and radeontool is somewhat stupid :)

To fix this, change radeontool.c line 501 from
radeon_cntl_mem = map_devince_memory(base,0x700000);

to

radeon_cntl_mem = map_devince_memory(0xe4000000,0x700000);

recompile, and dump again (logfile_nouveau and log_nouv_r12 being the interesting ones)

Created attachment 15482 [details]
radeontool regs output - different configs

Please make the with-randr12 dump again, with the following change to nv_output.c:

diff --git a/src/nv_output.c b/src/nv_output.c
index 5841492..a4e1917 100644
--- a/src/nv_output.c
+++ b/src/nv_output.c
@@ -1113,6 +1113,7 @@ void NvSetupOutputs(ScrnInfoPtr pScrn)
                                sprintf(outputname, "DVI-A-%d", dvia_count++);
                        break;
                case OUTPUT_TMDS:
+                       pNv->dcb_table.entry[i].heads = 2;
                        sprintf(outputname, "DVI-D-%d", dvid_count++);
                        break;
                case OUTPUT_TV:

Created attachment 15486 [details] [review]
Possibility 1

A possible patch (given that the hack in the previous comment fixed it)

Created attachment 15487 [details] [review]
Possibility 2

A second option

Created attachment 15488 [details]
xlog randr12, regs

hack from comment 9 make it working

If you feel like having a poke at some registers, I've a couple of candidates -- CR59 and 0x6808c0. What this means in practice, is running *without* the comment 9 hack, and trying (via ssh) all four combinations of:

./radeontool regset C0:59 M        where M is 0 or 1
and
./radeontool regset 0x6808c0 N     where N is 0 or 0x100

It may not do much, but at least we can discount them afterwards :-)

I can't see any difference. The regs was 1 / 0x100.

Please test whether it now works with an unmodified copy of the latest code in git. If it does, testing whether changing between CRTC 0 and 1 with "xrandr --output DVI-D-0 --mode something --crtc n" works would be a good additional test.

Now it works :) Also switching the crtc. Thank you.