Created attachment 129367 [details]
fixed fbdev.c source file

Commit ae0aeffae6657464f610da13e7b3349d0d97e81b breaks a failsafe built into xserver/fbdevhw.c. When linux is run in virtualbox without using guest modules, the framebuffer is not a valid pci device but can be opened directly using /dev/fb0.

Both FBDevPciProbe and FBDevPreInit ultimately call either fbdev_pci_open or fbdev_open, depending on the value of the pci_device pointer. If the pointer is null, fbdev_open is called; if the pointer is not null, fbdev_pci_open is called. If the pci data is garbage and fbdev_pci_open is called, the operation fails, and I believe that's what's happening when running the OS in vbox without the vboxvideo driver. The failsafe occurs when you pass NULL to either FBDevPciProbe or FBDevPreInit. A NULL pci_device pointer parameter will cause both of these functions to call fbdev_open instead of fbdev_pci_open, and that's the key to the failsafe. fbdev_open first tries to acquire the name of the framebuffer device via the environment (dev = getenv("FRAMEBUFFER")) and if it's unable to get the device name, it makes a last ditch effort using /dev/fb0. That's how it worked when fbdev was unable to acquire valid pci data for the system graphics device (vbox without vboxvideo).

This can be fixed as follows:

FBDevPciProbe:

if (fbdevHWProbe(dev, device, NULL)) { 
    pScrn->driverVersion = FBDEV_VERSION;
    pScrn->driverName = FBDEV_DRIVER_NAME;
    pScrn->name = FBDEV_NAME;
    pScrn->Probe = FBDevProbe;
    pScrn->PreInit = FBDevPreInit;
    pScrn->ScreenInit = FBDevScreenInit;
    pScrn->SwitchMode = fbdevSwitchMode;
    pScrn->AdjustFrame = fbdevAdjustFrame;
    pScrn->EnterVT = fbdevEnterVT;
    pScrn->LeaveVT = fbdevLeaveVT;
    pScrn->ValidMode = fbdevValidMode;

    xf86DrvMsg(pScrn->scrnIndex, X_CONFIG,
        "claimed PCI slot %d@%d:%d:%d\n", 
        dev->bus, dev->domain, dev->dev, dev->func);
    xf86DrvMsg(pScrn->scrnIndex, X_INFO,
        "using %s\n", device ? device : "default device");
}
else if (fbdevHWProbe(NULL, device, NULL)) {
    pScrn->driverVersion = FBDEV_VERSION;
    pScrn->driverName = FBDEV_DRIVER_NAME;
    pScrn->name = FBDEV_NAME;
    pScrn->Probe = FBDevProbe;
    pScrn->PreInit = FBDevPreInit;
    pScrn->ScreenInit = FBDevScreenInit;
    pScrn->SwitchMode = fbdevSwitchMode;
    pScrn->AdjustFrame = fbdevAdjustFrame;
    pScrn->EnterVT = fbdevEnterVT;
    pScrn->LeaveVT = fbdevLeaveVT;
    pScrn->ValidMode = fbdevValidMode;

    xf86DrvMsg(pScrn->scrnIndex, X_INFO,
        "using %s\n", device ? device : "default device");
}
else {
    pScrn = NULL;
}

FBDevPreInit:

if (!fbdevHWInit(pScrn, pci_dev, xf86FindOptionValue(fPtr->pEnt->device->options, "fbdev")))
    if (!fbdevHWInit(pScrn, NULL, xf86FindOptionValue(fPtr->pEnt->device->options, "fbdev")))
        return FALSE;

I believe the FBDevPciProbe fix can be better coded with a little thought, but this does work to fix the issue. I've attached a fixed version of fbdev.c