I can confirm this problem also for Kernel 4.3.3:

commit 35483418917d63df90bae5b2d0b7b047d7ed8ec7
Author: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
Date:   Mon Dec 14 21:41:43 2015 -0800

    Linux 4.3.3

running nouveau with:
VGA compatible controller: NVIDIA Corporation G84 [GeForce 8600 GTS] (rev a1)

when starting firefox (44.04b):

 firefox: page allocation failure: order:5, mode:0x2040d0
 CPU: 2 PID: 12962 Comm: firefox Tainted: G           O    4.3.3+ #13
 Hardware name: ASUSTEK COMPUTER INC P5W DH Deluxe/P5W DH Deluxe, BIOS 3002    07/22/2010
  0000000000000000 ffff8800a38ef6a8 ffffffff8122dda4 ffff8801bfd0f018
  00000000002040d0 ffff8800a38ef738 ffffffff811164db 00000000000007c0
  ffffffff81629a38 ffff880000000042 0000000000000286 0000000000000008
 Call Trace:
  [<ffffffff8122dda4>] dump_stack+0x4c/0x78
  [<ffffffff811164db>] warn_alloc_failed+0xdb/0x130
  [<ffffffff811194b3>] __alloc_pages_nodemask+0x713/0x800
  [<ffffffff81150f51>] cache_alloc_refill+0x341/0x5f0
  [<ffffffff8115173d>] __kmalloc+0x10d/0x230
  [<ffffffffa09d6511>] ? nvkm_ramht_new+0x41/0x100 [nouveau]
  [<ffffffffa09d6511>] nvkm_ramht_new+0x41/0x100 [nouveau]
  [<ffffffffa0a415af>] g84_fifo_chan_ctor+0x14f/0x180 [nouveau]
  [<ffffffffa0a433ac>] g84_fifo_gpfifo_new+0xdc/0x320 [nouveau]
  [<ffffffffa0a2c8a3>] ? nvkm_disp_class_get+0x33/0x70 [nouveau]
  [<ffffffffa0a3ad57>] nvkm_fifo_class_new+0x17/0x20 [nouveau]
  [<ffffffffa0a2bf96>] nvkm_udevice_child_new+0x26/0x30 [nouveau]
  [<ffffffffa09d3a02>] nvkm_ioctl_new+0x132/0x2a0 [nouveau]
  [<ffffffffa0a2bf70>] ? nvkm_udevice_map+0x60/0x60 [nouveau]
  [<ffffffffa09d4160>] nvkm_ioctl+0x150/0x250 [nouveau]
  [<ffffffffa0a701a2>] nvkm_client_ioctl+0x12/0x20 [nouveau]
  [<ffffffffa09d1043>] nvif_object_ioctl+0x43/0x50 [nouveau]
  [<ffffffffa09d15e4>] nvif_object_init+0xd4/0x140 [nouveau]
  [<ffffffffa0a8632b>] nouveau_channel_new+0xbb/0x680 [nouveau]
  [<ffffffffa0a85523>] nouveau_abi16_ioctl_channel_alloc+0x123/0x340 [nouveau]
  [<ffffffffa03358ad>] drm_ioctl+0x2ad/0x4a0 [drm]
  [<ffffffff81111297>] ? unlock_page+0x27/0x30
  [<ffffffffa0a85400>] ? nouveau_abi16_ioctl_setparam+0x10/0x10 [nouveau]
  [<ffffffffa0a6e527>] nouveau_drm_ioctl+0x67/0xd0 [nouveau]
  [<ffffffff81170b53>] do_vfs_ioctl+0x83/0x500
  [<ffffffff810414c4>] ? __do_page_fault+0x184/0x390
  [<ffffffff8117101c>] SyS_ioctl+0x4c/0x90
  [<ffffffff814178d7>] entry_SYSCALL_64_fastpath+0x12/0x6a
 Mem-Info:
 active_anon:64168 inactive_anon:104618 isolated_anon:0
  active_file:212131 inactive_file:200223 isolated_file:0
  unevictable:0 dirty:19221 writeback:0 unstable:0
  slab_reclaimable:859742 slab_unreclaimable:17819
  mapped:26813 shmem:100 pagetables:5377 bounce:0
  free:38731 free_pcp:79 free_cma:0
 DMA free:15908kB min:24kB low:28kB high:36kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
 lowmem_reserve[]: 0 2984 5947 5947
 DMA32 free:71412kB min:4944kB low:6180kB high:7416kB active_anon:130456kB inactive_anon:208916kB active_file:425800kB inactive_file:400368kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3128832kB managed:3056324kB mlocked:0kB dirty:39152kB writeback:0kB mapped:53632kB shmem:164kB slab_reclaimable:1729508kB slab_unreclaimable:32416kB kernel_stack:3216kB pagetables:10476kB unstable:0kB bounce:0kB free_pcp:180kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:260 all_unreclaimable? no
 lowmem_reserve[]: 0 0 2962 2962
 Normal free:67604kB min:4908kB low:6132kB high:7360kB active_anon:126216kB inactive_anon:209556kB active_file:422724kB inactive_file:400524kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3145728kB managed:3033980kB mlocked:0kB dirty:37732kB writeback:0kB mapped:53620kB shmem:236kB slab_reclaimable:1709460kB slab_unreclaimable:38860kB kernel_stack:4160kB pagetables:11032kB unstable:0kB bounce:0kB free_pcp:136kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:260 all_unreclaimable? no
 lowmem_reserve[]: 0 0 0 0
 DMA: 1*4kB (U) 0*8kB 0*16kB 1*32kB (U) 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15908kB
 DMA32: 8515*4kB (UEM) 4254*8kB (UEM) 221*16kB (UEM) 9*32kB (EM) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 71916kB
 Normal: 13336*4kB (UEM) 1687*8kB (UEM) 37*16kB (UEM) 18*32kB (M) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 68008kB
 417445 total pagecache pages
 4935 pages in swap cache
 Swap cache stats: add 682285, delete 677350, find 976311/1124043
 Free swap  = 24652224kB
 Total swap = 25165812kB
 1572638 pages RAM
 0 pages HighMem/MovableOnly
 46085 pages reserved
 0 pages hwpoisoned

Same here on kernel 4.4.0-rc7, NV86 (GeForce 8500 GT). Right now even glxgears produces the error:

[160746.465409] glxgears: page allocation failure: order:5, mode:0x26040c0
[160746.465415] CPU: 0 PID: 18218 Comm: glxgears Not tainted 4.4.0-rc7-nouveau #1
[160746.465416] Hardware name: . ./I-45C(Intel i945GC-ICH7), BIOS 6.00 PG 09/18/2008
[160746.465418]  0000000000000000 000000004a0928fb ffffffff811f4533 00000000026040c0
[160746.465422]  ffffffff810d6eb5 ffff880000000042 0000000000000002 026240c0ffffffe0
[160746.465424]  ffffffff815797a0 0000000000000020 000000004a0928fb 0000000000000002
[160746.465426] Call Trace:
[160746.465434]  [<ffffffff811f4533>] ? dump_stack+0x40/0x5d
[160746.465439]  [<ffffffff810d6eb5>] ? warn_alloc_failed+0xf5/0x150
[160746.465442]  [<ffffffff810d9533>] ? __alloc_pages_nodemask+0x163/0x820
[160746.465446]  [<ffffffff8110661d>] ? cache_alloc_refill+0x2ed/0x530
[160746.465448]  [<ffffffff81106a80>] ? __kmalloc+0x90/0xd0
[160746.465479]  [<ffffffffa0144efb>] ? nvkm_ramht_new+0x3b/0xe0 [nouveau]
[160746.465495]  [<ffffffffa01ab83a>] ? g84_fifo_chan_ctor+0x13a/0x170 [nouveau]
[160746.465508]  [<ffffffffa01ad349>] ? g84_fifo_gpfifo_new+0xb9/0x2e0 [nouveau]
[160746.465521]  [<ffffffffa01977f0>] ? nvkm_udevice_child_get+0x70/0x110 [nouveau]
[160746.465530]  [<ffffffffa01427e7>] ? nvkm_ioctl_new+0x147/0x290 [nouveau]
[160746.465543]  [<ffffffffa0197760>] ? nvkm_udevice_map+0x40/0x40 [nouveau]
[160746.465552]  [<ffffffffa0142e17>] ? nvkm_ioctl+0xf7/0x240 [nouveau]
[160746.465565]  [<ffffffffa01ed198>] ? nouveau_channel_prep+0x1b8/0x2c0 [nouveau]
[160746.465573]  [<ffffffffa0140685>] ? nvif_object_init+0xb5/0x120 [nouveau]
[160746.465584]  [<ffffffffa01ed358>] ? nouveau_channel_new+0xb8/0x680 [nouveau]
[160746.465595]  [<ffffffffa01ec483>] ? nouveau_abi16_ioctl_channel_alloc+0xd3/0x2d0 [nouveau]
[160746.465603]  [<ffffffffa0068618>] ? drm_copy_field+0x38/0x50 [drm]
[160746.465607]  [<ffffffffa006825c>] ? drm_ioctl+0x12c/0x4b0 [drm]
[160746.465618]  [<ffffffffa01ec3b0>] ? nouveau_abi16_ioctl_setparam+0x10/0x10 [nouveau]
[160746.465629]  [<ffffffffa01d618b>] ? nouveau_drm_ioctl+0x5b/0xb0 [nouveau]
[160746.465641]  [<ffffffffa01d845b>] ? nouveau_compat_ioctl+0xb/0x20 [nouveau]
[160746.465644]  [<ffffffff811501a1>] ? compat_SyS_ioctl+0xb1/0x1110
[160746.465646]  [<ffffffff8110e48b>] ? vfs_getattr_nosec+0x2b/0x30
[160746.465648]  [<ffffffff8110e638>] ? vfs_fstat+0x28/0x50
[160746.465651]  [<ffffffff810026a1>] ? do_fast_syscall_32+0x91/0x140
[160746.465655]  [<ffffffff813d5322>] ? sysenter_flags_fixed+0x8/0x12
[160746.465656] Mem-Info:
[160746.465660] active_anon:32765 inactive_anon:43188 isolated_anon:0
[160746.465660]  active_file:262031 inactive_file:229737 isolated_file:0
[160746.465660]  unevictable:0 dirty:1 writeback:0 unstable:0
[160746.465660]  slab_reclaimable:76470 slab_unreclaimable:6333
[160746.465660]  mapped:28713 shmem:1115 pagetables:919 bounce:0
[160746.465660]  free:174183 free_pcp:96 free_cma:0
[160746.465668] DMA free:13032kB min:32kB low:40kB high:48kB active_anon:64kB inactive_anon:460kB active_file:1128kB inactive_file:564kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15992kB managed:15908kB mlocked:0kB dirty:0kB writeback:0kB mapped:608kB shmem:84kB slab_reclaimable:116kB slab_unreclaimable:64kB kernel_stack:16kB pagetables:4kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[160746.465669] lowmem_reserve[]: 0 3251 3251 3251
[160746.465676] DMA32 free:683700kB min:7280kB low:9100kB high:10920kB active_anon:130996kB inactive_anon:172292kB active_file:1046996kB inactive_file:918384kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:3390396kB managed:3330656kB mlocked:0kB dirty:4kB writeback:0kB mapped:114244kB shmem:4376kB slab_reclaimable:305764kB slab_unreclaimable:25268kB kernel_stack:2896kB pagetables:3672kB unstable:0kB bounce:0kB free_pcp:384kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:640 all_unreclaimable? no
[160746.465677] lowmem_reserve[]: 0 0 0 0
[160746.465680] DMA: 6*4kB (UME) 2*8kB (ME) 2*16kB (UM) 3*32kB (UM) 1*64kB (M) 2*128kB (UE) 3*256kB (UME) 3*512kB (UME) 2*1024kB (ME) 2*2048kB (UE) 1*4096kB (M) = 13032kB
[160746.465692] DMA32: 67543*4kB (UME) 31650*8kB (UME) 7187*16kB (UME) 1124*32kB (UME) 149*64kB (UM) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 683868kB
[160746.465700] 493070 total pagecache pages
[160746.465702] 164 pages in swap cache
[160746.465704] Swap cache stats: add 11080, delete 10916, find 520799/521195
[160746.465705] Free swap  = 3695116kB
[160746.465706] Total swap = 3719040kB
[160746.465707] 851597 pages RAM
[160746.465708] 0 pages HighMem/MovableOnly
[160746.465709] 14956 pages reserved

I have seen this one as well on my G96 and MCP79, running 4.3.3. It seems to happen once RAM has been filled (should it be with buffers kept around or by applications currently running), even if SWAP is completely empty.

Created attachment 121071 [details] [review]
Use vzalloc instad of kzalloc in nvkm_ramht_new()

I've had lots of success using the attached patch. 4 days and still running strong, no page allocation failures. But, some core developer should probably confirm that using vmalloc instead of kmalloc is ok in that function.

I've run into this exact same failure on Linux kernel 4.4.7-gentoo.

01:00.0 VGA compatible controller: NVIDIA Corporation G84 [GeForce 8600 GT] (rev a1)

systemmonitor: page allocation failure: order:5, mode:0x240c0c0
CPU: 2 PID: 25678 Comm: systemmonitor Not tainted 4.4.7-gentoo #1
Hardware name:                  /D975XBX2, BIOS BX97520J.86A.2836.2008.0728.1946 07/28/2008
 0000000000000000 ffffffff815e2614 000000000240c0c0 ffff88010005b970
 ffffffff810bbc14 ffff880193810000 0000000000000000 ffffffff81a7cbc0
 0000000000000005 0000000000000040 ffff880193810000 000000000240c0c0
Call Trace:
 [<ffffffff815e2614>] ? dump_stack+0x46/0x59
 [<ffffffff810bbc14>] ? warn_alloc_failed+0xd4/0x120
 [<ffffffff810be22f>] ? __alloc_pages_nodemask+0x16f/0x910
 [<ffffffff810bebb7>] ? alloc_kmem_pages+0x17/0x80
 [<ffffffff810d142f>] ? kmalloc_order+0xf/0x40
 [<ffffffff8135dfcb>] ? nvkm_ramht_new+0x3b/0xe0
 [<ffffffff813bbc8a>] ? g84_fifo_chan_ctor+0x13a/0x170
 [<ffffffff813bd739>] ? g84_fifo_gpfifo_new+0xb9/0x2e0
 [<ffffffff813a8190>] ? nvkm_udevice_child_get+0x60/0xf0
 [<ffffffff8135b99d>] ? nvkm_ioctl_new+0x11d/0x260
 [<ffffffff813a8110>] ? nvkm_udevice_map+0x40/0x40
 [<ffffffff8135bfb7>] ? nvkm_ioctl+0xf7/0x240
 [<ffffffff813599d5>] ? nvif_object_init+0xb5/0x120
 [<ffffffff813fb828>] ? nouveau_channel_new+0xa8/0x660
 [<ffffffff813fa9f3>] ? nouveau_abi16_ioctl_channel_alloc+0xd3/0x2d0
 [<ffffffff8135a1c0>] ? nvkm_client_notify+0x20/0x20
 [<ffffffff813347d9>] ? drm_ioctl+0x119/0x480
 [<ffffffff810e315a>] ? page_add_file_rmap+0x2a/0x50
 [<ffffffff813fa920>] ? nouveau_abi16_ioctl_setparam+0x10/0x10
 [<ffffffff813e4f9b>] ? nouveau_drm_ioctl+0x5b/0xb0
 [<ffffffff811112f3>] ? do_vfs_ioctl+0x293/0x470
 [<ffffffff81033c69>] ? __do_page_fault+0x169/0x380
 [<ffffffff81111506>] ? SyS_ioctl+0x36/0x70
 [<ffffffff815e8a1b>] ? entry_SYSCALL_64_fastpath+0x16/0x6e
Mem-Info:
active_anon:254439 inactive_anon:246298 isolated_anon:0
 active_file:689086 inactive_file:691252 isolated_file:0
 unevictable:44 dirty:24538 writeback:0 unstable:0
 slab_reclaimable:50180 slab_unreclaimable:9054
 mapped:71902 shmem:9819 pagetables:6004 bounce:0
 free:28497 free_pcp:31 free_cma:0
DMA free:15828kB min:20kB low:24kB high:28kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15928kB managed:15844kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:16kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
lowmem_reserve[]: 0 3234 7957 7957
DMA32 free:37400kB min:4576kB low:5720kB high:6864kB active_anon:312136kB inactive_anon:457768kB active_file:1151848kB inactive_file:1153596kB unevictable:120kB isolated(anon):0kB isolated(file):0kB present:3389592kB managed:3314852kB mlocked:120kB dirty:26704kB writeback:0kB mapped:122180kB shmem:15400kB slab_reclaimable:81552kB slab_unreclaimable:13116kB kernel_stack:2544kB pagetables:9160kB unstable:0kB bounce:0kB free_pcp:4kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 4722 4722
Normal free:60760kB min:6684kB low:8352kB high:10024kB active_anon:705620kB inactive_anon:527424kB active_file:1604496kB inactive_file:1611412kB unevictable:56kB isolated(anon):0kB isolated(file):0kB present:4980736kB managed:4836224kB mlocked:56kB dirty:71448kB writeback:0kB mapped:165428kB shmem:23876kB slab_reclaimable:119168kB slab_unreclaimable:23084kB kernel_stack:4976kB pagetables:14856kB unstable:0kB bounce:0kB free_pcp:120kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
lowmem_reserve[]: 0 0 0 0
DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 1*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (U) 3*4096kB (M) = 15828kB
DMA32: 8708*4kB (UME) 334*8kB (UME) 17*16kB (UME) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 37776kB
Normal: 15206*4kB (UME) 3*8kB (U) 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 60848kB
1390493 total pagecache pages
276 pages in swap cache
Swap cache stats: add 8903, delete 8627, find 314/531
Free swap  = 8354200kB
Total swap = 8388604kB
2096564 pages RAM
0 pages HighMem/MovableOnly
54834 pages reserved


(In reply to Zlatko Calusic from comment #4)
> I've had lots of success using the attached patch. 4 days and still running
> strong, no page allocation failures. But, some core developer should
> probably confirm that using vmalloc instead of kmalloc is ok in that
> function.

I would venture a guess that the memory allocated for a GPU FIFO buffer needs to be physically continuous, as it will be used for DMA, so using vmalloc is a bad idea that may lead to incorrect pages being read/written by DMA.

Aside: Does this bug report belong on this bug tracker, or would it be more appropriate on the kernel bug tracker? This is a problem in the kernel, not in the DDX.

(In reply to Matt Whitlock from comment #5)
> I would venture a guess that the memory allocated for a GPU FIFO buffer
> needs to be physically continuous, as it will be used for DMA

I take it back. Looking into the source code, the allocation is happening in nvkm_ramht_new(…), which is allocating a struct nvkm_ramht containing a variably-sized array of struct nvkm_ramht_data elements. As these elements, in turn, contain pointers, it seems very unlikely that they'd be DMA'd, so presumably the struct nvkm_ramht doesn't need to be physically contiguous and could safely be allocated by vzalloc(…).

FYI: Ilia Mirkin submitted the patch from Comment 4 to the Nouveau listserv on 6 Mar 2016. I think I'll apply this patch to my 4.4.7 kernel.

http://permalink.gmane.org/gmane.comp.freedesktop.xorg.nouveau/24081

This should be now upstream with

commit 78a121d82da8aff3aca2a6a1c40f5061081760f0
Author: Ilia Mirkin <imirkin@alum.mit.edu>
Date:   Sun Mar 6 16:06:06 2016 -0500

    drm/nouveau/core: use vzalloc for allocating ramht
    
    Most calls to nvkm_ramht_new use 0x8000 as the size. This results in a
    fairly sizeable chunk of memory to be allocated, which may not be
    available with kzalloc. Since this is done fairly rarely (once per
    channel), use vzalloc instead.
    
    Signed-off-by: Ilia Mirkin <imirkin@alum.mit.edu>
    Signed-off-by: Ben Skeggs <bskeggs@redhat.com>

Zlatko - I didn't realize (or perhaps totally forgot) you had provided the same patch, didn't mean to steal credit or anything like that. Just ran into the same issue myself and sent a patch. In the future, feel free to send patches to the mailing list.

This issue should be resolved in v4.6-rc1+.