If you really want the system to remain usable while you're thrashing all of your memory, you're going to need to mlock not just the driver, but the entire server and all of the clients you care about.  Moving to Server/general.

I'm not talking about slowness because of thrashing. I'm talking about *hard lockups*.

So I've tested some more.

- I've been wrong suspecting hard lockups; if I hit ctl-alt-f1, *and then wait 5-10 minutes*, I get at the console and can kill the compilers and get back to the running X session and be fine.

- *But* if I run the vesa driver instead of the nv driver, there's no problem at all, X is unresponsive only for about 5 seconds when first hitting swap and then every now and then, and I can close the window running the compile session and be fine all without leaving X at all. So I don't loose control in the way I do with the nv driver. It is what I'm used to from my past X/Linux experience. But sadly vesa won't do more than 1024x768 pixels it seems, of course..

- *Now* I've also tried the commercial nvidia driver. Of course it's running fast and hippy, *but* it actually exhibits exactly the same bad behaviour as the nv driver; meaning, hit the key combo to get onto the console, then go drink a coffee or tea, then kill the compile from the console. (For some reason the fonts are also too big when running under this driver, although it *does* run at the same native 1440 x 900 resolution as does "nv".) I think an interesting point may be that this driver doesn't require much CPU, so it seems less likely to me to be some issue about the X driver and the kcryptd contending for the CPU cores; still, tell me if I should test with a swap on a non-encrypted partition. (BTW cat /dev/mapper/sda8_crypt > /dev/null will run at 40MB/sec and only use 50% of one core, so the bottleneck for encryption is generally not the cpu, but the builtin hard disk speed.)

Now I hope there is a way to get this bevahiour under control, it should be possible after all, or is it a hardware issue only exposed when using non-vesa interfaces?

In any case, if mlock'ing some or all of X helps around X locking up, then I'll be happy with that. I do *not* require all applications to run mlocked as well! Again, I'm talking about something much worse than the usual sluggyness exposed by swapping(/thrashing), and the vesa driver shows that not running any application mlocked actually can work just fine. (There were times in the kernel 2.4 age when swapping was really bad, but I'm running a 2.6 kernel here!) I'll be happy to patch and test X with mlock calls if you can tell me where to try them.

I'll leave it up to someone else to decide whether this is an issue to be dealt with in the Server/General area or in the driver.

I've tried every suggestion from the nvidia people on http://www.nvnews.net/vbulletin/showthread.php?t=58498 to make things reliable, and found only *one* thing that helped: switching off the second cpu core.

Be assured that this is the option I wanted the most..

So I'm still hoping mlockall would help.

Created attachment 16200 [details] [review]
Crude patch for running mlockall() in Xorg

Here's a very crude hack to run mlockall in the Xorg binary (I didn't see where the main procedure of Xorg is defined, so I put it into loader.c). I can confirm that cj_mlockall is being called successfully. I've only checked it with the nvidia driver (but from the former experience I expect the nv driver to behave similar).

I'm not yet very conclusive whether it really helps much--in a first test it retained me control nicely, in a second test X would be unresponsive for about a minute before letting me to kill the test. So it looks like something between the behaviour of the vesa driver and an unpatched X; switching off the second core seems to help rather more.

This might be a problem [in combination] with CONFIG_USER_SCHED.

I'm running a new kernel with this change now:

@@ -1,7 +1,7 @@
 #
 # Automatically generated make config: don't edit
 # Linux kernel version: 2.6.27.7
-# Wed Dec  3 19:24:49 2008
+# Wed Dec  3 19:29:40 2008
 #
 CONFIG_64BIT=y
 # CONFIG_X86_32 is not set
@@ -81,10 +81,8 @@ CONFIG_IKCONFIG_PROC=y
 CONFIG_LOG_BUF_SHIFT=16
 # CONFIG_CGROUPS is not set
 CONFIG_HAVE_UNSTABLE_SCHED_CLOCK=y
-CONFIG_GROUP_SCHED=y
-CONFIG_FAIR_GROUP_SCHED=y
-# CONFIG_RT_GROUP_SCHED is not set
-CONFIG_USER_SCHED=y
+# CONFIG_GROUP_SCHED is not set
+# CONFIG_USER_SCHED is not set
 # CONFIG_CGROUP_SCHED is not set
 CONFIG_SYSFS_DEPRECATED=y
 CONFIG_SYSFS_DEPRECATED_V2=y

and things seem to be much better.

Also see my report at http://bugzilla.kernel.org/show_bug.cgi?id=10781

Although the kernel I was running then didn't seem to have CONFIG_USER_SCHED enabled:

chris@novo:/boot$ ls -rt config*|while read f; do l $f; grep SCHED $f; echo; done|less
-rw-r--r-- 1 root root 63490 2008-04-27 22:10 config-2.6.22.19
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_IOSCHED="cfq"
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
CONFIG_NET_SCHED=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
# CONFIG_SCHEDSTATS is not set

-rw-r--r-- 1 root root 72175 2008-06-11 06:50 config-2.6.25.6
CONFIG_GROUP_SCHED=y
CONFIG_FAIR_GROUP_SCHED=y
# CONFIG_RT_GROUP_SCHED is not set
CONFIG_USER_SCHED=y
# CONFIG_CGROUP_SCHED is not set
CONFIG_IOSCHED_NOOP=y
CONFIG_IOSCHED_AS=y
CONFIG_IOSCHED_DEADLINE=y
CONFIG_IOSCHED_CFQ=y
CONFIG_DEFAULT_IOSCHED="cfq"
# CONFIG_SCHED_SMT is not set
CONFIG_SCHED_MC=y
# CONFIG_SCHED_HRTICK is not set
CONFIG_NET_SCHED=y
CONFIG_USB_EHCI_TT_NEWSCHED=y
CONFIG_SCHED_DEBUG=y
# CONFIG_SCHEDSTATS is not set

-rw-r--r-- 1 root root 73063 2008-06-22 21:44 config-2.6.25.8
...

This is not a bug.  Your machine is under both cpu and memory pressure, and to 
relieve memory pressure, it uses swap which puts it under higher cpu pressure 
due to your use of dm-crypt.