On my desktop system with radeonsi and radeon.ko the dmesg starts to fill with vm errors, then after a few seconds a gpu fault is detected, the screen turns off, and it cannot recover from that.
On an intel haswell laptop the screen locks up immediately, and the system becomes totally unresponsive.
I don't think this is the correct reaction to a simple programming mistake.
System wide resource limitations for users should be implemented with cgroups or such, IMO it cannot be responsibility of a single library. You will likely encounter similar situation using some other library, as example: "while(1) fork();". You should setup limitations for memory usage per user or per process.