If all the modules given fail to load, the daemon will try to shut down and crash when trying to clean up:

E: [pulseaudio] main.c: Module load failed.
E: [pulseaudio] main.c: Failed to initialize daemon.
E: [pulseaudio] xmalloc.c: Assertion 'size > 0' failed at pulse/xmalloc.c:62, function pa_xmalloc(). Aborting.

Easily fixed with this patch:

Index: src/pulsecore/module.c
===================================================================
--- src/pulsecore/module.c	(revision 27650)
+++ src/pulsecore/module.c	(working copy)
@@ -217,6 +217,9 @@
     pa_assert(c);
     pa_assert(c->modules);
 
+    if (pa_idxset_isempty(c->modules))
+        return;
+
     /* Unload modules in reverse order by default */
     indices = pa_xnew(uint32_t, pa_idxset_size(c->modules));
     i = 0;