I completely cleared Xorg modifier map (modmap):

$ ./clearmods 
+ xmodmap -e 'clear shift'
+ xmodmap -e 'clear lock'
+ xmodmap -e 'clear control'
+ xmodmap -e 'clear mod1'
+ xmodmap -e 'clear mod2'
+ xmodmap -e 'clear mod3'
+ xmodmap -e 'clear mod4'
+ xmodmap -e 'clear mod5'

$ xmodmap 
xmodmap:  up to 0 keys per modifier, (keycodes in parentheses):

shift     
lock      
control   
mod1      
mod2      
mod3      
mod4      
mod5      

As you can see there is no keysym mapped as modifier. Now I press the "Shift_L" key, for example, and get following in xev:

KeyPress event, serial 23, synthetic NO, window 0x200001,
    root 0x12c, subw 0x0, time 12108291, (109,83), root:(111,85),
    state 0x0, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XmbLookupString gives 0 bytes: 
    XFilterEvent returns: False

KeyRelease event, serial 26, synthetic NO, window 0x200001,
    root 0x12c, subw 0x0, time 12108419, (109,83), root:(111,85),
    state 0x1, keycode 50 (keysym 0xffe1, Shift_L), same_screen YES,
    XLookupString gives 0 bytes: 
    XFilterEvent returns: False

You can see that state is changed from 0x0 to 0x1 which means server thinks the shift was pressed but it shouldn't because it is not mapped to "shift" modifier in modmap. This also happens for "Alt" key which is usually mapped to "mod1" modifier.

However when I unmap all keysyms associated with "control" modifier server recognizes it correctly and no longer changes state from 0x0 to 0x4 when I press CTRL.

Reproducible with the latest Xorg (I have 1.9.99.1). Let me know if you need more info.