Created attachment 140436 [details]
Shader test for this case is attached.
GLSL's approach to NaN is pretty weak.
The following rules apply to both single and double precision operations:
Dividing by 0 results in the appropriately signed IEEE Inf. Any denormalized
value input into a shader or potentially generated by an operation in a shader
can be flushed to 0. In general, correct signedness of 0 is not required. The
rounding mode cannot be set and is undefined. Support for signaling NaNs is
not required and exceptions are never raised. Operations and built-in functions
that operate on a NaN are not required to return a NaN as the result.
So NaN * 0 -> 0 appears to be a valid transformation. I haven't gone back and checked what's in the core specs, but I doubt it's any different.
(In reply to Ilia Mirkin from comment #1)
> So NaN * 0 -> 0 appears to be a valid transformation. I haven't gone back
> and checked what's in the core specs, but I doubt it's any different.
For this particular shader, it's actually -Inf * 0.0 -> NaN that is checked.
The optimization though looks unsafe to me. Although sure enough glsl allows this crazyness (I mean why does it even have a isnan function if you're not required to generate NaNs...), but some (newer) apps might expect ieee754 behavior (or they might expect d3d10 behavior, which is the same wrt NaNs).
Basically, unless you use precise, it's like compiling -ffast-math. A lot of apps go to lengths to avoid the possibility of generating Inf or NaN in shaders due the the problems that they cause.