Bug 18599

Created attachment 20420 [details]
Pseudo code from our OpenGL log showing code that fails and code that doesn't fail.

Overview:
---------
Our product uses OpenGL but attempts to buffer the glBegin statements until we are sure that a normal/vertex will be delivered. We do this because to prevent creating empty begin/end blocks because some of our polygons are collapsed but it isn't know at the time the block is first opened. We cache the glBegin request and then When the first normal/vertex is to be delivered we open the block and deliver the first normal/vertex. This introduced an interesting problem with multi-color translucent smooth shading. Because we buffer the glBegin the color we set on the first vertex occurs outside the block instead of inside. In other words we effectively send instructions as follows:
glBegin
glColor4fv
glNormal3d
glVertex3d
glColor4fv
glNormal3d
glVertex3d
...
glEnd

but because of our buffering of glBegin we actually deliver the following to OpenGL:
glColor4fv
glBegin
glNormal3d
glVertex3d
glColor4fv
glNormal3d
glVertex3d
...
glEnd

Note the swapped glBegin with the glColor4fv at the beginning of the delivered instructions. This shouldn't matter because OpenGL is a state engine and in fact produces the correct output if display lists are not used however if we build display lists and then execute them Mesa asserts on line 391 of tnl/t_vertex.c: assert(a[j].inputstride == vptr->stride); I'm not sure if the assertion is correct or not. Unfortunately/fortunately (depending on how you look at it) the default compilation for Mesa's release build has assertions enabled so that GLX servers using Mesa instead of native OpenGL drivers will assert with our product unless we supply our own build of the Mesa libraries with assertions disabled. We can work around the problem by making sure color gets delivered before beginning the block but I thought I should point out the problem if it is indeed a problem.

Steps to reproduce:
-------------------
Attached is a file that contains a log of "pseudo" code (from a dump of our OpenGL instruction log) that is the smallest example I could build. The example had to have one four node polygon with all the colors having the same value and the two other four node polygons having gradient colors The direction of the gradient seemed to matter as well (the gradient in my case varies laterally across the cell width instead of longitudinally across the shared cell edge. The first half of the attachment our applications OpenGL log from the pseudo code that asserts and the second half is the output from the pseudo code that does not assert. The only difference between the two sets is the order of the glColor4fv and the opening glBegin. 

Sorry that the code probably doesn't compile its just an approximation of the code we send to OpenGL. Look for the first glColor4fv call. Note that in the first "BAD" block the glColor4fv call comes before the glBegin(GL_POLYGON) whereas in the second example it comes after. The first case causes Mesa to assert and the second does not.

Actual Results:
---------------
Assertion failure, tnl/t_vertex.c:391, assert(a[j].inputstride == vptr->stride);

Expected Results:
-----------------
Since OpenGL is a state engine the delivery of the glColor4fv before the glBegin shouldn't have made any difference and indeed when display lists are not used it doesn't seem to make any visual difference. Also when display lists are used but assertions disabled it doesn't seem to make any visual difference.

Build Date & Platform:
----------------------
The bug report version label above did not give me an option to enter 7.2 so I put CVS as the version number. This occurs at least on Linux 64 and 32 bit platforms running Mesa 7.2 and Mesa 5.0.