Created attachment 119318 [details]
trace that demonstrates 2 FS and 2 VS programs generated in a single draw call

FF programs are affected by various state. It seems it's possible to do things "just right" to cause them to be regenerated twice in a single draw. The attached trace produces the following for me:

48: message: shader compiler issue 1: type: 1,
48: message: shader compiler issue 1: type: 0,
53: message: shader compiler issue 1: type: 1,
53: message: shader compiler issue 1: type: 0,
172: message: shader compiler issue 1: type: 1,
172: message: shader compiler issue 1: type: 0,
172: message: shader compiler issue 1: type: 1,
172: message: shader compiler issue 1: type: 0,
275: message: shader compiler issue 1: type: 1,
275: message: shader compiler issue 1: type: 0,
275: message: shader compiler issue 1: type: 0,
2172: message: shader compiler issue 1: type: 1
2172: message: shader compiler issue 1: type: 0
2172: message: shader compiler issue 1: type: 0
2239: message: shader compiler issue 1: type: 1
2239: message: shader compiler issue 1: type: 0
2328: message: shader compiler issue 1: type: 1
2328: message: shader compiler issue 1: type: 0
2328: message: shader compiler issue 1: type: 1
2328: message: shader compiler issue 1: type: 0

Type 0/1 are VS and FS (id corresponds to PIPE_SHADER_*). Note that the draw calls at 172 and 2328 both generate 2 programs each, while draw calls 275 and 2172 generate 2 programs for one of the types.

One set is generated via the path

vbo_exec_DrawRangeElementsBaseVertex -> validate_DrawElements_common -> check_valid_to_render -> _mesa_valid_to_render -> _mesa_update_state -> update_program

while the second is generated via

vbo_exec_DrawRangeElementsBaseVertex -> vbo_validated_drawrangeelements -> vbo_bind_arrays -> _mesa_update_state -> update_program.

I guess in between the _NEW_PROGRAM bit manages to set itself.