Created attachment 117768 [details]
Linux distribution: Ubuntu 14.04 LTS 64 bits
piglit: (HEAD, origin/master, origin/HEAD, master) 846755508ac44924b154a280e7a84c8004b6b085 from git://git.freedesktop.org/git/piglit
1. Execute command:
./piglit run tests/all results/all-reference
2. Wait the result
3. Execute command:
./piglit summary console results/all2
4. Compare the stats
2. Piglit reports:
[28030/28030] crash: 26, fail: 135, pass: 18692, skip: 9174, warn: 3 - Thank you for running Piglit!
3. Summary is:
4. Numbers are different (example: 135 fail vs 164)
Same problem when generating an HTML or a CVS report, numbers are again different.
This isn't actually a bug. What's happening is that 'piglit run' and 'piglit summary' are counting two different things.
Piglit run counts the number of tests as defined in the profile (in your case tests/all.py), it counts them up and sees that there are 28030 Test derived objects in the python file.
Piglit summary counts the number of tests + subtests as a total (tests with subtests are treated as a group rather than as a test), so while there were 28030 Test.run() calls, there were actually 28904 test + subtest results generated.
There is no way for the runner to know if a test has subtests or how many it has until it has finished running that test; and I prefer to keep the number of tests counted deterministic at start time (ie, not changing as the run goes, which would be confusing when running a summary against two incomplete results).
I've attached a very simple results file that hopefully will make it clearer how this works. I'll also send a patch that updates the README file with this information.
Sorry for the confusion.
Created attachment 117774 [details]
A very short summary showing how subtests are handled