Created attachment 117768 [details] results.json.bz2 Linux distribution: Ubuntu 14.04 LTS 64 bits piglit: (HEAD, origin/master, origin/HEAD, master) 846755508ac44924b154a280e7a84c8004b6b085 from git://git.freedesktop.org/git/piglit Steps: ------ 1. Execute command: ./piglit run tests/all results/all-reference 2. Wait the result 3. Execute command: ./piglit summary console results/all2 4. Compare the stats Actual result: -------------- 2. Piglit reports: [28030/28030] crash: 26, fail: 135, pass: 18692, skip: 9174, warn: 3 - Thank you for running Piglit! 3. Summary is: pass: 28904 fail: 164 crash: 26 skip: 9457 timeout: 0 warn: 3 incomplete: 0 dmesg-warn: 0 dmesg-fail: 0 total: 38554 4. Numbers are different (example: 135 fail vs 164) Info: ------ Same problem when generating an HTML or a CVS report, numbers are again different.
This isn't actually a bug. What's happening is that 'piglit run' and 'piglit summary' are counting two different things. Piglit run counts the number of tests as defined in the profile (in your case tests/all.py), it counts them up and sees that there are 28030 Test derived objects in the python file. Piglit summary counts the number of tests + subtests as a total (tests with subtests are treated as a group rather than as a test), so while there were 28030 Test.run() calls, there were actually 28904 test + subtest results generated. There is no way for the runner to know if a test has subtests or how many it has until it has finished running that test; and I prefer to keep the number of tests counted deterministic at start time (ie, not changing as the run goes, which would be confusing when running a summary against two incomplete results). I've attached a very simple results file that hopefully will make it clearer how this works. I'll also send a patch that updates the README file with this information. Sorry for the confusion.
Created attachment 117774 [details] A very short summary showing how subtests are handled
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.