| Summary: | Different stats in output reports | ||
|---|---|---|---|
| Product: | piglit | Reporter: | cprigent <christophe.prigent> |
| Component: | infrastructure | Assignee: | Dylan Baker <baker.dylan.c> |
| Status: | RESOLVED NOTABUG | QA Contact: | Piglit Mailing List <piglit> |
| Severity: | normal | ||
| Priority: | medium | ||
| Version: | unspecified | ||
| Hardware: | x86-64 (AMD64) | ||
| OS: | Linux (All) | ||
| Whiteboard: | |||
| i915 platform: | i915 features: | ||
| Attachments: |
results.json.bz2
A very short summary showing how subtests are handled |
||
This isn't actually a bug. What's happening is that 'piglit run' and 'piglit summary' are counting two different things. Piglit run counts the number of tests as defined in the profile (in your case tests/all.py), it counts them up and sees that there are 28030 Test derived objects in the python file. Piglit summary counts the number of tests + subtests as a total (tests with subtests are treated as a group rather than as a test), so while there were 28030 Test.run() calls, there were actually 28904 test + subtest results generated. There is no way for the runner to know if a test has subtests or how many it has until it has finished running that test; and I prefer to keep the number of tests counted deterministic at start time (ie, not changing as the run goes, which would be confusing when running a summary against two incomplete results). I've attached a very simple results file that hopefully will make it clearer how this works. I'll also send a patch that updates the README file with this information. Sorry for the confusion. Created attachment 117774 [details]
A very short summary showing how subtests are handled
|
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.
Created attachment 117768 [details] results.json.bz2 Linux distribution: Ubuntu 14.04 LTS 64 bits piglit: (HEAD, origin/master, origin/HEAD, master) 846755508ac44924b154a280e7a84c8004b6b085 from git://git.freedesktop.org/git/piglit Steps: ------ 1. Execute command: ./piglit run tests/all results/all-reference 2. Wait the result 3. Execute command: ./piglit summary console results/all2 4. Compare the stats Actual result: -------------- 2. Piglit reports: [28030/28030] crash: 26, fail: 135, pass: 18692, skip: 9174, warn: 3 - Thank you for running Piglit! 3. Summary is: pass: 28904 fail: 164 crash: 26 skip: 9457 timeout: 0 warn: 3 incomplete: 0 dmesg-warn: 0 dmesg-fail: 0 total: 38554 4. Numbers are different (example: 135 fail vs 164) Info: ------ Same problem when generating an HTML or a CVS report, numbers are again different.