Bug 91680 - Different stats in output reports
Summary: Different stats in output reports
Alias: None
Product: piglit
Classification: Unclassified
Component: infrastructure (show other bugs)
Version: unspecified
Hardware: x86-64 (AMD64) Linux (All)
: medium normal
Assignee: Dylan Baker
QA Contact: Piglit Mailing List
Depends on:
Reported: 2015-08-18 15:11 UTC by cprigent
Modified: 2015-08-18 20:50 UTC (History)
0 users

See Also:
i915 platform:
i915 features:

results.json.bz2 (910.74 KB, text/plain)
2015-08-18 15:11 UTC, cprigent
A very short summary showing how subtests are handled (424 bytes, application/octet-stream)
2015-08-18 20:50 UTC, Dylan Baker

Description cprigent 2015-08-18 15:11:49 UTC
Created attachment 117768 [details]

Linux distribution: Ubuntu 14.04 LTS 64 bits
piglit: (HEAD, origin/master, origin/HEAD, master) 846755508ac44924b154a280e7a84c8004b6b085 from git://git.freedesktop.org/git/piglit

1. Execute command:
./piglit run tests/all results/all-reference
2. Wait the result
3. Execute command:
./piglit summary console results/all2
4. Compare the stats

Actual result:
2. Piglit reports:
[28030/28030] crash: 26, fail: 135, pass: 18692, skip: 9174, warn: 3 - Thank you for running Piglit!
3. Summary is:
        pass: 28904
        fail: 164
       crash: 26
        skip: 9457
     timeout: 0
        warn: 3
  incomplete: 0
  dmesg-warn: 0
  dmesg-fail: 0
       total: 38554
4. Numbers are different (example: 135 fail vs 164)

Same problem when generating an HTML or a CVS report, numbers are again different.
Comment 1 Dylan Baker 2015-08-18 20:48:42 UTC
This isn't actually a bug. What's happening is that 'piglit run' and 'piglit summary' are counting two different things.

Piglit run counts the number of tests as defined in the profile (in your case tests/all.py), it counts them up and sees that there are 28030 Test derived objects in the python file.

Piglit summary counts the number of tests + subtests as a total (tests with subtests are treated as a group rather than as a test), so while there were 28030 Test.run() calls, there were actually 28904 test + subtest results generated.

There is no way for the runner to know if a test has subtests or how many it has until it has finished running that test; and I prefer to keep the number of tests counted deterministic at start time (ie, not changing as the run goes, which would be confusing when running a summary against two incomplete results).

I've attached a very simple results file that hopefully will make it clearer how this works. I'll also send a patch that updates the README file with this information.

Sorry for the confusion.
Comment 2 Dylan Baker 2015-08-18 20:50:19 UTC
Created attachment 117774 [details]
A very short summary showing how subtests are handled

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.