When some test with subtests starts crashing, it will not show in a piglit summary as a regression, since the actual subtests will go from pass to notrun and the failure will be only reported for the "unknown" test which will go from notrun to crash, and hence neither will end in the list of regressions. It can be still found in the changes summary, however the list of regressions is no longer reliable due to this. I'm not sure if this is a bug or a feature, however it seems somewhat unfortunate since this can lead to missed regressions. Was probably introduced by commit 938ec48e2575b78defd06d169f704ed8d4f11bce Author: Fabian Bieler <fabianbieler@fastmail.fm> Date: Sat Jan 6 23:36:02 2018 +0100 framework: Handle crashing subtest. Piglit silently ignored crashes in subtests. It's impossible to know what subtest crashed. Theoretically it might even be possible that a test crashes after the last call to piglit_report_subtest_result and thus no subtest crashed. Though the odds of that happening are probably pretty long. If a test with subtests crashes, this commit adds an additional subtest named "unknown" with result "crash". The issue that subsequent subtests are not run is not touched upon by this commit. This is more of a work-around. A proper fix would modify the C subtest framework to print a list of subtests it intends to run at the start of the test. This would enable the python subtest framework to know which subtest crashed. But that's a lot of work (it would require modifying more than 100 tests) and this is better than nothing. See also: https://bugs.freedesktop.org/show_bug.cgi?id=74642 Reviewed-by: Brian Paul <brianp@vmware.com>
Created attachment 136858 [details] [review] Framework/summary: Include crashing subtests in fixes and regressions. If a subtest crashed before 938ec48e2 no crash was recorded at all. A subtest that regressed from pass to crash was recorded as pass and notrun, respectively, too. The commit in question attempts to improve on the situation by at least recording some crash, albeit only of subtest "unknown". This is no proper fix for bug #74642 as noted in the commit message. Crashing subtests not showing up in the regressions tab of summaries is a problem, but that has been the case since at least 2014 (see bug #74642). The attached test tries to remedy that by at least including the subtest named "unknown" in the list of regressions (and fixes, if applicable).
That's no entirely accurate. The previous behavior would report crash of the entire test. So you could see: test1 pass -> crash subtest0 pass -> notrun subtest1 pass -> notrun the new behavior is: test1 2/2 -> 0/0 subtest0 pass -> notrun subtest1 pass -> notrun unknown notrun -> crash
Strange, I can't seem to reproduce that. I reverted 938ec48e2, ran arb_copy_buffer@data-sync once normally and once with a call to abort() after the first call to piglit_report_subtest_result() (tests/spec/arb_copy_buffer/data-sync.c:82). Neither "piglit summary console" nor "piglit summary html" report a crashing test or a regression. Am I missing something?
the behavior Jan sees is what the framework is *supposed* to do. This is pretty frustrating actually because it makes it impossible for us to track regressions in CI because the output is now inconsistant. If a test crashes and has subtests it doesn't behave the same way as other tests. If the test crashes just set the overall result to crash, it should be the "worst" result, and thus the whole test should show crash.
(In reply to Fabian Bieler from comment #3) > Strange, I can't seem to reproduce that. > I reverted 938ec48e2, ran arb_copy_buffer@data-sync once normally and once > with a call to abort() after the first call to > piglit_report_subtest_result() (tests/spec/arb_copy_buffer/data-sync.c:82). > > Neither "piglit summary console" nor "piglit summary html" report a crashing > test or a regression. > > Am I missing something? most of the crashes in OpenCL tests happen during kernel compile (before the first subtest). Maybe there's a difference in behaviour in those cases?
I think there's actually a bug in the summary code. I reverted this patch, and added the Fabian's patch to the data-sync test. The raw JSON contains what I expect, but `piglit summary console` returns all tests passed. I'll look into it some more.
(In reply to Jan Vesely from comment #5) > (In reply to Fabian Bieler from comment #3) > > Strange, I can't seem to reproduce that. > > I reverted 938ec48e2, ran arb_copy_buffer@data-sync once normally and once > > with a call to abort() after the first call to > > piglit_report_subtest_result() (tests/spec/arb_copy_buffer/data-sync.c:82). > > > > Neither "piglit summary console" nor "piglit summary html" report a crashing > > test or a regression. > > > > Am I missing something? > > most of the crashes in OpenCL tests happen during kernel compile (before the > first subtest). Maybe there's a difference in behaviour in those cases? You're absolutely right. Aborting before the first call to piglit_report_subtest_result shows me the behavior you described. I'll revert 938ec48e2 and have a look if I can find a solution to get this behavior for tests that crash after a call to piglit_report_subtest_result, too.
Fabian, it's the totalling code that's wrong. It doesn't account for crashes in tests with subtests. I'm looking at it now.
I've sent a patch for this, I cc'd both Jan and Fabian
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/mesa/piglit/issues/6.
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.