| Summary: | Crashing tests with subtests don't show in a list of regressions | ||
|---|---|---|---|
| Product: | piglit | Reporter: | Pavel Ondračka <pavel.ondracka> |
| Component: | infrastructure | Assignee: | Fabian Bieler <fabianbieler> |
| Status: | RESOLVED MOVED | QA Contact: | Piglit Mailing List <piglit> |
| Severity: | normal | ||
| Priority: | medium | ||
| Version: | unspecified | ||
| Hardware: | Other | ||
| OS: | All | ||
| Whiteboard: | |||
| i915 platform: | i915 features: | ||
| Attachments: | Framework/summary: Include crashing subtests in fixes and regressions. | ||
|
Description
Pavel Ondračka
2018-01-19 11:54:49 UTC
Created attachment 136858 [details] [review] Framework/summary: Include crashing subtests in fixes and regressions. If a subtest crashed before 938ec48e2 no crash was recorded at all. A subtest that regressed from pass to crash was recorded as pass and notrun, respectively, too. The commit in question attempts to improve on the situation by at least recording some crash, albeit only of subtest "unknown". This is no proper fix for bug #74642 as noted in the commit message. Crashing subtests not showing up in the regressions tab of summaries is a problem, but that has been the case since at least 2014 (see bug #74642). The attached test tries to remedy that by at least including the subtest named "unknown" in the list of regressions (and fixes, if applicable). That's no entirely accurate. The previous behavior would report crash of the entire test. So you could see: test1 pass -> crash subtest0 pass -> notrun subtest1 pass -> notrun the new behavior is: test1 2/2 -> 0/0 subtest0 pass -> notrun subtest1 pass -> notrun unknown notrun -> crash Strange, I can't seem to reproduce that. I reverted 938ec48e2, ran arb_copy_buffer@data-sync once normally and once with a call to abort() after the first call to piglit_report_subtest_result() (tests/spec/arb_copy_buffer/data-sync.c:82). Neither "piglit summary console" nor "piglit summary html" report a crashing test or a regression. Am I missing something? the behavior Jan sees is what the framework is *supposed* to do. This is pretty frustrating actually because it makes it impossible for us to track regressions in CI because the output is now inconsistant. If a test crashes and has subtests it doesn't behave the same way as other tests. If the test crashes just set the overall result to crash, it should be the "worst" result, and thus the whole test should show crash. (In reply to Fabian Bieler from comment #3) > Strange, I can't seem to reproduce that. > I reverted 938ec48e2, ran arb_copy_buffer@data-sync once normally and once > with a call to abort() after the first call to > piglit_report_subtest_result() (tests/spec/arb_copy_buffer/data-sync.c:82). > > Neither "piglit summary console" nor "piglit summary html" report a crashing > test or a regression. > > Am I missing something? most of the crashes in OpenCL tests happen during kernel compile (before the first subtest). Maybe there's a difference in behaviour in those cases? I think there's actually a bug in the summary code. I reverted this patch, and added the Fabian's patch to the data-sync test. The raw JSON contains what I expect, but `piglit summary console` returns all tests passed. I'll look into it some more. (In reply to Jan Vesely from comment #5) > (In reply to Fabian Bieler from comment #3) > > Strange, I can't seem to reproduce that. > > I reverted 938ec48e2, ran arb_copy_buffer@data-sync once normally and once > > with a call to abort() after the first call to > > piglit_report_subtest_result() (tests/spec/arb_copy_buffer/data-sync.c:82). > > > > Neither "piglit summary console" nor "piglit summary html" report a crashing > > test or a regression. > > > > Am I missing something? > > most of the crashes in OpenCL tests happen during kernel compile (before the > first subtest). Maybe there's a difference in behaviour in those cases? You're absolutely right. Aborting before the first call to piglit_report_subtest_result shows me the behavior you described. I'll revert 938ec48e2 and have a look if I can find a solution to get this behavior for tests that crash after a call to piglit_report_subtest_result, too. Fabian, it's the totalling code that's wrong. It doesn't account for crashes in tests with subtests. I'm looking at it now. I've sent a patch for this, I cc'd both Jan and Fabian -- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/mesa/piglit/issues/6. |
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.