Bug 101850

The bug is actually due to those tests not respecting "the contract".

IGTs are supposed to exit with code 79 if --list-subtests is supplied and we have a binary without any subtests. 

Those tests used the wrong igt_main macro and the ./bin --list-subtests exited with 0.

Yet another thing to add to the distcheck :-)

Thanks for reporting, I'll send a patch in a minute.

Actually it's not as straightforward as I thought.

Quote form the igt_command_line.sh checker:

# Subtest enumeration of kernel selftest launchers depends
# on the running kernel. If selftests are not enabled,
# they will output nothing and exit with 0.

I'll have to figure out how to handle this more gracefully and on which side this special case should be handled, before jumping to a solution.

This also means that you are running a testlist which has the whole binary instead of a subtest and/or you ran it on a kernel with selftests disabled.

(In reply to arkadiusz.hiler@intel.com from comment #1)
> The bug is actually due to those tests not respecting "the contract".
> 
> IGTs are supposed to exit with code 79 if --list-subtests is supplied and we
> have a binary without any subtests. 
> 
> Those tests used the wrong igt_main macro and the ./bin --list-subtests
> exited with 0.
> 
> Yet another thing to add to the distcheck :-)
> 
> Thanks for reporting, I'll send a patch in a minute.

This may help: Actually there're subtests

(05:22 AM) [gfx@KBL-3] [tests]$ : sudo ./drm_mm --l
sanitycheck
init
debug
reserve
insert
replace
insert_range
align
align32
align64
evict
evict_range
bottomup
topdown
color
color_evict
color_evict_range
(05:23 AM) [gfx@KBL-3] [tests]$ : $?
0: command not found
$ : sudo ./drv_selftest --l
mock_sanitycheck
mock_fence
mock_scatterlist
mock_syncmap
mock_uncore
mock_breadcrumbs
mock_timelines
mock_requests
mock_objects
mock_dmabuf
mock_vma
mock_evict
mock_gtt
live_sanitycheck
live_uncore
live_requests
live_objects
live_dmabuf
live_coherency
live_gtt
live_contexts
live_hangcheck
$ : $?
0: command not found

(In reply to arkadiusz.hiler@intel.com from comment #2)
> Actually it's not as straightforward as I thought.
> 
> Quote form the igt_command_line.sh checker:
> 
> # Subtest enumeration of kernel selftest launchers depends
> # on the running kernel. If selftests are not enabled,
> # they will output nothing and exit with 0.
> 
> I'll have to figure out how to handle this more gracefully and on which side
> this special case should be handled, before jumping to a solution.
> 
> This also means that you are running a testlist which has the whole binary
> instead of a subtest and/or you ran it on a kernel with selftests disabled.

By "you ran it on a kernel with selftests disabled" you refer to the "CONFIG_STATIC_KEYS_SELFTEST" variable in the kernel config file?

Let me explain how it works and what may be the issue.


How it works
============

drv_selftest and drm_mm (DRM_DEBUG_MM_SELFTEST and CONFIG_DRM_I915_SELFTEST options accordingly) are thin shims around in-kernel selftests. It just provides convinient way of triggering them.

Result of --list-subtests depends on the kernel - it lists all the subtests that are available in the kernel currently running.

Piglit executes test binaries with --list-subtests to build internal testlist. If the binary ends with return code 0 then everything listed is added in form igt@binary@subtest. If the return code is 79 then the whole binary is added in the form igt@binary.

Note that igt@binary is not in the list if the binary is supposed to have subtests.


The issue
=========

In your testlist you have igt@drm_mm and igt@drv_selftest and those won't be handled by piglit, as those binaries are in the category of "the ones having subtests".

You should revise the way you create your testlist, and list drv_selftest and drm_mm with the subtests (e.g. igt@drm_mm@sanitycheck).

I'll take a look into Piglit internals to have some saner way of reporting that the test is "not available" than just blowing up with enigmatic exceptions :-)

(In reply to arkadiusz.hiler@intel.com from comment #5)
> Let me explain how it works and what may be the issue.
> 
> 
> How it works
> ============
> 
> drv_selftest and drm_mm (DRM_DEBUG_MM_SELFTEST and CONFIG_DRM_I915_SELFTEST
> options accordingly) are thin shims around in-kernel selftests. It just
> provides convinient way of triggering them.
> 
> Result of --list-subtests depends on the kernel - it lists all the subtests
> that are available in the kernel currently running.
> 
> Piglit executes test binaries with --list-subtests to build internal
> testlist. If the binary ends with return code 0 then everything listed is
> added in form igt@binary@subtest. If the return code is 79 then the whole
> binary is added in the form igt@binary.
> 
> Note that igt@binary is not in the list if the binary is supposed to have
> subtests.
> 
> 
> The issue
> =========
> 
> In your testlist you have igt@drm_mm and igt@drv_selftest and those won't be
> handled by piglit, as those binaries are in the category of "the ones having
> subtests".
> 
> You should revise the way you create your testlist, and list drv_selftest
> and drm_mm with the subtests (e.g. igt@drm_mm@sanitycheck).
> 
> I'll take a look into Piglit internals to have some saner way of reporting
> that the test is "not available" than just blowing up with enigmatic
> exceptions :-)


Hi arkadiusz

thanks for this explication about how to works those tests, after read this comment we noticed that in the server where we compile igt the current kernel did not has the kernel flags that you mention enabled (DRM_DEBUG_MM_SELFTEST and CONFIG_DRM_I915_SELFTEST), so we compiled a new kernel for our server enabling these flags and now we are able to list the subtests from drv_selftest and drm_mm.

Thanks a lot


closing this bug because we found the fix in our side ;)