Bug 101850

Summary: [PIGLIT] Piglit does not recognize some tests
Product: DRI Reporter: hecmundo <hector.edmundox.ramirez.gomez>
Component: IGTAssignee: Arek Hiler <arkadiusz.hiler>
Status: CLOSED WORKSFORME QA Contact:
Severity: normal    
Priority: medium    
Version: unspecified   
Hardware: x86-64 (AMD64)   
OS: Linux (All)   
Whiteboard:
i915 platform: i915 features:

Description hecmundo 2017-07-19 21:03:09 UTC
Description
=======================================
Piglit does not recognize these two tests:
igt@drm_mm
igt@drv_selftest


Versions
======================================
Component: intel-gpu-tools
tag: intel-gpu-tools-1.19-96-gfb1ddc4
commit: fb1ddc47003ad6a683db79beeb81b6cbab1feb7c

Component: piglit
tag: piglit-v1
commit: 56e7e5583cd4a3ca15a8cda154d46d168959dd25


How to reproduce
=====================================
Run an IGT execution on a platform. Testslit in use must contain the two problematic tests mentioned in description. After initiating the execution, piglit will fail

1) Run as this command:
$ : /home/gfx/intel-graphics/intel-gpu-tools/piglit/piglit run igt -o /home/gfx/iterationT --test-list /home/gfx/dead.testlist -l verbose --sync
Traceback (most recent call last):
  File "/home/gfx/intel-graphics/intel-gpu-tools/piglit/piglit", line 174, in <module>
    main()
  File "/home/gfx/intel-graphics/intel-gpu-tools/piglit/piglit", line 170, in main
    sys.exit(runner(args))
  File "/home/gfx/intel-graphics/intel-gpu-tools/piglit/framework/exceptions.py", line 51, in _inner
    func(*args, **kwargs)
  File "/home/gfx/intel-graphics/intel-gpu-tools/piglit/framework/programs/run.py", line 357, in run
    profile.run(profiles, args.log_level, backend, args.concurrency)
  File "/home/gfx/intel-graphics/intel-gpu-tools/piglit/framework/profile.py", line 387, in run
    profiles = [(p, list(p.itertests())) for p in profiles]
  File "/home/gfx/intel-graphics/intel-gpu-tools/piglit/framework/profile.py", line 387, in <listcomp>
    profiles = [(p, list(p.itertests())) for p in profiles]
  File "/home/gfx/intel-graphics/intel-gpu-tools/piglit/framework/profile.py", line 317, in itertests
    opts[n] = self.test_list[n]
  File "/home/gfx/intel-graphics/intel-gpu-tools/piglit/framework/profile.py", line 161, in __getitem__
    return self.__container[key.lower()]
KeyError: 'igt@drm_mm'



Expected  behavior
=====================================
Execution should begin normally


Current Behavior
=====================================
Execution does not begin due to unrecognized tests
Comment 1 Arek Hiler 2017-07-20 13:40:50 UTC
The bug is actually due to those tests not respecting "the contract".

IGTs are supposed to exit with code 79 if --list-subtests is supplied and we have a binary without any subtests. 

Those tests used the wrong igt_main macro and the ./bin --list-subtests exited with 0.

Yet another thing to add to the distcheck :-)

Thanks for reporting, I'll send a patch in a minute.
Comment 2 Arek Hiler 2017-07-20 14:00:07 UTC
Actually it's not as straightforward as I thought.

Quote form the igt_command_line.sh checker:

# Subtest enumeration of kernel selftest launchers depends
# on the running kernel. If selftests are not enabled,
# they will output nothing and exit with 0.

I'll have to figure out how to handle this more gracefully and on which side this special case should be handled, before jumping to a solution.

This also means that you are running a testlist which has the whole binary instead of a subtest and/or you ran it on a kernel with selftests disabled.
Comment 3 hecmundo 2017-07-20 19:25:14 UTC
(In reply to arkadiusz.hiler@intel.com from comment #1)
> The bug is actually due to those tests not respecting "the contract".
> 
> IGTs are supposed to exit with code 79 if --list-subtests is supplied and we
> have a binary without any subtests. 
> 
> Those tests used the wrong igt_main macro and the ./bin --list-subtests
> exited with 0.
> 
> Yet another thing to add to the distcheck :-)
> 
> Thanks for reporting, I'll send a patch in a minute.

This may help: Actually there're subtests

(05:22 AM) [gfx@KBL-3] [tests]$ : sudo ./drm_mm --l
sanitycheck
init
debug
reserve
insert
replace
insert_range
align
align32
align64
evict
evict_range
bottomup
topdown
color
color_evict
color_evict_range
(05:23 AM) [gfx@KBL-3] [tests]$ : $?
0: command not found
$ : sudo ./drv_selftest --l
mock_sanitycheck
mock_fence
mock_scatterlist
mock_syncmap
mock_uncore
mock_breadcrumbs
mock_timelines
mock_requests
mock_objects
mock_dmabuf
mock_vma
mock_evict
mock_gtt
live_sanitycheck
live_uncore
live_requests
live_objects
live_dmabuf
live_coherency
live_gtt
live_contexts
live_hangcheck
$ : $?
0: command not found
Comment 4 hecmundo 2017-07-20 19:38:06 UTC
(In reply to arkadiusz.hiler@intel.com from comment #2)
> Actually it's not as straightforward as I thought.
> 
> Quote form the igt_command_line.sh checker:
> 
> # Subtest enumeration of kernel selftest launchers depends
> # on the running kernel. If selftests are not enabled,
> # they will output nothing and exit with 0.
> 
> I'll have to figure out how to handle this more gracefully and on which side
> this special case should be handled, before jumping to a solution.
> 
> This also means that you are running a testlist which has the whole binary
> instead of a subtest and/or you ran it on a kernel with selftests disabled.

By "you ran it on a kernel with selftests disabled" you refer to the "CONFIG_STATIC_KEYS_SELFTEST" variable in the kernel config file?
Comment 5 Arek Hiler 2017-07-21 08:37:39 UTC
Let me explain how it works and what may be the issue.


How it works
============

drv_selftest and drm_mm (DRM_DEBUG_MM_SELFTEST and CONFIG_DRM_I915_SELFTEST options accordingly) are thin shims around in-kernel selftests. It just provides convinient way of triggering them.

Result of --list-subtests depends on the kernel - it lists all the subtests that are available in the kernel currently running.

Piglit executes test binaries with --list-subtests to build internal testlist. If the binary ends with return code 0 then everything listed is added in form igt@binary@subtest. If the return code is 79 then the whole binary is added in the form igt@binary.

Note that igt@binary is not in the list if the binary is supposed to have subtests.


The issue
=========

In your testlist you have igt@drm_mm and igt@drv_selftest and those won't be handled by piglit, as those binaries are in the category of "the ones having subtests".

You should revise the way you create your testlist, and list drv_selftest and drm_mm with the subtests (e.g. igt@drm_mm@sanitycheck).

I'll take a look into Piglit internals to have some saner way of reporting that the test is "not available" than just blowing up with enigmatic exceptions :-)
Comment 6 Humberto Israel Perez Rodriguez 2017-07-26 15:38:52 UTC
(In reply to arkadiusz.hiler@intel.com from comment #5)
> Let me explain how it works and what may be the issue.
> 
> 
> How it works
> ============
> 
> drv_selftest and drm_mm (DRM_DEBUG_MM_SELFTEST and CONFIG_DRM_I915_SELFTEST
> options accordingly) are thin shims around in-kernel selftests. It just
> provides convinient way of triggering them.
> 
> Result of --list-subtests depends on the kernel - it lists all the subtests
> that are available in the kernel currently running.
> 
> Piglit executes test binaries with --list-subtests to build internal
> testlist. If the binary ends with return code 0 then everything listed is
> added in form igt@binary@subtest. If the return code is 79 then the whole
> binary is added in the form igt@binary.
> 
> Note that igt@binary is not in the list if the binary is supposed to have
> subtests.
> 
> 
> The issue
> =========
> 
> In your testlist you have igt@drm_mm and igt@drv_selftest and those won't be
> handled by piglit, as those binaries are in the category of "the ones having
> subtests".
> 
> You should revise the way you create your testlist, and list drv_selftest
> and drm_mm with the subtests (e.g. igt@drm_mm@sanitycheck).
> 
> I'll take a look into Piglit internals to have some saner way of reporting
> that the test is "not available" than just blowing up with enigmatic
> exceptions :-)


Hi arkadiusz

thanks for this explication about how to works those tests, after read this comment we noticed that in the server where we compile igt the current kernel did not has the kernel flags that you mention enabled (DRM_DEBUG_MM_SELFTEST and CONFIG_DRM_I915_SELFTEST), so we compiled a new kernel for our server enabling these flags and now we are able to list the subtests from drv_selftest and drm_mm.

Thanks a lot


closing this bug because we found the fix in our side ;)

Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.