Repeating what I've said on the other bug report: I think we need to conditionalize this test on pci ids. Skip it on !ULT, make it fail on ULT if pc8+ doesn't work. Otherwise we won't ever have solid tracking of any pc8+ regressions.

/sys/kernel/debug/dri/0/i915_capabilities ?

(In reply to comment #2)
> /sys/kernel/debug/dri/0/i915_capabilities ?

gen: 7
pch: 3
is_mobile: yes
is_i85x: no
is_i915g: no
is_i945gm: no
is_g33: no
need_gfx_hws: yes
is_g4x: no
is_pineview: no
is_broadwater: no
is_crestline: no
is_ivybridge: no
is_valleyview: no
is_haswell: yes
is_preliminary: no
has_force_wake: yes
has_fbc: yes
has_pipe_cxsr: no
has_hotplug: yes
cursor_needs_physical: no
has_overlay: no
overlay_needs_physical: no
supports_tv: no
has_bsd_ring: yes
has_blt_ring: yes
has_vebox_ring: yes
has_llc: yes
has_ddi: yes
has_fpga_dbg: yes

(In reply to comment #3)
> (In reply to comment #2)
> > /sys/kernel/debug/dri/0/i915_capabilities ?

My apologies, I was making a suggestion to Daniel about another approach instead of relying on PCI-ID for test filtering.

I completely disagree and think we should close this as notabug.

If you get a SKIP it's because your machine is not properly configured. The graphics driver can't do anything to configure your machine. The configuration depends on all the device drivers, runtime PM policies, correct BIOS and detaching devices that wake up too much.

I have been exchanging emails with QA (Ouping + Gordon), trying to help them properly configure their machines for PC8, and they do have machines that can reach PC8+ residencies. I expect QA to run the pc8 test on those machines and only then report the failures.

If we replace the SKIP with FAIL on machines that are not properly configured we'll start getting bug reports that are not really bugs and we won't be able to fix. We'll also have to always analyze the logs to check if the report is really a regression or if it's just a badly configured machine.

(In reply to comment #5)
> If we replace the SKIP with FAIL on machines that are not properly
> configured we'll start getting bug reports that are not really bugs and we
> won't be able to fix. We'll also have to always analyze the logs to check if
> the report is really a regression or if it's just a badly configured machine.

And I absolutely want these bug reports. You're right that misconfigured test machines isn't our bug, but we need to notice such issues, track and fix them. I think adding a QA component in our internal JIRA would be good for that, I'll chat with Gavin about this.

So yes, if a test can't work because the machine is misconfigured it should fail, not skip. That's how all the other testcase already work, and occasionally we get a bug report which isn't a kernel issue but a setup issue. The other option of silently not testing features (which is currently the case, at least according to QA's nightly test db there's not a single machine where the pc8 tests are executed).

Ok, igt/pc8 now fails hard if the machine is misconfigured and can't do pc8+, but should be able to.

Run on latest -nightly kernel
output:
Test assertion failure function basic_subtest, file pc8.c:608:
Failed assertion: pc8_plus_enabled()
Machine is not reaching PC8+ states, please check its configuration.
Subtest basic: FAIL
Checking the if the DRM resources match.
Test assertion failure function drm_resources_equal_subtest, file pc8.c:639:
Failed assertion: pc8_plus_enabled()
Subtest drm-resources-equal: FAIL
Testing batchbuffers.
Test assertion failure function batch_subtest, file pc8.c:665:
Failed assertion: pc8_plus_enabled()
Subtest batch: FAIL
Test assertion failure function i2c_subtest, file pc8.c:691:
Failed assertion: pc8_plus_enabled()
Subtest i2c: FAIL
Stress testing.
Test assertion failure function stress_test, file pc8.c:707:
Failed assertion: pc8_plus_enabled()
Subtest stress-test: FAIL
Test requirement not met in function main, file pc8.c:764:
Test requirement: (!(do_register_compare))
Subtest register-compare: SKIP

(In reply to comment #8)
> Run on latest -nightly kernel
> output:
> Test assertion failure function basic_subtest, file pc8.c:608:
> Failed assertion: pc8_plus_enabled()
> Machine is not reaching PC8+ states, please check its configuration.
> Subtest basic: FAIL
> Checking the if the DRM resources match.
> Test assertion failure function drm_resources_equal_subtest, file pc8.c:639:
> Failed assertion: pc8_plus_enabled()
> Subtest drm-resources-equal: FAIL
> Testing batchbuffers.
> Test assertion failure function batch_subtest, file pc8.c:665:
> Failed assertion: pc8_plus_enabled()
> Subtest batch: FAIL
> Test assertion failure function i2c_subtest, file pc8.c:691:
> Failed assertion: pc8_plus_enabled()
> Subtest i2c: FAIL
> Stress testing.
> Test assertion failure function stress_test, file pc8.c:707:
> Failed assertion: pc8_plus_enabled()
> Subtest stress-test: FAIL
> Test requirement not met in function main, file pc8.c:764:
> Test requirement: (!(do_register_compare))
> Subtest register-compare: SKIP

That just means that your hsw ult isn't correclty set up for testing pc8. I've thought you're working together with Paulo to fix this.

> That just means that your hsw ult isn't correclty set up for testing pc8.
> I've thought you're working together with Paulo to fix this.

Paulo, 
Can you make any suggestion?

Closing old verified.