Setup: - Intel Skylake or BXT HW - Git versions of: - kernel (supports atomic modesetting) - X server / Xwayland - Mesa - Weston (latest releases may be enough, but hasn't been tested) - New enough dependencies for above, so that modifiers are supported - X Benchmarks running with Vsync turned off, e.g. GpuTest v0.7 Test-case: - Start Weston with --xwayland - Start windowed Benchmark that runs on given HW at approximately 1.5x speed compared to monitor refresh rate (e.g. ~90FPS benchmark on 60Hz monitor) [1] - Monitor at which frequency Weston composites the application frames [2] - Do similar test also on X, e.g. under Ubuntu Unity Expected outcome: - Both X and Weston compositors update application frames to screen at monitor frequency (60Hz) Actual outcome: - Benchmark itself runs at about same speed under both windowing systems - X compositor does app frame updates at expected rate (60) - Xwayland causes Weston to do updates at half the rate (30) Notes: - Benchmark itself can even run a bit faster on Weston than on X, e.g. because compositor memory bandwidth overhead is smaller when it does less updates. It's just that users sees only every 3rd frame of them... - Xwayland should forward the frames immediately, but it's almost like frames were Vsynched to display refresh *twice* (with some extra delay added in-between to guarantee that FPS drops from expected 60 to 30 FPS) - I've seen similar issue with benchmarks running at 5-120FPS, 90 FPS is just where it's easiest to see. After benchmark goes >120 FPS, both X and Weston do compositor updates at 60 FPS. - The same issue could be there also when application is Vsynched, I haven't investigated that (Vsync makes it harder to notice / investigate) Most of the games (e.g. in Steam) don't support Wayland natively, so they get run through Xwayland. Games are often run with Vsync disabled, for higher FPS. This bug really ruins their performance, which hinders Wayland adoption. [1] First select GpuTest benchmark that is close enough, and then change window size to fine-tune the perf to your HW [2] I'm monitoring whole system updates by tracing user-space kprobes set to buffer swap functions in all 3D libraries. Simpler method could be starting Weston with Mesa LIBGL_SHOW_FPS=true option and redirecting its output to a file monitored with "tail -f".
Adding link to GpuTest. I've seen this also with other benchmarks, but GpuTest contains varied enough set of tests that it's easier to get suitable FPS rate where the issue is easier to investigate. On weaker HW, open source glmark2 might also contain some subtest (maybe terrain) that doesn't have too high FPS. I haven't tried any games myself, but as long as game reports and can provide stable FPS, and can be configured to run at suitable FPS range, it should be OK too. If game doesn't support disabling Vsync, Mesa vblank_mode=0 env var can be used for that.
Does adjusting the repaint window help? In /etc/xdg/weston/weston.ini: [core] repaint-window=N The default value of N is 7; adjusting it upwards to 15 or downwards to something like 3 could maybe help.
Would a test case that just calls *SwapBuffers() at a fixed frequency be good enough? And using glXGetSyncValuesOML or equiv to check the presentation jitter as exposed to the application. (E.g. piglit/tests/spec/glx_oml_sync_control/timing.c) Bonus points for the test harness probing the infrastructure itself to find the actual framerate.
(In reply to Daniel Stone from comment #2) > Does adjusting the repaint window help? In /etc/xdg/weston/weston.ini: > [core] > repaint-window=N > > The default value of N is 7; adjusting it upwards to 15 or downwards to > something like 3 could maybe help. Thanks, I'll test that tomorrow.
This is Weston issue. It happens also with native Weston applications. When testing this on Broxton... > [core] > repaint-window=N With a Wayland native 3D benchmarking application that updates its window at 65-70 FPS... Repaint value, Weston average update FPS: - 1: 30 FPS - 3: 30-35 FPS - 5: 35-40 FPS - 7: 40-45 FPS - 9: 40-50 FPS - 11: 45-55 FPS - 13: 50-60 FPS - 15: 60 FPS When application runs through Xwayland, the situation is pretty similar. When application runs below 60 FPS, there's a similar issue. Application running at 27 FPS, drops to 15 FPS with default repaint-window value "7", and it needs to be at least "13" for Weston updates to happen at 20 FPS (= what it should be). -> Seems that Weston repaint-window option should default to 15. What it actually does, and why it currently defaults to lower value?
For an explanation, see: http://ppaalanen.blogspot.com/2015/02/weston-repaint-scheduling.html The GPU is traditionally a single task at a time, non-preemptive, sequential processor. The application and the compositor compete of the GPU. If an application is not throttled to the frame callback, it's possible it will start a new frame before it has given the compositor a chance to get the current frame on screen. When this happens, the compositor has to wait for the app's new frame to finish before the compositing task can run, and only after that the compositor can actually show the frame on screen. Because Weston (all Wayland compositors, ideally) only repaints at most once for each refresh and always syncs to vblank, it predicts what is the next vblank it could hit, and makes a delay so that it will have repaint-window milliseconds of time to submit and finish the composite GPU job. If other GPU jobs make the compositor miss the vblank, it'll postpone the presentation by a whole refresh period before weston even considers repainting again. So the first thing is to make the application "sync to vblank", IOW, make it repaint only as a response to the frame callback events and see what effect that makes. repaint-window should be the lowest possible value where the compositor still makes the vblank deadline, to reduce application-to-screen latency.
Unfortunately it seems that Mesa LIBGL_SHOW_FPS=1 doesn't work with Weston or glmark2, so one needs to use some other means to measure FPS (like I do). (In reply to Chris Wilson from comment #3) > Would a test case that just calls *SwapBuffers() at a fixed frequency be > good enough? And using glXGetSyncValuesOML or equiv to check the > presentation jitter as exposed to the application. (E.g. > piglit/tests/spec/glx_oml_sync_control/timing.c) I wasn't able to reproduce the problem with simple-egl test, e.g. with: weston-simple-egl -b -o -u 15000 But that doesn't seem to use *SwapBuffers(). I'm not sure whether SwapBuffers() is the only reason why it doesn't trigger the issue. Maybe the test-case actually need to also fully use GPU, for this issue to be visible. These *do* reproduce the problem though: glmark2-wayland --annotate -b refract -s WxH glmark2-wayland --annotate -b terrain -s WxH (--annotate option shows the FPS on screen. You can get it e.g. on Ubuntu just with "sudo apt install glmark2-wayland".) Just increase the window size W & H parameters until reported FPS is somewhere between 70-100 FPS. If you have even a slightly faster machine, even "refract" test is too fast. But if you can get it to suitable FPS range, it works exactly like the cases I tested earlier (GpuTop v0.7 tests & internal Wayland benchmark). However, the "terrain" test is worse than the other test-cases. Increasing the repaint-window value from default 7 to 15, will improve Weston FPS from initial 30 FPS to higher value, but it still doesn't get close to the expected 60 FPS. -> I assume in that particular case, compositor would also require higher GPU priority than the test. Setting priorities is fairly new Mesa (and i915 kernel module) feature.
(In reply to Eero Tamminen from comment #7) > I wasn't able to reproduce the problem with simple-egl test, e.g. with: > weston-simple-egl -b -o -u 15000 > > But that doesn't seem to use *SwapBuffers(). > > I'm not sure whether SwapBuffers() is the only reason why it doesn't trigger > the issue. Maybe the test-case actually need to also fully use GPU, for > this issue to be visible. It certainly does call eglSwapBuffers: https://gitlab.freedesktop.org/wayland/weston/blob/master/clients/simple-egl.c#L568 > Just increase the window size W & H parameters until reported FPS is > somewhere between 70-100 FPS. So presumably it's running with eglSwapInterval(0) which doesn't block on frame callbacks at all? > -> I assume in that particular case, compositor would also require higher > GPU priority than the test. Setting priorities is fairly new Mesa (and i915 > kernel module) feature. We do use it: https://gitlab.freedesktop.org/wayland/weston/commit/b678befb6ed055e6c66466505d9195a3cebf8073
(In reply to Pekka Paalanen from comment #6) > Because Weston (all Wayland compositors, ideally) only repaints at most once > for each refresh and always syncs to vblank, it predicts what is the next > vblank it could hit, and makes a delay so that it will have repaint-window > milliseconds of time to submit and finish the composite GPU job. If other > GPU jobs make the compositor miss the vblank, it'll postpone the > presentation by a whole refresh period before weston even considers > repainting again. > > So the first thing is to make the application "sync to vblank", IOW, make it > repaint only as a response to the frame callback events and see what effect > that makes. With Vsync things are fine. This bug is about behavior when Vsync is disabled, and Weston being clearly worse at that, than X compositor(s) like Compiz. -> I updated the title to be more specific about this. Where this would matters: * All (3D) benchmarks run with Vsync disabled, and often games are run with Vsync disabled too. * Those are often run in fullscreen where it wouldn't matter if compositing were disabled, but Weston has regressed in that with modifier support, see bug: 106732 As there's a workaround, and bug 106732 will fix the main issue with this, I changed this to low/minor. > repaint-window should be the lowest possible value where the compositor > still makes the vblank deadline, to reduce application-to-screen latency. Ok, so prioritizing compositor frames wouldn't help, only repaint value does. Do you happen to know, is Compiz (clearly) better at this because it uses higher latency value? Or does it adapt the value based on previous application frame timings?
weston-simple-egl is also extremely fast to render (one triangle!), it doesn't put any load on the GPU. If you don't throttle it at all, on my intel desktop it runs at 12000 fps in its default size. So it makes sense it cannot reproduce the problem, because it cannot block the compositor in its GPU task. Until we can have the GPU preempted when the compositor's high-priority GPU job becomes runnable, I don't think the situation can get any better for abusive (SwapInterval=0) applications.
(In reply to Eero Tamminen from comment #9) > * All (3D) benchmarks run with Vsync disabled, and often games are run with > Vsync disabled too. If one benchmarks the GPU, why does it matter what goes on the screen? You cannot update the screen any faster than its refresh rate anyway. If you are benchmarking the full gfx stack all the way to screen, then run the application with vsync and if it consistently maintains the full refresh rate, measure the GPU load to get a performance number. I have a feeling that games are just poor. It's physically impossible to update the screen faster than the screen refreshes anyway. If screen update rate affects input latency for the game state, the game engine is badly designed. Traditionally it's been easy to just pump up the fps to work around all problems (and sell tons of expensive hardware) rather than doing something sensible like drawing at the right time instead of continuously hammering. Games need to be designed to have vsync on. > * Those are often run in fullscreen where it wouldn't matter if compositing > were disabled, but Weston has regressed in that with modifier support, see > bug: 106732 That is temporary from the beginning. We'll get it back. > As there's a workaround, and bug 106732 will fix the main issue with this, I > changed this to low/minor. > > > > repaint-window should be the lowest possible value where the compositor > > still makes the vblank deadline, to reduce application-to-screen latency. > > Ok, so prioritizing compositor frames wouldn't help, only repaint value does. > > Do you happen to know, is Compiz (clearly) better at this because it uses > higher latency value? Or does it adapt the value based on previous > application frame timings? I don't know what compiz does. I can guess though: - composite bypass for fullscreen works - it repaints ASAP - it tears in presenting - it tears in reading the application frame If you set repaint-window to 500 ms (anything equal or greater to refresh period), you will have Weston repainting ASAP. Usually that means as soon as DRM pageflip event comes, as your benchmarking app has posted a new frame by then. Btw. this scheme will become miserable as soon as you have more than one app updating. However, Weston will never tear on presenting to screen, and Weston will never sample from a client buffer that has not finished drawing (implicit fencing).
(In reply to Daniel Stone from comment #8) > (In reply to Eero Tamminen from comment #7) > > I wasn't able to reproduce the problem with simple-egl test, e.g. with: > > weston-simple-egl -b -o -u 15000 > > > > But that doesn't seem to use *SwapBuffers(). > > > > I'm not sure whether SwapBuffers() is the only reason why it doesn't trigger > > the issue. Maybe the test-case actually need to also fully use GPU, for > > this issue to be visible. > > It certainly does call eglSwapBuffers: > https://gitlab.freedesktop.org/wayland/weston/blob/master/clients/simple-egl. > c#L568 I see. I need to track eglSwapBuffersWithDamageEXT() instead. Thanks! (eglSwapBuffers() is called exactly 3 times, and only on startup.) > > Just increase the window size W & H parameters until reported FPS is > > somewhere between 70-100 FPS. > > So presumably it's running with eglSwapInterval(0) which doesn't block on > frame callbacks at all? All gfx performance benchmarks run (by default) with Vsync disabled. They cannot measure performance differences if they're limited by Vsync. :-) > > -> I assume in that particular case, compositor would also require higher > > GPU priority than the test. Setting priorities is fairly new Mesa (and i915 > > kernel module) feature. > > We do use it: > https://gitlab.freedesktop.org/wayland/weston/commit/ > b678befb6ed055e6c66466505d9195a3cebf8073 Weston logs: Failed to obtain a high priority context. I use latest git version of drm-tip kernel, Mesa, Weston, and weston-launch is suid root. What else is needed for context priority setting to succeed?
(In reply to Pekka Paalanen from comment #11) > (In reply to Eero Tamminen from comment #9) >> * All (3D) benchmarks run with Vsync disabled, and often games are run with >> Vsync disabled too. > > If one benchmarks the GPU, why does it matter what goes on the screen? You > cannot update the screen any faster than its refresh rate anyway. Some reasons: * Customers think that jerky output from benchmarks looks bad (especially if they're their own benchmarks) * When screen output doesn't visually correspond to what benchmark is reporting, people can start thinking that somebody is cheating * For 3D driver benchmarking one should use offscreen tests, but things modeling real applications often have only onscreen version. When compositor overhead is basically random (instead of 60Hz, or benchmark FPS), you cannot calculate/factor it * If compositor on some platform skips screen updates, that gives unfair performance advantage (number) on that platform > If you are benchmarking the full gfx stack all the way to screen, then run > the application with vsync and if it consistently maintains the full refresh > rate, measure the GPU load to get a performance number. While GPU load can give you some magnitude of performance, and for some really trivial 3D operations [1], it could even be used to estimate the performance, it's not really useful (accurate enough) for real 3D applications. [1] such as compositing, which is 100% memory bandwidth bound and for which one can actually calculate the expected performance. > I have a feeling that games are just poor. It's physically impossible to > update the screen faster than the screen refreshes anyway. If screen update > rate affects input latency for the game state, the game engine is badly > designed. Traditionally it's been easy to just pump up the fps to work > around all problems (and sell tons of expensive hardware) rather than doing > something sensible like drawing at the right time instead of continuously > hammering. > > Games need to be designed to have vsync on. Problem is that games have variable frame rate and people like to crank settings as high as possible. They also have higher than 60 FPS monitor. Solutions for avoiding getting only half the FPS with Vsync are either using FreeSync/Gsync and monitor capable of that, or disabling Vsync. Btw. How well Xwayland & Wayland compositors work with FreeSync (and Gsync)? (FreeSync has been part of DP & HDMI specs quite a while now.) > However, Weston will never tear on presenting to screen, and Weston will > never sample from a client buffer that has not finished drawing (implicit > fencing). Windows (at last with Intel drivers) lets rendering tear if application is both fullscreen and has disabled Vsync. I think this is reasonable, as compositor shouldn't then be involved. Disabling Vsync kind of means that application and/or user doesn't care about tearing.
(In reply to Eero Tamminen from comment #13) > (In reply to Pekka Paalanen from comment #11) ... > * If compositor on some platform skips screen updates, that gives unfair > performance advantage (number) on that platform ... > [1] such as compositing, which is 100% memory bandwidth bound and for which > one can actually calculate the expected performance. I understand all the above (including the ones I cut out in the quote) as just more reasons to run benchmarks with vsync on. If one does not actually display every frame, like you said, the compositor overhead will be a random factor since it will be processing some frames but not all. Another way is to increase the benchmark complexity until it just starts missing vblanks. But then again the compositor's design will play a part: a compositor could in its repaint check whether a client buffer is ready, and if it is not, postpone it to the next refresh. This is actually something I would like in Weston, so that applications could not delay screen updates arbitrarily and make e.g. pointer cursor motion choppy. So really, maybe benchmarks should be using fences and reading their completion timestamps to figure out how long it took to execute a GPU task. Since most GPUs still don't really pre-empt, that should be quite robust against compositor overheads. > > Games need to be designed to have vsync on. > > Problem is that games have variable frame rate and people like to crank > settings as high as possible. They also have higher than 60 FPS monitor. > > Solutions for avoiding getting only half the FPS with Vsync are either using > FreeSync/Gsync and monitor capable of that, or disabling Vsync. > > Btw. How well Xwayland & Wayland compositors work with FreeSync (and Gsync)? > > (FreeSync has been part of DP & HDMI specs quite a while now.) Variable refresh rate is an aid to let keep vsync on. I only recently saw a discussion about how variable refresh rate should be exposed in KMS UABI. Therefore I believe there is no UABI for it yet, which means that nothing on DRM KMS can make use of it yet. > > However, Weston will never tear on presenting to screen, and Weston will > > never sample from a client buffer that has not finished drawing (implicit > > fencing). > > Windows (at last with Intel drivers) lets rendering tear if application is > both fullscreen and has disabled Vsync. I think this is reasonable, as > compositor shouldn't then be involved. Disabling Vsync kind of means that > application and/or user doesn't care about tearing. I think that could be possible to implement, provided two things: - the compositor has a working composite bypass path and the fullscreen window hits it - the compositor actually knows the applications has "disabled vsync" (currently there is no way to know reliably) However, disabling vsync by setting eglSwapInterval=0 is not a reliable indication that the application actually wants to tear: Wayland applications can do it to be sure that eglSwapBuffers does not block unnecessarily while they still want vsync and handle frame syncing manually. Do you have ideas how to fix the issues you reported? I'm not sure what to do. An application posts a GPU task that is so long that it forces the compositor to miss the vblank, decimating the compositor's framerate. We could protect the compositor's framerate from being decimated by a) not using client buffers that are not finished drawing (with explicit fences), and b) have the compositor GPU task pre-empt any applications GPU tasks. We'd need both to ensure the compositor can update in time, but it won't change the fact that the compositor will only update once for each vblank. However, solution a) probably has the drawback that if the application is unthrottled and hammering the compositor with new frames, the compositor might never have a finished client buffer when it repaints because the client has replaced it with a new one already. So the compositor would have to hold on to some buffer to ensure it gets finished and can be shown. Then again, I question the sensibility of optimizing a system for unnatural benchmarks in the first place. I really want to push the world away from the "maximize fps at all costs" mentality that means that games waste power by drawing frames that will never be shown. VR would probably be a very good setting to improve the display pipeline: it demands vsync, high framerate (because the actual displays are high framerate), and low latency. I think would be much better to optimize for that instead of traditional benchmarks that don't care if a frame gets shown. With VR there is simply no room to waste power by drawing frames you never see.
Oh right, like Michel Dänzer said in IRC, the repaint-window should probably be dynamically adjusted based on performance. That would be a nice research project. Do you think that could be the solution, Eero?
(In reply to Pekka Paalanen from comment #15) > Oh right, like Michel Dänzer said in IRC, the repaint-window should probably > be dynamically adjusted based on performance. That would be a nice research > project. Do you think that could be the solution, Eero? It definitely seems something worth trying. (In reply to Pekka Paalanen from comment #14) [...] > Variable refresh rate is an aid to let keep vsync on. > > I only recently saw a discussion about how variable refresh rate should be > exposed in KMS UABI. Therefore I believe there is no UABI for it yet, which > means that nothing on DRM KMS can make use of it yet. Oh, I had no idea one needs Windows for that. :-/ While games are not really my thing [1], I would image gamers to want 144Hz monitor with "FreeSync" and configure the game so that he gets somewhere between e.g. 72-144Hz update frequency with variable Vsync rate. However, something like that should be done with fullscreen / compositor bypass. [1] I'm more into emulators (one of the maintainers for one), those also need Vsync differing from 60Hz (in my case 50 or 71). Because those frequencies are fixed during emulator run, changing whole display refresh rate would be one option there, but "FreeSync" would be nice there too.
(In reply to Eero Tamminen from comment #16) > (In reply to Pekka Paalanen from comment #15) > > Oh right, like Michel Dänzer said in IRC, the repaint-window should probably > > be dynamically adjusted based on performance. That would be a nice research > > project. Do you think that could be the solution, Eero? > > It definitely seems something worth trying. Now if someone had the will and the time... > While games are not really my thing [1], I would image gamers to want 144Hz > monitor with "FreeSync" and configure the game so that he gets somewhere > between e.g. 72-144Hz update frequency with variable Vsync rate. > > However, something like that should be done with fullscreen / compositor > bypass. Yes, very much. In the past, Mario Kleiner actually posted some tentative patches IIRC for allowing fullscreen composite-bypass path to reduce the display latency even further by essentially removing the compositor deadline for new content and flipping client buffers very close the vblank or something like that. But if the compositor does flips very close to vblank to reduce display latency, then that buffer has better finished drawing already, or it will definitely miss the vblank. Another problem with DRM KMS is that once you program a flip, you cannot cancel it and you cannot program another until it has finished. This means that programming a flip ASAP when a client provides new content means the future content for display is locked down early, which means there is no chance for the client to submit another frame closer to the same vblank. So while a Wayland compositor has a wl_buffer mail-box towards the client which the client can update at any time, the compositor needs to sample that mailbox some time before the vblank and it can do that at most once per display refresh. When to sample is defined by repaint-window. > [1] I'm more into emulators (one of the maintainers for one), those also > need Vsync differing from 60Hz (in my case 50 or 71). Because those > frequencies are fixed during emulator run, changing whole display refresh > rate would be one option there, but "FreeSync" would be nice there too. Right. We have had some tentative Wayland protocol bits to allow a client to suggest a refresh rate, which the compositor might honour by reprogramming the display video mode without VRR. VRR would be very interesting to support, even if only for fullscreen composite-bypass.
Just because VRR allows you to place content at quite arbitrary times, doesn't mean it's a great idea. Animations want an extremely stable and consistent frame rate in order to present smooth motion: if that framerate can be sped up without burning your CPU/GPU/lap then that's great, but presenting ASAP gives worse results than a fixed cycle. Even games don't want to 'present ASAP' with no idea of when ASAP actually is. They need to build their scene with a particular time in mind, and then show it to the user as close to that time as possible. They are a little less sensitive to the need for a constant refresh duration than animations are, but they still need to know the exact frame display time in order to not look horribly jumpy.
Btw. I'm fine if this bug is wontfixed; the issue is now documented here, there was good discussion about it, and there's the repaint-window workaround. Fullscreen bug 106732 & bug 106734 are the real issues which need to be fixed. I assume that with suitable context priorities, and high enough repaint-window, Weston compositor updates will be OK also for non-fullscreen benchmark windows (public benchmarks measure throughput, not latency, so high repaint-window value is fine for them).
-- GitLab Migration Automatic Message -- This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity. You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/wayland/weston/issues/115.
Use of freedesktop.org services, including Bugzilla, is subject to our Code of Conduct. How we collect and use information is described in our Privacy Policy.