Can we fix this by just changing from wl_fixed_t type to double in Weston for computing and storing the pointer location, or does the rounding to zero happen already in libinput?

This is actually a very good issue to notice *before* we stabilized the pointer-lock Wayland protocol extension, because it raises the question whether wl_fixed_t is appropriate for the relative motion events or do we need a more precise value type.

(In reply to Pekka Paalanen from comment #1)
> Can we fix this by just changing from wl_fixed_t type to double in Weston
> for computing and storing the pointer location, or does the rounding to zero
> happen already in libinput?

Libinput works with double's internally so nothing should have been shaved off.

> 
> This is actually a very good issue to notice *before* we stabilized the
> pointer-lock Wayland protocol extension, because it raises the question
> whether wl_fixed_t is appropriate for the relative motion events or do we
> need a more precise value type.

If deltas and coordinates in wl_fixed_t is enough for clients to work properly, we could just cut off the left over fraction from converting from double to wl_fixed_t and append it to the next event. This would fix the issue of small movements being dropped. What it wouldn't fix is clients receiving these small movements, and to fix that we either need a new larger data type in the Wayland wire protocol, or have a non-fraction and fraction values as separate ints or something.

Libinput until 58e0fe270 was accidentally rounding some data internally, but now as far as I can tell it's weston.

Pointer-lock is what a game would use to get "raw" mouse data?  I think there's a question as to whether that will be normalized at all by libinput - it may be the case that it'll come out the other side entirely on the left of the radix point, and this problem doesn't actually exist there.

Actually, I think even normalized, with the most ridiculous mouse I've seen (8200dpi) it's still like .049 units per "dot" without the accel curve's intervention.  If I'm doing my math right we're good to 102400DPI before we can't represent a dot in the absence of an accel curve.  I don't think anyone will argue a need for that.  It won't be me in any event.

Storing doubles internally and sending wl_fixed over the wire seems like it should be a solution to me.

Saving the rounded part and appending it to the next motion would fix it too, I think, but this seems a little uglier from an implementation perspective to me.

I guess right now clients are receiving a lot of motion events they don't need - they're even receiving motion events with no change when the round off occurs.  Should they only receive change above a certain threshold?

(In reply to Derek Foreman from comment #3)
> Libinput until 58e0fe270 was accidentally rounding some data internally, but
> now as far as I can tell it's weston.
>  
> Pointer-lock is what a game would use to get "raw" mouse data?  I think
> there's a question as to whether that will be normalized at all by libinput
> - it may be the case that it'll come out the other side entirely on the left
> of the radix point, and this problem doesn't actually exist there.

(FPS) Games will probably use a protocol called "wl_relative_pointer" that I will send to the list in the near future. The wl_relative_pointer will send motion events both accelerated and non-accelerated. The non-accelerated should probably be normalized before leaving the compositor at least as clients will not know what device an event comes from. I guess we could also pass along resolution information with the relative motion event and not normalize the non-accelerated coordinates, letting the client normalize, but not sure thats better. We could still not call this "raw" however, since some devices (touchpads) it is not reasonable to send the raw motion events.

> 
> Actually, I think even normalized, with the most ridiculous mouse I've seen
> (8200dpi) it's still like .049 units per "dot" without the accel curve's
> intervention.  If I'm doing my math right we're good to 102400DPI before we
> can't represent a dot in the absence of an accel curve.  I don't think
> anyone will argue a need for that.  It won't be me in any event.
> 
> Storing doubles internally and sending wl_fixed over the wire seems like it
> should be a solution to me.
> 
> Saving the rounded part and appending it to the next motion would fix it
> too, I think, but this seems a little uglier from an implementation
> perspective to me.

The reason for saving and appending the rounded part is that rounding before sending over the wire the client would still loose the delta when receiving relative motion events, resulting in the scene/pointer not moving for small movements.

> 
> I guess right now clients are receiving a lot of motion events they don't
> need - they're even receiving motion events with no change when the round
> off occurs.  Should they only receive change above a certain threshold?

weston only sends a new motion event if the coordinates of the events changed since last time. It does however send if the sub-pixel coordinates change, which I think it should continue to do.

(In reply to Derek Foreman from comment #3)
> Actually, I think even normalized, with the most ridiculous mouse I've seen
> (8200dpi) it's still like .049 units per "dot" without the accel curve's
> intervention.  If I'm doing my math right we're good to 102400DPI before we
> can't represent a dot in the absence of an accel curve.  I don't think
> anyone will argue a need for that.  It won't be me in any event.
> 
> Storing doubles internally and sending wl_fixed over the wire seems like it
> should be a solution to me.

I missed something. Why don't you see a problem with relative motion, when you filed this bug in the first place? Assuming both would use wl_fixed_t.

Or was the comment not about relative motion events?

(In reply to Jonas Ådahl from comment #2)
> If deltas and coordinates in wl_fixed_t is enough for clients to work
> properly, we could just cut off the left over fraction from converting from
> double to wl_fixed_t and append it to the next event. This would fix the
> issue of small movements being dropped.

I'd rather not go mangling the event data like this. I can't explain it, but it feels "dirty". You do not deliver all of a physical (raw) event, but let it somehow affect following events, that is, the physical vs. protocol events are not exactly N:1. I can't give an example when that matters. I do expect, that not all clients will linearly accumulate the relative motion. They could do something else, like control a 3D-rotation where the order of operations matters. I just have a feeling that "carrying fractions over" will shoot someone's foot in the future.

> What it wouldn't fix is clients
> receiving these small movements, and to fix that we either need a new larger
> data type in the Wayland wire protocol, or have a non-fraction and fraction
> values as separate ints or something.

Yeah. Let's go with this. For instance, int32_t for integer part plus uint32_t for fractional part. It takes the same amount of bytes as a double. It's simple. It's obvious. It's not surprising.

(In reply to Pekka Paalanen from comment #5)
> (In reply to Derek Foreman from comment #3)
> > Actually, I think even normalized, with the most ridiculous mouse I've seen
> > (8200dpi) it's still like .049 units per "dot" without the accel curve's
> > intervention.  If I'm doing my math right we're good to 102400DPI before we
> > can't represent a dot in the absence of an accel curve.  I don't think
> > anyone will argue a need for that.  It won't be me in any event.
> > 
> > Storing doubles internally and sending wl_fixed over the wire seems like it
> > should be a solution to me.
> 
> I missed something. Why don't you see a problem with relative motion, when
> you filed this bug in the first place? Assuming both would use wl_fixed_t.
> 
> Or was the comment not about relative motion events?

I had errantly assumed that relative motion would be passed "raw", or simply normalized but not accelerated.

> (In reply to Jonas Ådahl from comment #2)
> > If deltas and coordinates in wl_fixed_t is enough for clients to work
> > properly, we could just cut off the left over fraction from converting from
> > double to wl_fixed_t and append it to the next event. This would fix the
> > issue of small movements being dropped.
> 
> I'd rather not go mangling the event data like this. I can't explain it, but
> it feels "dirty". You do not deliver all of a physical (raw) event, but let
> it somehow affect following events, that is, the physical vs. protocol
> events are not exactly N:1. I can't give an example when that matters. I do
> expect, that not all clients will linearly accumulate the relative motion.
> They could do something else, like control a 3D-rotation where the order of
> operations matters. I just have a feeling that "carrying fractions over"
> will shoot someone's foot in the future.
> 
> > What it wouldn't fix is clients
> > receiving these small movements, and to fix that we either need a new larger
> > data type in the Wayland wire protocol, or have a non-fraction and fraction
> > values as separate ints or something.
> 
> Yeah. Let's go with this. For instance, int32_t for integer part plus
> uint32_t for fractional part. It takes the same amount of bytes as a double.
> It's simple. It's obvious. It's not surprising.

I do wonder if libinput could just calibrate the low end of its accel curve so that a single "dot" mouse motion would result in 1/256 units of motion.

Some of these gamer mice also have ludicrous poll rates (1kHz is common, I'm not making this up.  Please future reader don't feel the need to explain screen refresh rates to me - I did not design these mice.) is it worthwhile to try to keep the size of these events smaller than a double?

If that's not a concern, your int+frac notation seems to solve the problem with a maximum of overkill and a minimum of ambiguity. 

I just hope developers don't decide throwing away the frac part is ok :)

I have added a new bug about direct access to libinput events marshal by the compositor.

https://bugs.freedesktop.org/show_bug.cgi?id=90068

As you know most games or game apis do not use "relative pointers" they deal with the HID input APIs directly. The events are sent by the window manager.

Having access to the libinput events (or a similar API) directly will allow both relative motion events and also allow for high dpi mice. libinput can then be extended separately of Wayland to allow transmitting dpi, sensitivity, deadzone information and any anything.

If the compositor is not happy about forwarding the event due to window focus or a grab then it does not.

This is slightly separate of the confinement and locking issue but definitely preferably to "relative pointers" as it allows extensibility for 3D and VR whilst the compositor remains in control.

(In reply to Derek Foreman from comment #6)
> (In reply to Pekka Paalanen from comment #5)
> > (In reply to Derek Foreman from comment #3)
> > > Actually, I think even normalized, with the most ridiculous mouse I've seen
> > > (8200dpi) it's still like .049 units per "dot" without the accel curve's
> > > intervention.  If I'm doing my math right we're good to 102400DPI before we
> > > can't represent a dot in the absence of an accel curve.  I don't think
> > > anyone will argue a need for that.  It won't be me in any event.
> > > 
> > > Storing doubles internally and sending wl_fixed over the wire seems like it
> > > should be a solution to me.
> > 
> > I missed something. Why don't you see a problem with relative motion, when
> > you filed this bug in the first place? Assuming both would use wl_fixed_t.
> > 
> > Or was the comment not about relative motion events?
> 
> I had errantly assumed that relative motion would be passed "raw", or simply
> normalized but not accelerated.

I am not even sure the raw events should be normalized. As far as I know they are not on other platforms (but worth checking). 

Gamers expect that a higher dpi setting will move the mouse faster, and use the mouse dpi switch for sensitivity as Sniper mode or run mode etc. 

You do not want it that your 400 <-> 8200 dpi setting or Sniper mode button works in one way on Windows but then is slower/faster on Linux in the same game with the same settings.

> 
> > (In reply to Jonas Ådahl from comment #2)
> > > If deltas and coordinates in wl_fixed_t is enough for clients to work
> > > properly, we could just cut off the left over fraction from converting from
> > > double to wl_fixed_t and append it to the next event. This would fix the
> > > issue of small movements being dropped.
> > 
> > I'd rather not go mangling the event data like this. I can't explain it, but
> > it feels "dirty". You do not deliver all of a physical (raw) event, but let
> > it somehow affect following events, that is, the physical vs. protocol
> > events are not exactly N:1. I can't give an example when that matters. I do
> > expect, that not all clients will linearly accumulate the relative motion.
> > They could do something else, like control a 3D-rotation where the order of
> > operations matters. I just have a feeling that "carrying fractions over"
> > will shoot someone's foot in the future.
> > 
> > > What it wouldn't fix is clients
> > > receiving these small movements, and to fix that we either need a new larger
> > > data type in the Wayland wire protocol, or have a non-fraction and fraction
> > > values as separate ints or something.
> > 
> > Yeah. Let's go with this. For instance, int32_t for integer part plus
> > uint32_t for fractional part. It takes the same amount of bytes as a double.
> > It's simple. It's obvious. It's not surprising.
> 
> I do wonder if libinput could just calibrate the low end of its accel curve
> so that a single "dot" mouse motion would result in 1/256 units of motion.
> 
> Some of these gamer mice also have ludicrous poll rates (1kHz is common, I'm
> not making this up.  Please future reader don't feel the need to explain
> screen refresh rates to me - I did not design these mice.) is it worthwhile
> to try to keep the size of these events smaller than a double?

I think 1000hz plus would also begin to cause a problem because the extra calculations libinput is doing are essentially delaying the event delivery and causing extra cpu usage. You may have so much data buffered that the next time the client requests the raw events the compositor has not finished processing them.

> 
> If that's not a concern, your int+frac notation seems to solve the problem
> with a maximum of overkill and a minimum of ambiguity. 
> 
> I just hope developers don't decide throwing away the frac part is ok :)

SDL only supports mouse deltas as a single 32bit integer so the fractional part is already being dropped even as a double.
The same with UE4 it only supports raw mouse 32bit integers.

If you overflowed in one event you would probably have to just deliver the rest of the delta in the next event with the same timestamp.

(In reply to x414e54 from comment #8)
> I am not even sure the raw events should be normalized. As far as I know
> they are not on other platforms (but worth checking). 
> 
> Gamers expect that a higher dpi setting will move the mouse faster, and use
> the mouse dpi switch for sensitivity as Sniper mode or run mode etc. 
> 
> You do not want it that your 400 <-> 8200 dpi setting or Sniper mode button
> works in one way on Windows but then is slower/faster on Linux in the same
> game with the same settings.

fwiw, we only normalize the "default" setting, whatever that is. since we can't detect when the HW changes resolution, doubling the resolution through a switch on the mouse will provide deltas twice as high.

> > Some of these gamer mice also have ludicrous poll rates (1kHz is common, I'm
> > not making this up.  Please future reader don't feel the need to explain
> > screen refresh rates to me - I did not design these mice.) is it worthwhile
> > to try to keep the size of these events smaller than a double?
> 
> I think 1000hz plus would also begin to cause a problem because the extra
> calculations libinput is doing are essentially delaying the event delivery
> and causing extra cpu usage. You may have so much data buffered that the
> next time the client requests the raw events the compositor has not finished
> processing them.

I wonder if that's a real issue at this point. The calculations for normal mice are fairly limited (acceleration only) so before we start optimising here I'd like to see some real-world issues.
fwiw, one of the things the X server used to do for exactly this problem was to squash motion events together. easy since everything was already in absolute coords by then but some smarter squashing could probably be done by the compositor.

(In reply to Peter Hutterer from comment #9)
> (In reply to x414e54 from comment #8)
> > I am not even sure the raw events should be normalized. As far as I know
> > they are not on other platforms (but worth checking). 
> > 
> > Gamers expect that a higher dpi setting will move the mouse faster, and use
> > the mouse dpi switch for sensitivity as Sniper mode or run mode etc. 
> > 
> > You do not want it that your 400 <-> 8200 dpi setting or Sniper mode button
> > works in one way on Windows but then is slower/faster on Linux in the same
> > game with the same settings.
> 
> fwiw, we only normalize the "default" setting, whatever that is. since we
> can't detect when the HW changes resolution, doubling the resolution through
> a switch on the mouse will provide deltas twice as high.

I still have a few issues because this does not solve the problem that the delta is going to be different on Linux than on Windows for the same hardware. To get the same value I would have to find the dpi of the mouse that was used and then un-normalize the relative deltas.

Especially in something like an FPS they do not deal with mouse movement in physical units and expect an integer delta. 
Especially as mouse sensitivity is applied in the main engine code not in the platform specific code you are going to loose a lot of information.

Also I might be wrong but for example assuming 1000hz poll at 8400 dpi, normalized based on a 4200 default.
Any movement less than 1/4 of an inch per second would get each individual delta rounded by the application unless they were coalesced.
If the application rounded down then the mouse would not move at all and if it always round up then the game would assume the mouse moved 1 inch and not 1/4.

Is it actually part of the wayland spec that the mouse deltas would be normalized to 1000dpi for example? 
As it is not specifically mentioned, so there might be some window managers that do not normalize and some that do.

> 
> > > Some of these gamer mice also have ludicrous poll rates (1kHz is common, I'm
> > > not making this up.  Please future reader don't feel the need to explain
> > > screen refresh rates to me - I did not design these mice.) is it worthwhile
> > > to try to keep the size of these events smaller than a double?
> > 
> > I think 1000hz plus would also begin to cause a problem because the extra
> > calculations libinput is doing are essentially delaying the event delivery
> > and causing extra cpu usage. You may have so much data buffered that the
> > next time the client requests the raw events the compositor has not finished
> > processing them.
> 
> I wonder if that's a real issue at this point. The calculations for normal
> mice are fairly limited (acceleration only) so before we start optimising
> here I'd like to see some real-world issues.

Well it may or may no be an issue. If you are sure every compositor is going to be able to process and queue every mouse event within <1ms without adding additional latency then I guess it is not an issue.

> fwiw, one of the things the X server used to do for exactly this problem was
> to squash motion events together. easy since everything was already in
> absolute coords by then but some smarter squashing could probably be done by
> the compositor.

Yes this works fine for normal GUI use but for certain applications you may need to know the timestamp of each individual event or actually receive each individual event within 1ms. Maybe to draw the specific path the mouse took or to calculate who clicked first. So coalescing them is going to be a problem and seems to defeats the purpose of a high poll rate mouse. There would need to be some way to disable the coalescing or squashing.

This is why I was thinking towards something more of a real raw input system and just take the compositor out of the loop completely.

(In reply to x414e54 from comment #10)
> I still have a few issues because this does not solve the problem that the
> delta is going to be different on Linux than on Windows for the same
> hardware. To get the same value I would have to find the dpi of the mouse
> that was used and then un-normalize the relative deltas.

good news, I guess :) as of libinput 0.19 the unaccelerated deltas are as the hardware sends it. only the accelerated deltas are normalized (for mice with 100dpi and above, lower-res mice are a bit different).

main problem that forced our hand there was the rounding up of deltas, a 400dpi mouse normalized to 1000dpi will send a min delta of 2.5 and that's not discoverable. normalizing down is easier to deal with, even if current systems may drop the data.

> Is it actually part of the wayland spec that the mouse deltas would be
> normalized to 1000dpi for example? 
> As it is not specifically mentioned, so there might be some window managers
> that do not normalize and some that do.

it's not part of the spec, no. libinput does it and I expect virtually all general-purpose compositors to use it though.

> Well it may or may no be an issue. If you are sure every compositor is going
> to be able to process and queue every mouse event within <1ms without adding
> additional latency then I guess it is not an issue.

I'm not sure because I haven't measured it. feel free to do so though, if we find issues we can fix them. right now it's speculation though.

> Yes this works fine for normal GUI use but for certain applications you may
> need to know the timestamp of each individual event or actually receive each
> individual event within 1ms. Maybe to draw the specific path the mouse took
> or to calculate who clicked first. So coalescing them is going to be a
> problem and seems to defeats the purpose of a high poll rate mouse. There
> would need to be some way to disable the coalescing or squashing.

X has that too, for that reason. it would squash the pointer-moving motion events when needed (i.e. events come in faster than the server can process them) but still keep the history intact. XGetMotionEvents() gets you that history so you can rebuild the path.
 
> This is why I was thinking towards something more of a real raw input system
> and just take the compositor out of the loop completely.

any raw input event system usually runs into the issue of "how much non-raw will you support" and that's where it gets tricky, because suddenly you're not that raw anymore. example: most touchpads these days only have a single button, if you want right-click/tap/etc. functionality then you need more than a raw system. if you're happy to ignore that and just support mice, passing evdev fds to the client is the easiest approach - no input system as such required here.

Okay great if the unaccelerated deltas are not normalize that solves that problem!

I did think of another issue is that client relies on the timestamp of the click event to work out where the cursor was at that time but the timestamps are in milliseconds? 

Some mice are now reaching over 2000hz which means the button event would loose the precision to match to the correct motion event. 

Again more theoretical than practical but probably seems like we are already at the limit of 1ms with 1000hz mice.
  
> > This is why I was thinking towards something more of a real raw input system
> > and just take the compositor out of the loop completely.
> 
> any raw input event system usually runs into the issue of "how much non-raw
> will you support" and that's where it gets tricky, because suddenly you're
> not that raw anymore. example: most touchpads these days only have a single
> button, if you want right-click/tap/etc. functionality then you need more
> than a raw system. if you're happy to ignore that and just support mice,
> passing evdev fds to the client is the easiest approach - no input system as
> such required here.

Yeah I guess this is a fair point, there are also those "magic mice" which work the same for secondary click. 

But you could have the game engine implement full multitouch functionality and use gestures. You could use as many click buttons as fingers or separate the touchpad into four click areas. So it kinda seems a better way even for touch-pads. Though maybe conflicts/prevents with system specified gestures so you would probably want to have a system for an application to register gestures with Wayland or a way for Wayland to listen in for gestures and grab the fd back.

I was originally hoping for something like the Android MotionEvent which seems mostly just a java abstraction of evdev. You just get all axis motion routed through one event system and then have an enum for the "pointer" or device type e.g. finger, mouse, joystick etc. It is then really easy for games to support every type of input because it always falls back to the default implementation (e.g. simulating clicks as touches, mouse as a joystick axis, etc.).

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/wayland/weston/issues/60.