Bug 99914

Created attachment 129848 [details]
evemu-record output

This is a short record of the issue at hand, and the following happens:

- Initially, the mouse is positioned near the center of my screen.
- I press the "northwest" key, which should move the mouse towards the top left corner: it moves to bottom right.
- I position the mouse back.
- I press the "northeast" key, which should move it towards the top right corner: it moves to the bottom right.
- I position the mouse back.
- I press "southwest", to move towards the bottom left: moves to bottom right instead.
- I position the mouse back again.
- I press "southeast", to move towards the bottom right: it does (as it did for the others), but moves to the very corner, instead of towards it.

On OSX, these keys move the cursor to the ~middle of the respective quadrants of the screen, as far as I know.

As another - hopefully helpful - datapoint, I modified my firmware to move the absolute mouse to specific coordinates, and trial & error suggests that ~960x550 is the bottom right corner of my screen.

Because the firmware reports its x/y max as 32k, this explains why the cursor always ends up on the bottom right corner. So I'm left wondering how I can tell libinput to map the virtual touchpad's 32k*32k area to whatever my resolution under X11 is?

ok, there are a couple of issues here. The first one is that the device has everything on the same event node. That's not something that libinput handles well because historically the kernel split up the event nodes for us into one for keyboard,one for touchpad, one for mouse (though we do handle mouse+keyboard on the same node).

For a touchpad to be identified as such you need BTN_TOOL_FINGER and BTN_TOUCH, both are missing.

The event stream is confusing with a lot of relative deltas of -2/-2, roughly 4ms apart. So that would move the pointer to the top-left in a very short time. Given the speed of the events this would likely see maximum acceleration and thus be even faster.

There are a few ABS_X/ABS_Y events but the max value there is 3072 which is a fraction of the 32k axis range. The pointer likely jumps a few pixels down but immediately moves back up due to the REL_X/REL_Y events. Probably too fast to be visible.


(In reply to Gergely Nagy from comment #1)
> - I press the "northwest" key, which should move the mouse towards the top
> left corner: it moves to bottom right.

indicates the screen is mounted the wrong way? the deltas are negative, so the screen is upside-down.

> - I position the mouse back.
> - I press the "northeast" key, which should move it towards the top right
> corner: it moves to the bottom right.

I don't see any positive REL_X/Y events, so the buttons are hooked up the wrong way.



(In reply to Gergely Nagy from comment #2)
> Because the firmware reports its x/y max as 32k, this explains why the
> cursor always ends up on the bottom right corner. So I'm left wondering how
> I can tell libinput to map the virtual touchpad's 32k*32k area to whatever
> my resolution under X11 is?

you can't, libinput doesn't know about that. The solution to this is to fix the firmware so the absolute range reflects the reachable axes and then let the rest of the stack DTRT.

(In reply to Peter Hutterer from comment #3)
> ok, there are a couple of issues here. The first one is that the device has
> everything on the same event node. That's not something that libinput
> handles well because historically the kernel split up the event nodes for us
> into one for keyboard,one for touchpad, one for mouse (though we do handle
> mouse+keyboard on the same node).

FWIW, if I change the firmware to still report a 32k*32k resolution, but limit movement to ~960*550, everything works fine. So it seems three things on the same node is handled well.

> For a touchpad to be identified as such you need BTN_TOOL_FINGER and
> BTN_TOUCH, both are missing.

I'll see how I can add these, thanks!

> The event stream is confusing with a lot of relative deltas of -2/-2,
> roughly 4ms apart. So that would move the pointer to the top-left in a very
> short time. Given the speed of the events this would likely see maximum
> acceleration and thus be even faster.

Those relative movements are me holding the mouse keys to reposition the cursor to the ~center of the screen. The touchpad code only sends the ABS_X/ABS_Y stuff.

> There are a few ABS_X/ABS_Y events but the max value there is 3072 which is
> a fraction of the 32k axis range. The pointer likely jumps a few pixels down
> but immediately moves back up due to the REL_X/REL_Y events. Probably too
> fast to be visible.

The pointer jumps to the bottom right corner, and stays there until I press the normal mouse keys to move it back to the center for the next test. All the REL_X/REL_Y events are direct results of me pressing keys. If you check the SYN_REPORT lines in the logs, the first REL_X/REL_Y event after an ABS_X/ABS_Y one usually has a >+1600ms delta.

> (In reply to Gergely Nagy from comment #1)
> > - I press the "northwest" key, which should move the mouse towards the top
> > left corner: it moves to bottom right.
> 
> indicates the screen is mounted the wrong way? the deltas are negative, so
> the screen is upside-down.

Err.. the deltas are negative, because I move the mouse cursor back to the center. If its in the bottom right corner, to move to the center, the deltas will need to be negative.

> > - I position the mouse back.
> > - I press the "northeast" key, which should move it towards the top right
> > corner: it moves to the bottom right.
> 
> I don't see any positive REL_X/Y events, so the buttons are hooked up the
> wrong way.

You don't see a positive REL_X/Y, because I did not have to move the mouse in those directions.

All the REL_X/REL_Y events are results of me moving the mouse cursors with mouse keys, as opposed to the touchpad keys.

> (In reply to Gergely Nagy from comment #2)
> > Because the firmware reports its x/y max as 32k, this explains why the
> > cursor always ends up on the bottom right corner. So I'm left wondering how
> > I can tell libinput to map the virtual touchpad's 32k*32k area to whatever
> > my resolution under X11 is?
> 
> you can't, libinput doesn't know about that. The solution to this is to fix
> the firmware so the absolute range reflects the reachable axes and then let
> the rest of the stack DTRT.

I reviewed the evemu-record, and it seems I sent a wrong one, where I had the firmware set to limit movement to a lower resolution, as an attempt to figure things out. I've done a new record now (will be attaching it in a bit), and it clearly shows the firmware sending the right coordinates, from its point of view.

What you can see in the record:

+----+----+
| NW | NE |
+----+----+
| SW | SE |
+----+----+

- NW key: middle of the first quadrant: 8191 is 32767 / 4. ABS_X/ABS_Y set to 8191.
- NE key: middle of the second quadrant: 24574 is 16k + 8k, the middle of the second quadrant. I suspect there's no ABS_Y sent, because that's already 8191.
- SW key: middle of the third quadrant: ABS_X moved back to 8k, ABS_Y to 24k.
- SE key: middle of the fourth quadrant: ABS_X and ABS_Y moved to 24k.

In all of these cases, the cursor ends up in the bottom right corner of the screen, because all of these values are higher than the ~960x550 which seems to be the limit.

So, it seems to me that the whole 32k is reachable, the correct events appear to be sent... so something in the stack is likely not doing the right thing, and that something is increasingly looking like not being libinput, as far as I see.

Perhaps xf86-input-libinput? Or shall I look further up, and experiment with a different DE? (I'm on GNOME3 at the moment, if that matters)

Or... can it be due to not having the BTN_TOUCH, and BTN_TOOL_FINGER things? Perhaps combined with being on the same node as a mouse?

Created attachment 129859 [details]
Updated evemu-record output

Some more data points: I've been staring at xf86-input-libinput, and noticed that it sets the axis max to 0xffff for absolute devices. So I changed the firmware to report 64k*64k, and then started to move the cursor with the touchpad functions towards the bottom right corner. Lo and behold, it reached the corner at 1980x1080, which happens to be the resolution of my screen.

So it appears that no scaling is done, even though https://cgit.freedesktop.org/xorg/driver/xf86-input-libinput/tree/src/xf86libinput.c#n68 suggests there should be. IIRC, I had to remove line 1344 (https://cgit.freedesktop.org/xorg/driver/xf86-input-libinput/tree/src/xf86libinput.c#n1344) for absolute events to get handled... so perhaps the problem is being on the same node and/or not having the BTN_TOUCH_* flags? (For the record, all we want from this touchpad emulation is absolute positioning, no clicking, palm detection, or anything like that - there is no real touchpad in play, just regular keyboard keys sending touchpad events)

(In reply to Gergely Nagy from comment #4)
> (In reply to Peter Hutterer from comment #3)
> > ok, there are a couple of issues here. The first one is that the device has
> > everything on the same event node. That's not something that libinput
> > handles well because historically the kernel split up the event nodes for us
> > into one for keyboard,one for touchpad, one for mouse (though we do handle
> > mouse+keyboard on the same node).
> 
> FWIW, if I change the firmware to still report a 32k*32k resolution, but
> limit movement to ~960*550, everything works fine. So it seems three things
> on the same node is handled well.

for all absolute events, libinput scales it based on the abs min/max values. without BTN_TOOL_FINGER, the device won't be recognised as touchpad and default to absolute pointer (similar to e.g. VM virtual mouse). That should work with libinput, but the libinput xorg driver (should) ignore the abs events then, it can only handle one type of pointer. once you add BTN_TOOL_FINGER it'll be a touchpad and you'll lose the rel bits (and key bits).

all that aside, run libinput-debug-events for testing, that's easiest to figure out what libinput does.
The output here for the new recording is:
-event7   DEVICE_ADDED     Shortcut Shortcut  seat0 default group10 cap:kp left scroll-nat calib scroll-button
 event7   POINTER_MOTION_ABSOLUTE  +1.99s     25.00/ 25.00
 event7   POINTER_MOTION_ABSOLUTE  +3.90s     74.99/ 25.00
 event7   POINTER_MOTION_ABSOLUTE  +6.13s     25.00/ 74.99
 event7   POINTER_MOTION_ABSOLUTE  +8.02s     74.99/ 74.99

The numbers are relative to 100, so 75 means 75% of the axis.

btw, you don't have an axis resolution set, that's not good for an absolute device.

The Xorg.log says:
  
  Feb 23 19:29:37 jelly /usr/libexec/gdm-x-session[10694]: (EE) libinput: Shortcut Shortcut: Discarding absolute event from relative device. Please file a bug

so the device is detected, but xf86-input-libinput cannot handle both relative and absoluute at the same time. That's a known bug, I tried to fix that the other way but aside from your device there isn't really a true need for it right now :)

 
> All the REL_X/REL_Y events are results of me moving the mouse cursors with
> mouse keys, as opposed to the touchpad keys.

right, I guess by now you realised that event recordings can only get one so far ;)

(In reply to Gergely Nagy from comment #6)
> Some more data points: I've been staring at xf86-input-libinput, and noticed
> that it sets the axis max to 0xffff for absolute devices.

yes, that's a side-effect of the xserver design, we need to have the full axis range, but libinput doesn't provide that one ahead of time. So we just map to a range of [0, 0xffff] instead and transform the coordinates to that (see xf86libinput_handle_absmotion). There shouldn't be a scaling bug since it works for libinput directly. The real problem is likely that the device gets set up as relative device and then the server gets confused when you send absolute events. I don't know off-heart what the server uses for scaling in that case.

Either way, a few well-placed ErrorF() should help (works like printf)

> so perhaps the
> problem is being on the same node and/or not having the BTN_TOUCH_* flags?

if it is just an absolute pointing device and not a real touchpad, you shouldn't have BTN_TOUCH or BTN_TOOL_FINGER, so you're good there. But having real+abs on one node just wont' work (yet) because of xf86-input-libinput limitations.

(In reply to Peter Hutterer from comment #7)

> for all absolute events, libinput scales it based on the abs min/max values.
> without BTN_TOOL_FINGER, the device won't be recognised as touchpad and
> default to absolute pointer (similar to e.g. VM virtual mouse). That should
> work with libinput, but the libinput xorg driver (should) ignore the abs
> events then, it can only handle one type of pointer. once you add
> BTN_TOOL_FINGER it'll be a touchpad and you'll lose the rel bits (and key
> bits).

Yeah, the abs events are ignored, that's why I tried to hack the xorg driver a bit, to at least let it through.

So, if I understand correctly, if I separated the rel & abs mouse stuff, into two different nodes, then both of them should work? (Assuming the absolute mouse part reports itself properly)

> all that aside, run libinput-debug-events for testing, that's easiest to
> figure out what libinput does.
> The output here for the new recording is:
> -event7   DEVICE_ADDED     Shortcut Shortcut  seat0 default group10 cap:kp
> left scroll-nat calib scroll-button
>  event7   POINTER_MOTION_ABSOLUTE  +1.99s     25.00/ 25.00
>  event7   POINTER_MOTION_ABSOLUTE  +3.90s     74.99/ 25.00
>  event7   POINTER_MOTION_ABSOLUTE  +6.13s     25.00/ 74.99
>  event7   POINTER_MOTION_ABSOLUTE  +8.02s     74.99/ 74.99
> 
> The numbers are relative to 100, so 75 means 75% of the axis.

Sounds like this part is allright then!

> btw, you don't have an axis resolution set, that's not good for an absolute
> device.

Mhm. Will try to figure out what these should be set to, thanks!

> The Xorg.log says:
>   
>   Feb 23 19:29:37 jelly /usr/libexec/gdm-x-session[10694]: (EE) libinput:
> Shortcut Shortcut: Discarding absolute event from relative device. Please
> file a bug
> 
> so the device is detected, but xf86-input-libinput cannot handle both
> relative and absoluute at the same time. That's a known bug, I tried to fix
> that the other way but aside from your device there isn't really a true need
> for it right now :)

There are two keyboards in the making with this firmware (the Shortcut[0], and the Keyboardio Model 01[1]), with hopefully more to follow. If you think it would be worth it, I can try digging into xf86-input-libinput, to try and add support for both abs and rel devices. If not, then I'll try to make the firmware report the two parts as two nodes.

(In reply to Peter Hutterer from comment #8)
> (In reply to Gergely Nagy from comment #6)
> > Some more data points: I've been staring at xf86-input-libinput, and noticed
> > that it sets the axis max to 0xffff for absolute devices.
> 
> yes, that's a side-effect of the xserver design, we need to have the full
> axis range, but libinput doesn't provide that one ahead of time. So we just
> map to a range of [0, 0xffff] instead and transform the coordinates to that
> (see xf86libinput_handle_absmotion). There shouldn't be a scaling bug since
> it works for libinput directly. The real problem is likely that the device
> gets set up as relative device and then the server gets confused when you
> send absolute events. I don't know off-heart what the server uses for
> scaling in that case.

Looks like it will just treat absolute coordinates as screen coordinates 1:1, but I'll dig deeper.

> Either way, a few well-placed ErrorF() should help (works like printf)

Thanks!

> > so perhaps the
> > problem is being on the same node and/or not having the BTN_TOUCH_* flags?
> 
> if it is just an absolute pointing device and not a real touchpad, you
> shouldn't have BTN_TOUCH or BTN_TOOL_FINGER, so you're good there. But
> having real+abs on one node just wont' work (yet) because of
> xf86-input-libinput limitations.

Understood. So, it appears this is not a libinput thing, but an xf86-input-libinput limitation. And there are two options going forward:

- Report the keyboard+mouse parts as one device, absmouse as another, and hope for the best.
- Improve xf86-input-libinput to handle the rel+abs on the same node case.

Either way, this issue is not with libinput, and can be closed, I suppose. Thanks a lot for your help, it's much clearer now how this whole thing works! I'll experiment with both options, and if it comes to that, will open an issue on xf86-input-libinput.

(In reply to Gergely Nagy from comment #9)
> (In reply to Peter Hutterer from comment #7)
> 
> > for all absolute events, libinput scales it based on the abs min/max values.
> > without BTN_TOOL_FINGER, the device won't be recognised as touchpad and
> > default to absolute pointer (similar to e.g. VM virtual mouse). That should
> > work with libinput, but the libinput xorg driver (should) ignore the abs
> > events then, it can only handle one type of pointer. once you add
> > BTN_TOOL_FINGER it'll be a touchpad and you'll lose the rel bits (and key
> > bits).
> 
> Yeah, the abs events are ignored, that's why I tried to hack the xorg driver
> a bit, to at least let it through.
> 
> So, if I understand correctly, if I separated the rel & abs mouse stuff,
> into two different nodes, then both of them should work? (Assuming the
> absolute mouse part reports itself properly)

yes. This is more of an X problem than a libinput problem because, well, 30 years ago a device couldn't be a pointer and a touchscreen at the same time ;) that's stereotyping it of course, but at least it's more fun than the real reasons

if you look at the xf86-input-libinput code you'll notice we split devices with keyboard & pointer on the same node into two X devices and route the events accordingly. same reason, basically.

> > so the device is detected, but xf86-input-libinput cannot handle both
> > relative and absoluute at the same time. That's a known bug, I tried to fix
> > that the other way but aside from your device there isn't really a true need
> > for it right now :)
> 
> There are two keyboards in the making with this firmware (the Shortcut[0],

haha, I thought Shortcut was just a leftover like "ToBeFilledIn" :)

> and the Keyboardio Model 01[1]), with hopefully more to follow. If you think
> it would be worth it, I can try digging into xf86-input-libinput, to try and
> add support for both abs and rel devices. If not, then I'll try to make the
> firmware report the two parts as two nodes.

I only worked on this a couple of days ago but didn't finish it. Please
continue from here, it has most of the infrastructure in place but needs the
actual testing and detail work:
https://github.com/whot/xf86-input-libinput/tree/wip/split-absrel-pointers


> Understood. So, it appears this is not a libinput thing, but an
> xf86-input-libinput limitation. And there are two options going forward:
> 
> - Report the keyboard+mouse parts as one device, absmouse as another, and
> hope for the best.

for general compatilibity, this is the best option

> - Improve xf86-input-libinput to handle the rel+abs on the same node case.
> 
> Either way, this issue is not with libinput, and can be closed, I suppose.
> Thanks a lot for your help, it's much clearer now how this whole thing
> works! I'll experiment with both options, and if it comes to that, will open
> an issue on xf86-input-libinput.

don't worry about it, I'll just reassign this one.

(In reply to Peter Hutterer from comment #10)
> > > so the device is detected, but xf86-input-libinput cannot handle both
> > > relative and absoluute at the same time. That's a known bug, I tried to fix
> > > that the other way but aside from your device there isn't really a true need
> > > for it right now :)
> > 
> > There are two keyboards in the making with this firmware (the Shortcut[0],
> 
> haha, I thought Shortcut was just a leftover like "ToBeFilledIn" :)

Heh :)

Oh, and I just noticed, that I didn't link them... doesn't matter much, but hey, to show that these devices will exist:

 [0]: http://shortcut.gg/
 [1]: http://keyboard.io/

> > and the Keyboardio Model 01[1]), with hopefully more to follow. If you think
> > it would be worth it, I can try digging into xf86-input-libinput, to try and
> > add support for both abs and rel devices. If not, then I'll try to make the
> > firmware report the two parts as two nodes.
> 
> I only worked on this a couple of days ago but didn't finish it. Please
> continue from here, it has most of the infrastructure in place but needs the
> actual testing and detail work:
> https://github.com/whot/xf86-input-libinput/tree/wip/split-absrel-pointers

Awesome, thanks! Will give it a go, much appreciated!

Got around to play a little with the `wip/split-absrel-pointers` branch, but it's not there yet. When plugging the keyboard in, Xorg.0.log says the following:

[ 13718.856] (II) XINPUT: Adding extended input device "Keyboardio Model 01" (type: MOUSE, id 16)
[ 13718.856] (II) XINPUT: Adding extended input device "Keyboardio Model 01" (type: KEYBOARD, id 19)
[ 13718.857] (II) XINPUT: Adding extended input device "Keyboardio Model 01" (type: TOUCHSCREEN, id 20)

The touchscreen part is new, and yay!

libinput-debug-events says:
-event25  DEVICE_ADDED     Keyboardio Model 01               seat0 default group7  cap:kp left scroll-nat calib scroll-button

As far as I saw, the "calib" capability is what I need.

However, when trying to use any absolute mouse functionality:

[ 13997.161] (EE) libinput: Keyboardio Model 01: Discarding absolute event from relative device. Please file a bug

Will try what happens if I remove the return from that branch - and a couple of other things.

Removing the return makes the mouse cursor jump to the same places as before. I'll try to debug things a bit further, to see if I can figure out what happens.

you'll need to add some bits to xf86libinput_pick_device() to make it pick the right subdevice. right now it merely searches for anything not keyboard, but you need to add the right conditions to search for the subdevice with the absolute capabilities. once that's in, the rest should mostly fall into place too.

Progress!

I managed to get things into a reasonable state, as in, all three types of subdevices (keyboard, mouse, absmouse) show up nicely, and they all show signs of working. The keyboard & mouse stuff works perfectly, the absolute mouse is so-so, the scaling seems to be a bit off. I'll explain in a bit!

Meanwhile, my current sources are at https://github.com/algernon/xf86-input-libinput/tree/wip/kaleidoscope/absrel - it's based on your wip branch + debian stuff, so I can easily test it. The debian parts do not modify anything under src/.

So, what happens is that the resolution is 32k*32k, and when I send an event with ABS_X and ABS_Y set to 8191, that positions the cursor to 1/4th of the screen vertically, and 1/2th horizontally (by the looks of it, anyway). What I'd expect, is that the cursor would be at 1/4th both vertically, and horizontally.

Mind you, I have a second screen plugged in (same resolution as my primary screen), but the device on the other end is powered off... is it possible that the second screen affects the scaling, even when powered off? I doubt it, but I have no other idea just yet.

I'll try to debug things further in the next few days.

one thing: the input driver does not (need to) care about output resolution,
you send everything relative to the input device. The server should take
care of the pointer positioning, including the one across the various
screens etc. I can't remember if dynamically disabled screens require client
intervention, but I don't think so. By default, the server behaviour is to
scale across all available screens though.

also, if you happen to use the binary nvidia driver, things are probably
weirder than if you don't :)

I am, indeed, using the binary NVidia driver. Will try without, thanks for the suggestion!

(In reply to Gergely Nagy from comment #17)
> I am, indeed, using the binary NVidia driver. Will try without, thanks for
> the suggestion!

Yep, that was it. The driver "remembered" that I had the other screen set up to be by the side of my primary screen, and even if the secondary was turned off, the resolution was taking that into account. Set the secondary to mirroring primary, and voila, everything works.

So with your WIP branch, and my tiny fixup, everything fell into place, and the keyboard+mouse+touchscreen combo on a single node works just fine.

Thanks a lot for your help!

Let me know if I can do anything to help get this into a release =)

Basic process is: finish the patch(es) and send them to the xorg-devel list for review. Once that's done we can merge it and it'll end up in the next release.

(In reply to Peter Hutterer from comment #19)
> Basic process is: finish the patch(es) and send them to the xorg-devel list
> for review. Once that's done we can merge it and it'll end up in the next
> release.

Patch sent (waiting moderator approval right now, will resend after subscribing, tomorrow or Monday, if it does not get approved before that).

https://lists.freedesktop.org/archives/xorg-devel/2017-March/053049.html

-- GitLab Migration Automatic Message --

This bug has been migrated to freedesktop.org's GitLab instance and has been closed from further activity.

You can subscribe and participate further through the new bug through this link to our GitLab instance: https://gitlab.freedesktop.org/xorg/driver/xf86-input-libinput/issues/7.