Bug 104087

I'll need an evemu-record of the various event nodes that the device provides, thanks.

Created attachment 136052 [details]
evemu-record

OS: Ubuntu 17.10 64-bit
Display manager: X11
Input drivers: Synaptics / Evdev
Input device name: "SMART Technologies SMART IFP UNKNOWN"

(In reply to Peter Hutterer from comment #1)
> I'll need an evemu-record of the various event nodes that the device
> provides, thanks.

I added a record file. I hope, it contains the information you need.
I tested the six different SMART devices (see list above). Only one responded to events: one of the "UNKNOWN", but not the one called "Pen".

hmm, I replayed the events here and there's heaps of stuff coming out of libinput and it matches the evemu recording. Try it yourself with sudo libinput debug-events.

so I'm not sure what is going on here. How many device nodes are exposed by the smart board? The xinput output suggests it's 6 in total? How do those device nodes look with evemu-record?

And fwiw, if evemu-record doesn't see events, then the issue is in the kernel.

Thanks. I will do some more tests in the next days. But I need some time at work to do them.

It's right, there are six device nodes. My "theory" so far is:
SMART Technologies SMART IFP Pen id=9 > I don't know.
SMART Technologies SMART IFP id=10 > The touch screen itself.
SMART Technologies SMART IFP UNKNOWN id=11 > The black pen.
SMART Technologies SMART IFP UNKNOWN id=12 > The red pen.
SMART Technologies SMART IFP UNKNOWN id=13 > One potential additional pen.
SMART Technologies SMART IFP UNKNOWN id=14 > One potential additional pen.

Created attachment 136201 [details]
wayland-libinput-unknown1.evemu

Created attachment 136202 [details]
wayland-synaptics-unknown1.evemu

Created attachment 136203 [details]
x11-libinput-unknown1.evemu

Created attachment 136204 [details]
x11-synaptics-unknown1.txt

I did some more tests: I tried to write with the pen in Xournal and use it to open and minimize applications. Here are the results:

1) X11 + xserver-xorg-input-synaptics
The pen works fine. You can use it to write and interact with windows etc.

2) X11 + libinput
The pen doesn’t work, although evemu-record recognizes events.

3) Wayland + xserver-xorg-input-synaptics
The pen doesn’t work, although evemu-record recognizes events.

4) Wayland + libinput
The pen doesn’t work, although evemu-record recognizes events.

If you replay the events, it looks like everything works! But it really doesn’t work for the cases 2), 3) and 4). That’s a little bit strange.
I attached evemu-record files for the four scenarios.

Is there something more I can do? Maybe a video?

can you attach a hid-record output from the device as well please? https://bentiss.github.io/hid-replay-docs/

it's a bit strange that they all have an eraser (rubber) tool but no pen. might be the hid descriptor. Anyway, the libinput bug here is that the device doesn't send the right BTN_TOOL_RUBBER 1/0 on touch down/up, so libinput ignores the events. That's fixable, but let's look at the hid recording too to see if we can fix this in the kernel. a dmesg would be nice too, thanks.

btw, evemu records pure kernel events, so it doesn't matter which userspace stack you're running. No need to record multiple times.

I'll need another recording please to verify something:
Please unplug the device (or reboot) and make sure that no pen is touching the device or otherwise generating events. Start evemu-record and only then trigger an event and check the first few events in the recording. Repeat for all device nodes the device exposes.

What I'm trying to figure out is whether the device sends BTN_TOOL_RUBBER at any time or whether it's a dead axis. If the former, it's a similar to bug 102570. If it is never sent at all, then it's a bug we have to fix through other means.

Created attachment 136283 [details]
dmesg

Created attachment 136284 [details]
hid-x11-libinput

Here's the hid recording and the output of dmesg.

I'm not sure, if I understood your last comment right. Shall I try to record some events while the device is unplugged? How shall I trigger an event exactly? I'm sorry, I'm not a native English speaker.

By the way, the pen has an integrated eraser/rubber at the end, which - I think - doesn't work with linux.

You can't record events while it's unplugged, that would be a bit difficult :)
What I mean is: plug the device in but make sure the pen is not near the device. Start the recordings and only after that bring the pen into proximity.

However, looking at the hid recordings the question is answered anyway, the Eraser/Invert bits are never set. And the device doesn't have InRange so BTN_TOOL_PEN is never set.

All in all, this device isn't sending particularly useful evdev protocol and tbh I'd much rather have this fixed in the kernel than putting another exception into libinput. Right now we have a device that should be a pen but only exposes BTN_TOOL_RUBBER and never actually sets it. Benjamin, anything we can do here?

ping - benjamin?

Apologies for the delay. I didn't realize you expected me to shoot first.

So after a little bit of analysis and some google findings (Thomas, some will be obvious to you, but we need it documented in the bug report):
- the device is a 'whiteboard'[1] as in it's a multitouch screen where you have also physical markers (red/black) that are detected by the system (I don't think the markers are actually leaving a trail, it's the system that needs to draw the lines).
- on Windows, there is a need for installing a tool to handle this (never a good sign for Linux)
- the HID recordings show several things:
  * first, there is a multi-pen collection (not used according to the logs). It is as if the whiteboard could be switch into multitouch mode for pen (with only one report to forward all the pens) -> this is not handled by Linux for now, and this is the `Pen` node you see
  * then there is a classic multitouch collection for fingers (the second node, without any suffix)
  * then we have 4 individual Pen collections (named UNKNOWN because contrary to the others, there is no physical description attached to them, even though the application is 'Pen')
  * 2 collections are hidden because they are not populated: 
    . one regarding the 'Armature' bit which seems to be a placeholder for some custom protocol (there is no mechanical arm on the beast, isn't there?)
    . one mouse emulation which is ignored as expected, given the multitouch collection works.
- given that the 2 markers (logical Pens) do not have hovering, there is no InRange HID bit exported, and so BTN_TOOL_PEN is never exported.
- the HID collections for the Pen export Inverted. It seems the pens have an eraser function, so I think if you use the eraser side, this bit will be set correctly
- the HID collections for the Pen export BarrelSwitch. I don't think your pens have a fancy button on it, so I do not know why they say it supports an extra button
- the product page of the 7000[2] version says it supports 4 pens, so I guess both devices must share the same firmware.

So, this all comes down to the fact that it's the first device we see that doesn't support hovering for a tablet-like tool.
We might be able to fix this in hid-core, but I am afraid this will create a bunch of conditionals, especially given this particular device doesn't export correctly the physical bit of the Pens collections.

A solution would be to have a separate driver to handle it in a local environment, but given there is the multitouch collection in the middle, we can't extract it from hid-multitouch for now.

Depending of the urgency of the issue (talking about a few months from now), we can properly fix this in the kernel. I think you have a working solution (stick to X with wacom or evdev for the pens). If the issue needs to be fixed on Wayland soon, we should probably handle this in Wayland (there is no hovering, so a button touch means in range).

One last thought. Peter, is it possible in wayland to use 2 pens at the same time (I would say libinput/compositor should be fine, apps will not)? Because with the current Xorg workaround, I am pretty sure Thomas will be able to use only one pen at a time.


[1] https://downloads.smarttech.com/media/sitecore/en/pdf/products/ifp/smart_6000_series_with_iq_appliance.pdf
[2] https://downloads.smarttech.com/media/sitecore/en/pdf/products/ifp/ed-ifp7000-factsheet-en.pdf

Peter and Benjamin, thank you for your effort and time you spendend so far!

@Benjamin: I think you summarized all important facts and issues in relation to this device.
The company offered some Linux support some years ago, but newer whiteboards aren't officially supported anymore.
You can neither use two pens at the same time nor the eraser side with Linux.

It would be great to have some fixes in the kernel if it is technically possibly.
These kind of whiteboards are used more and more in schools.

Let me know if there is anything more I can do.

the wayland protocol is similar to the libinput API and is per-tool. There is no specific limitation to have only one tool on the surface as long as the client tracks the proximity events independently.

fwiw, last time I used one of these 10+ years ago the pens were just pens and the eraser device was a separate entity. Would be interesting to know if that one sends the right hid reports for BTN_TOOL_RUBBER. At least ont he linked ifp7000 the eraser is visible in the photo too.

But yeah, the list of issues wrong here means I won't put libinput hacks in to deal with all this. The device is clearly not working and needs some TLC in the kernel to be usable. Otherwise we're just making our own lives difficult because the first person to touch the kernel for this will break any hacks we put in libinput and then we have a combinatorial explosion of workarounds.

Sorry Thomas, you'll need to look at the kernel and start fixing it there. Once it has some better support, we can put small hacks in but this is just too much.