Click here to Skip to main content
13,258,479 members (46,045 online)
Click here to Skip to main content
Add your own
alternative version

Stats

50.9K views
1.1K downloads
21 bookmarked
Posted 18 Jun 2015

A USB HID Keyboard, Mouse, Touchscreen emulator with Teensy

, 13 Nov 2017
Rate this:
Please Sign up or sign in to vote.
This article describes how to use a Teensy 3.1 board from PJRC.com to simulate keyboard, mouse and touch screen USB HID devices at the same time. This allows to remote-control a computer through one USB cable.

Teensy 3.1

The Teensy 3.1 is a tiny PCB board (35 x 18 mm) with a lot of hardware functionality. It can be programmed with the same developer environment as an Arduino board. Among the features of the Teensy is the capability of the microprocessor to emulate USB devices.

Programming

To program the Teensy's processor you need a micro USB cable and you have to install:

  1. The Arduino compiler version 1.67. (Download)
  2. The TeensyDuino libraries (Download)
    Direct links to version 1.27: MAC, Linux 32 bit, Linux 64 bit, Windows

The source code has to be written in C and the firmware that you write for the board is named a "Sketch".

You have to configure the compiler like in this screenshot:

You can also use the option "Serial + Keyboard + Mouse + Joystick" which will add an USB virtual COM port to the keyboard, mouse and joystick devices.
 

Problems with the actual version TeensyDuino 1.27

When you have installed and configured the compiler you can already start writing code that emulates keyboard, mouse and joystick. You will find several example projects already installed on your harddisk (see menu "File" -> "Examples").

But there is a severe problem with the original code from TeensyDuino 1.27:

The mouse emulation does not work on Linux.
The cause is a severe design error in the Linux X11 server which is full of bugs and misdesigns.

The X11 server does not accept a USB mouse device that sends absolute coordinates.
But if you want to position the mouse at an exact location on the screen, the mouse positioning with relative coordinates is useless for several reasons:

  1. As you don't know the current location of the mouse pointer on the computer that is connected to the Teensy you would have to first move the mouse to the top left corner of the screen and start a relative movement from there, which is ugly.
  2. When the Teensy USB device tells the remote computer that the mouse has moved by 100 units it depends on the settings in control panel and on the operating system, how much the mouse really moves on the screen. It may move 53 pixels or 144 pixels!
  3. When "Mouse Enhancement" is enabled in control panel it becomes even worse: The distance of movement will depend additionally on the movement speed.
  4. Relative movement works on Linux only correctly in steps of 1 due to bugs/misdesigns in X11.
  5. Summary: If you want to position the mouse pointer exactly with relative movement: FORGET IT !

For that reason PJRC has implemented absolute mouse positioning in the versions TeensyDuino 1.17 and 1.18. This works better but then some users complained that they now cannot move the mouse anymore relatively which they needed for their projects.

So in the TeensyDuino versions 1.19 to 1.27 PJRC implemented a USB device that allows relative and absolute movement in the same HID device. But Linux does not accept this HID device and the mouse does not move with absolute coordinates anymore.

SUMMARY:
Whatever version of TeensyDuino you use, you will never have a code that satisfies the needs of all users and all operating systems.

My changes

And here comes my project. The code that you can download above in the ZIP file is a modification of the Teensyduino 1.27 code. Copy the 5 files in the ZIP to the folder ArduinoCompiler/hardware/teensy/avr/cores/teensy3
I made the following changes to the original code:

  1. The mouse device uses ONLY relative coordinates which will work on all operating systems (just like a "normal" real mouse).
  2. I added a new touch screen device for absolute pointer movement that also works on all operating systems.
  3. I did not change the commands of the Mouse class that you already are using in your current sketches, so you simply replace the files in the ZIP file above and recompile your sketch without changes.
  4. I completely rewrote the Mouse class, cleand up the code and added more functionality.
  5. I added lots of comments to the code that was sparsely commented before.
  6. I have commented out the joystick device which I don't need. If you need it, read my comments in usb_desc.h how to enable it again.
  7. I removed the SerialEmu device which I don't need. If you should need it, you have to change only a very few lines in usb_desc.h.

I hope that PJRC will integrate my code into future versions of TeensyDuino but currently USB stuff is in a feature freeze.

For the case that PJRC does not implement my code, you find also the original code from 1.27 in the ZIP file. So you can compare what I have changed and you can also compare the orginal 1.27 code with any future version to see what PJRC has changed. (I recommend Araxis Merge to compare text files)

After applying my changes and connecting the programmed Teensy to a Windows computer you will see in control panel 3 new USB devices:

USB device descriptors

To create a new HID device in Teensy I had first to learn a lot about USB descriptors. There is no really well written manual about this complex stuff in internet. If you read the documents from USB.org you will get shocked about the horrible writing quality. You will probably not understand anything.

You find some more user friendly tutorials at Beyondlogic and Eleccelerator but even these stay cryptic for a beginner.

A VERY great help for Windows users is the program USBlyzer which exists as a 33 day trial version with full functionality. This USB Sniffer shows all device descriptos of all connected USB devices and the data packets sent through the USB cable(s).

To be sure that what I do is correct I copied the device descriptors of a real Genius optical mouse and a real touch screen: ELO TouchSystems CarrollTouch 4500U. The keyboard device stays unchanged.

Here are the HID device descriptors that I finally implemented:

Mouse
Usage Page (Generic Desktop)
Usage (Mouse)
Collection (Application)
    Usage Page (Button)
    Usage Minimum (Button 1)
    Usage Maximum (Button 3)
    Logical Minimum (0)
    Logical Maximum (1)
    Report Count (3)
    Report Size (1)
    Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
    Report Count (1)
    Report Size (5)
    Input (Cnst,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
    Usage Page (Generic Desktop)
    Usage (Pointer)
    Collection (Physical)
        Usage (X)
        Usage (Y)
        Logical Minimum (-32767)
        Logical Maximum (32767)
        Report Size (16)
        Report Count (2)
        Input (Data,Var,Rel,NWrp,Lin,Pref,NNul,Bit)
    End Collection
    Usage (Wheel)
    Logical Minimum (-127)
    Logical Maximum (127)
    Report Size (8)
    Report Count (1)
    Input (Data,Var,Rel,NWrp,Lin,Pref,NNul,Bit)
End Collection
Touchscreen (version 'Pointer')
Usage Page (Generic Desktop)
Usage (Pointer)
Collection (Application)
    Usage Page (Generic Desktop)
    Usage (Pointer)
    Collection (Physical)
        Usage Page (Button)
        Usage Minimum (Button 1)
        Usage Maximum (Button 1)
        Logical Minimum (0)
        Logical Maximum (1)
        Physical Minimum (0)
        Physical Maximum (1)
        Unit (None)
        Report Size (1)
        Report Count (1)
        Input (Data,Var,Abs,NWrp,Lin,NPrf,Null,Bit)
        Report Size (1)
        Report Count (7)
        Input (Cnst,Ary,Abs)
        Usage Page (Generic Desktop)
        Usage (X)
        Usage (Y)
        Logical Minimum (0)
        Logical Maximum (10000)
        Physical Minimum (0)
        Physical Maximum (10000)
        Unit (None)
        Report Size (16)
        Report Count (2)
        Input (Data,Var,Abs,NWrp,Lin,NPrf,Null,Bit)
    End Collection
End Collection

As you see the mouse now allows relative movement of 16 bit (-32767 to 32767) rather than 8 bit (-127 to 127) in the original code which was far too few.

The touch screen uses coordinates from 0 to 10000 which allows to pass values in percent of the screen width/height with a precision of two digits after the decimal point. So to position the mouse at the bottom right corner of the screen the Mouse class sends the values 100.00% for X and Y which corresponds to a value of 10000 beeing sent through the USB cable. To position the mouse in the center of the screen the Mouse class sends 5000, 5000.

Note that it is completely irrelevant how many pixels the real monitor resolution is. This works on ANY monitor size because the operating system that receives the touch screen coordinates converts them into screen coordinates in pixels. All touch screens work like this.

I tested my project on Windows XP, 7, 8, 10 and on Linux (Suse, Ubuntu, Knoppix) where it works seamlessly.

First alternate Touchscreen Descriptor

Update february 2016:
Some people reported here on Codeproject that the above touchscreen descriptor does not work on OSX and Android. So I created an alternate descriptor that you must enable in the file usb_desc.h by changing a zero into a one in this line:

#define TOUCH_DEVICE   1

This will replace the above descriptor with this one:

Touchscreen (version 'Single Touch')
Usage Page (Digitizer)
Usage (Pen)
Collection (Application)
    Usage (Stylus)
    Collection (Physical)
        Usage (Tip Switch)
        Usage (In Range)
        Logical Minimum (0)
        Logical Maximum (1)
        Report Size (1)
        Report Count (2)
        Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
        Report Size (1)
        Report Count (6)
        Input (Cnst,Ary,Abs)
        Usage Page (Generic Desktop)
        Usage (Pointer)
        Collection (Physical)
            Usage (X)
            Usage (Y)
            Logical Minimum (0)
            Logical Maximum (10000)
            Physical Minimum (0)
            Physical Maximum (10000)
            Unit (None)
            Report Size (16)
            Report Count (2)
            Input (Data,Var,Abs,NWrp,Lin,Pref,NNull,Bit)
        End Collection
    End Collection
End Collection

I tested this descriptor successfully on Windows XP, 7, 8, 10 and on Ubuntu and Knoppix.

A Codeproject user reports that this descriptor works also on Android.
Please report results on OSX!
 

Second alternate Touchscreen Descriptor

You can enable the third touchscreen descriptor in the file usb_desc.h with:

#define TOUCH_DEVICE 2

This descriptor is a multi-touch descriptor, but it uses only one finger.

Touchscreen (version 'Multi Touch')
Usage Page (Digitizer)
Usage (Touch Screen)
Collection (Application)
    Usage (Contact Count Maximum)
    Logical Maximum (1)
    Feature (Data,Var,Abs,NWrp,Lin,Pref,NNul,NVol,Bit)
    Usage (Contact Count)
    Report Count (1)
    Report Size (8)
    Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
    Usage (Finger)
    Collection (Logical)
        Usage (Contact Identifier)
        Report Size (8)
        Report Count (1)
        Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
        Usage (Tip Switch)
        Usage (In Range)
        Logical Minimum (0)
        Logical Maximum (1)
        Report Size (1)
        Report Count (2)
        Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
        Report Count (6)
        Input (Cnst,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
        Usage Page (Generic Desktop)
        Usage (X)
        Usage (Y)
        Logical Minimum (0)
        Logical Maximum (10000)
        Physical Minimum (0)
        Physical Maximum (10000)
        Unit (None)
        Report Size (16)
        Report Count (2)
        Input (Data,Var,Abs,NWrp,Lin,Pref,NNul,Bit)
    End Collection
End Collection

ATTENTION:
This descriptor behaves very different from the previous two descriptors.
When you call Mouse.moveTo() the mouse pointer will NOT move on the screen.
(The only way to see the mouse pointer moving is inside the drawing area of MSPAINT.EXE on Windows 8 or 10.)

The mouse pointer may even disappear when moving it (Linux and Windows). The reason is that on a real touch screen you do not need the mouse pointer because you see where you are touching, so the operating system hides the mouse pointer.

When you touch with the simulated finger, the mouse pointer jumps to the new location and executes a click.

This descriptor works immediately on Windows XP, 8, 10 and Ubuntu and Knoppix.
But nothing will happen on Windows 7 if you do not enable it first. Go to Control Panel -> Pen and Touch:

After the first installation (Windows says: "The device is ready now") it may be necessary to disconnect and reconnect the Teensy once to make it work.

Try which of the three touchscreen descriptors works for you by setting TOUCH_DEVICE to 0, 1 or 2.

Programming Mouse Movement

The commands in your sketches can stay unchanged, but you get extended functionality:

Move the mouse pointer with relative coordinates (-32767 to 32767) via mouse device:

Mouse.move(x, y);

Move the mouse pointer with absolute coordinates in percent (0 to 10000) via touchscreen device:

Mouse.moveTo(x, y);

Example:
Mouse.moveTo(5000, 5000);
moves the mouse to the center of the screen (50.00%)

Mouse.moveTo(10000, 10000);
moves the mouse to the bottom/right corner of the screen (100.00%)

Move the mouse pointer with absolute coordinates in pixels on a screen of 1024 x 768 pixels resolution via touchscreen device:

Mouse.screenSize(1024, 768);
Mouse.moveTo(x, y);

Example:
Mouse.moveTo(1024/2, 768/2);
moves the mouse to the center of the screen

Mouse.moveTo(1024, 768);
moves the mouse to the bottom/right corner of the screen

As you see:
After calling Mouse.screenSize() the coordinates automatically change from percent to pixel.
You have to call Mouse.screenSize() only once (or never).

 

Special Case: Apple Macintosh

The previous code of PJRC had a workaround for Apple Macintosh which puts a fix margin of 7.5% around the usable area of the mouse movement resulting in 85% of the screen area. See this posting.

I don't know if a touchscreen on MAC also needs this mouse workaround because I don't have a MAC to test it.
For the case that it should be required, I implemented four optional parameters for Mouse.screenSize() that allow to define this margin, but in a more flexible way than the original (very cryptic) PJRC code:

Mouse.screenSize(widthPixel, heightPixel, // in pixel
                 marginLeft, marginTop, marginRight, marginBottom); // in percent

Example for MAC:
Mouse.screenSize(1024, 768, 750, 750, 9250, 9250);

This line defines a screen size of 1024 x 768 pixels and a margin of 7.50% around the usable area. (100.00% - 7.50% = 92.50%).

This information is based on PJRC.
Can a MAC user please confirm that this workaround is really required?

Clicking

Due to the fact that there is now a mouse device that has 3 mouse buttons (left, middle, right) and a touch screen that has only one "button" which is the finger of the user, I added the option to click either with the mouse device or the touch screen device:

Mouse.click(button);

Or you can also use
Mouse.press(button); 
and 
Mouse.release(button); 

where button may be MOUSE_LEFT, MOUSE_MIDDLE, MOUSE_RIGHT or TOUCH_FINGER

If you want to to hold a button down (for example for a drag and drop operation):

Mouse.set_buttons(1, 0, 0, 0); // left mouse button down
Mouse.set_buttons(0, 1, 0, 0); // middle mouse button down
Mouse.set_buttons(0, 0, 1, 0); // right mouse button down
Mouse.set_buttons(0, 0, 0, 1); // finger touching
Mouse.set_buttons(0, 0, 0, 0); // release all buttons and the finger

Mouse Scrolling

And finally you can use the mouse wheel (-127 to 127) via mouse device:

Mouse.scroll(movement);

Please read the plenty comments that I have put in usb_mouse.h.

More projects with Teensy

This is my third electronic project with a Teensy published on Codeproject. You may also be interested in:

License

This article, along with any associated source code and files, is licensed under The Apache License, Version 2.0

Share

About the Author

Elmue
Software Developer (Senior) ElmüSoft
Chile Chile
Software Engineer since 27 years.

You may also be interested in...

Comments and Discussions

 
QuestionQueation about Android. Pin
Member 1080296212-Nov-17 17:23
memberMember 1080296212-Nov-17 17:23 
QuestionThanks you Pin
Member 1343241927-Sep-17 0:13
memberMember 1343241927-Sep-17 0:13 
QuestionUSB Multitouch Screen log files Pin
Member 1333791320-Sep-17 4:09
memberMember 1333791320-Sep-17 4:09 
Questionmultitouch? Pin
Member 1314397721-Apr-17 12:08
memberMember 1314397721-Apr-17 12:08 
AnswerRe: multitouch? Pin
Elmue22-Apr-17 5:36
memberElmue22-Apr-17 5:36 
GeneralRe: multitouch? Pin
Member 1314397722-Apr-17 11:52
memberMember 1314397722-Apr-17 11:52 
GeneralRe: multitouch? Pin
Member 1314397723-Apr-17 5:09
memberMember 1314397723-Apr-17 5:09 
AnswerRe: multitouch? Pin
Elmue23-Apr-17 19:14
memberElmue23-Apr-17 19:14 
GeneralRe: multitouch? Pin
Member 1314397723-Apr-17 19:22
memberMember 1314397723-Apr-17 19:22 
AnswerRe: multitouch? Pin
Elmue23-Apr-17 19:29
memberElmue23-Apr-17 19:29 
QuestionAdvice on a somewhat similar project? Pin
graydetroit7-Mar-17 18:21
membergraydetroit7-Mar-17 18:21 
AnswerRe: Advice on a somewhat similar project? Pin
Elmue8-Mar-17 14:25
memberElmue8-Mar-17 14:25 
GeneralMessage Closed Pin
8-Mar-17 17:53
membergraydetroit8-Mar-17 17:53 
GeneralRe: Advice on a somewhat similar project? Pin
Elmue9-Mar-17 8:15
memberElmue9-Mar-17 8:15 
GeneralMessage Closed Pin
9-Mar-17 17:27
membergraydetroit9-Mar-17 17:27 
AnswerHow to add new features to the emulated device Pin
Elmue14-Mar-17 13:48
memberElmue14-Mar-17 13:48 
GeneralRe: How to add new features to the emulated device Pin
graydetroit3-Apr-17 16:59
membergraydetroit3-Apr-17 16:59 
PraiseWorks great! Pin
Member 1274851819-Sep-16 21:13
memberMember 1274851819-Sep-16 21:13 
QuestionOn touch and off touch events Pin
Member 126592022-Aug-16 7:17
memberMember 126592022-Aug-16 7:17 
AnswerRe: On touch and off touch events Pin
Elmue3-Aug-16 18:00
memberElmue3-Aug-16 18:00 
GeneralRe: On touch and off touch events Pin
phillds13-Oct-17 3:26
memberphillds13-Oct-17 3:26 
PraiseThank you Pin
Member 125089917-May-16 10:43
memberMember 125089917-May-16 10:43 
QuestionTeensy 3.2 + OSX + Android ... Touch not working Pin
Georg1731-Dec-15 17:22
memberGeorg1731-Dec-15 17:22 
AnswerRe: Teensy 3.2 + OSX + Android ... Touch not working Pin
Georg171-Jan-16 8:30
memberGeorg171-Jan-16 8:30 
AnswerRe: Teensy 3.2 + OSX + Android ... Touch not working Pin
Elmue2-Jan-16 18:00
memberElmue2-Jan-16 18:00 
GeneralRe: Teensy 3.2 + OSX + Android ... Touch not working Pin
Member 122987741-Feb-16 19:10
memberMember 122987741-Feb-16 19:10 
GeneralRe: Teensy 3.2 + OSX + Android ... Touch not working Pin
Elmue4-Feb-16 13:13
memberElmue4-Feb-16 13:13 
GeneralRe: Teensy 3.2 + OSX + Android ... Touch not working Pin
Member 122987744-Feb-16 23:20
memberMember 122987744-Feb-16 23:20 
AnswerRe: A new version will come soon Pin
Elmue13-Feb-16 4:46
memberElmue13-Feb-16 4:46 
GeneralRe: A new version will come soon Pin
Member 1229877413-Feb-16 13:04
memberMember 1229877413-Feb-16 13:04 
AnswerRe: New version is ready now Pin
Elmue17-Feb-16 12:56
memberElmue17-Feb-16 12:56 
GeneralRe: New version is ready now Pin
Member 1229877418-Feb-16 2:41
memberMember 1229877418-Feb-16 2:41 
GeneralRe: Android Pin
Elmue18-Feb-16 4:01
memberElmue18-Feb-16 4:01 
GeneralRe: Android Pin
Member 1229877418-Feb-16 17:12
memberMember 1229877418-Feb-16 17:12 
NewsRe: Android version 2 Pin
Elmue22-Feb-16 15:54
memberElmue22-Feb-16 15:54 
GeneralRe: Android version 2 Pin
Member 1229877423-Feb-16 19:50
memberMember 1229877423-Feb-16 19:50 
GeneralRe: Android version 2 Pin
Elmue24-Feb-16 3:51
memberElmue24-Feb-16 3:51 
QuestionMouse ok, Touch how? Pin
Georg1730-Dec-15 3:59
memberGeorg1730-Dec-15 3:59 
AnswerRe: Mouse ok, Touch how? Pin
Elmue31-Dec-15 4:44
memberElmue31-Dec-15 4:44 
GeneralRe: Mouse ok, Touch how? Pin
Georg1731-Dec-15 16:54
memberGeorg1731-Dec-15 16:54 
AnswerRe: New version is ready Pin
Elmue17-Feb-16 12:57
memberElmue17-Feb-16 12:57 
BugNo mouse movement on OS X Pin
Member 1169077925-Sep-15 6:37
memberMember 1169077925-Sep-15 6:37 
GeneralRe: No mouse movement on OS X Pin
Elmue26-Sep-15 6:41
memberElmue26-Sep-15 6:41 
GeneralRe: No mouse movement on OS X Pin
Member 1169077928-Sep-15 1:38
memberMember 1169077928-Sep-15 1:38 
GeneralRe: No mouse movement on OS X Pin
Elmue30-Sep-15 7:26
memberElmue30-Sep-15 7:26 
AnswerRe: New version is ready Pin
Elmue17-Feb-16 12:58
memberElmue17-Feb-16 12:58 
Questionteensy 2.0? Pin
Member 1190821013-Aug-15 18:32
memberMember 1190821013-Aug-15 18:32 
AnswerRe: teensy 2.0? Pin
Elmue14-Aug-15 5:41
memberElmue14-Aug-15 5:41 
GeneralRe: teensy 2.0? Pin
Member 1190821014-Aug-15 8:39
memberMember 1190821014-Aug-15 8:39 
GeneralRe: teensy 2.0? Pin
Elmue17-Aug-15 12:50
memberElmue17-Aug-15 12:50 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Terms of Use | Mobile
Web01 | 2.8.171114.1 | Last Updated 13 Nov 2017
Article Copyright 2015 by Elmue
Everything else Copyright © CodeProject, 1999-2017
Layout: fixed | fluid