Taking on the job training to new heights
Taking on the job training to new heights
Move a virtual rock, with your real muscles
myRock is a demo I created in Unity that uses Myomo's myoelectric sensors. These sensors allow you to use the strength of your real muscles to lift and throw a giant virtual rock with the feeling real world weight and inertia. This is my first attempt at virtual earthbending. I can't show the input device I am using, but it is essentially an armband that reads the muscle activity of one's biceps and triceps. I flex my bicep muscle to lift the rock on the screen, and then flex my tricep to throw it. Below is a little demo of this.
This demo is made in Unity, and the rock is a 3D object with mass and the force of gravity acting on it. When I lift the rock, my lift force is proportional to how hard I flex my biceps. Then when I throw the rock, the pushing force is proportional to how hard I flex my triceps. This means I can lift and throw a rock with the same muscles I would use in real life. And because I am exerting real effort with my muscles, the rock feels like it has real world weight.
Right now, the game is played from a first person perspective, and the rock is thrown wherever the player character is facing. I made some versions of the demo with targets to hit for fun, but accuracy is still difficult at this point, like I imagine it would be lifting and throwing a real boulder. For the future I want to extend this so that any 3D object with collision in unity can be picked up and thrown by the player, and of course, I want to try this in VR.
Gaming for the impaired, a big first step
Bounce is a simple game that does a lot. It is played with the MyoPro, a powered orthotic device made by Myomo and used by patients with impaired arm movement to regain control of their arm. I created the game as a simple implementation of the tools I made to use the MyoPro with the Unity game engine. To create these tools, I took Myomo's existing code base for serial communication with the Myopro, and worked in c++ and c# to create an easy to use Unity plugin that allows any experienced Unity developer to use the Myopro as a game controller.
When I started on this project, Myomo had efficient c++ code for receiving streaming data from the MyoPro. I wanted to get this data into Unity so I could benefit from using a fully featured game engine. This kind of code interoperability is done with a Unity plugin. With a plugin, I wanted to create Unity-Developer facing code with an easy to use OO design that required as little knowledge about the MyoPro as possible, so that a new Unity developer could come in and start using the MyoPro as a controller without having to learn about bluetooth connections, serial communication, or command bytes.
I created the plugin so that for a Unity developer to connect to the MyoPro, they call a search function which populates a list of mac addresses, then pass that mac address to a connect function, which creates a new MyomoDevice object. Then, this object has a set of C# properties which will always return the latest values from the streaming data being handled in the background. The Unity developer does not need to maintain anything or parse any data; its all done by the plugin. Support for multiple connections in coming, and developers will just create a new MyomoDevice object corresponding to each new connection.
In the background, there is a lot of code that bas been abstracted out to make this possible. It starts with c++ classes in the Unity plugin that handle searching for and connecting to new devices. A new c++ object is created for parsing and storing the streaming data of each device. To expose these objects full of parsed streaming data to c# and Unity, pointers to these objects are passed to c# and wrapped in a c# object that a Unity developer can use. When Unity then asks for a property of this c# wrapper class, the c++ pointer is passed from c# back to c++, where functions can be called off the pointer to retrieve the latest streaming data. This data is then passed back up to the call to the c# property, and Unity gets the latest data from the device. Some properties even parse this data further to keep track of things that are not streamed back from the MyoPro, like the direction of movement of the arm.
The goal of Bounce, is to give MyoPro users an entertaining game that controls just like MyoPro, so that getting better at the game, means getting better at using the MyoPro, which in turn means helping users learn to use their arm again. In Bounce, the user raises their arm in the MyoPro to raise the paddle in the game, and lowers their arm in in the MyoPro to lower the paddle. Then using these controls they bounce a ball back and forth against an AI opponent.
Flying by the seat of your pants
Chair Simulator is a first person chair experience where turning a chair in real life turns your chair in the game. By turning to steer and leaning back in your chair to accelerate forward, you race across tops of skyscrapers to travel as far as you can for a high score!
Chair Simulator features the Seat Cushion Controller, the only input device needed to play the game. All of the hardware for detecting the movement of the chair is contained inside the cushion and sent to the game over USB. Inside the cushion, a Wii-Motion+ gyroscope and force are read by an Arduino microcontroller which calculates chair's position and sends this data to the game made in Unity3D.
I designed and programmed the Seat Cushion Controller, its communication with Unity3D, and its control in the game. My partner on the project created the procedural level generation, the soundtrack, and all 3D models. We completed this project in two weeks for the class Novel Interfaces for Interactive Environments at WPI.