Mostly for my own benefit, I’ve decided to take some time and trace the entire signal flow in the new versions of the gloves from signal in all the way to audio out. Maybe someone will find this, or at least some parts of it, useful!
Starting from a super high level: the ultimate goal here is to build a custom physical interface that I can use to make music, because I hate keyboards (see my first post [here]). I’ve decided to use Arduino to process the physical inputs and SuperCollider to translate those inputs into sound.
Starting from an even higher level: here’s a list of some of the technology I’ve tried using to accomplish this. I’ve succeeded with each of these to a greater or lesser extent, and abandoned them for one reason or another. Here’s a short list…
|Max MSP (without gloves)||Popular, so lots of documentation||Just annoying. Dragging virtual patch cables.|
|MIDI||Simple to program
|Mostly good for soft synths
Need some extra drivers to translate Serial
|MIDI to eurorack modular||Sounds great||Cons of both MIDI + eurorack CV|
|CV to eurorack modular||Extremely customizable
Bulky/hard to transport
Can only control designated CV parameters
|Serial to SuperCollider||Completely customizable
Sounds great (when programmed well)
|Learning curve is a right angle
Serial communication is difficult on Windows
So, here we are at SuperCollider. The only limit to SC is, really, the amount of time I’m willing to spend programming it. I can customize literally anything, including the sounds, the way the parameters are being used, the parameters themselves. The learning curve was brutal but I had a leg up because I’m a software developer; I can’t imagine trying to start a project like this without that kind of background.
With that out of the way, let’s talk about the signal flow. Here’s roughly how it goes:
Step 1.1: Analog flex sensors into Arduino
I began by building my own flex sensors out of conductive fabric; you can find a link to the instructables from the initial Scalar Glove post. However, since I have ridiculously slender hands, the sensors were too thick and interfered with my guitar playing. So I upgraded to these professionally fabricated ones.
They’re very simple to wire up, and they send an analog signal that can be processed by Arduino. One complicating factor I found was that the resting value of the sensor would change every time I fired up the glove, so I had to add a calibration step.
From there it’s just a matter of writing a binary value out. While I could theoretically do a gradient value out, I’ve decided to just have the scalar glove output gate signals. I’ve formatted the Serial output in a way that it can be easily parsed by PureData:
SCA i m r p
SCA is a designator that says that this is the output from the Scalar Glove. The four values correspond to the index, middle, rink, and pinky gate signals as either 0.0 or 1.0. The spaces between them separate the values and the new line indicates the end of a message.
Code is here.
Step 1.2: BNO055 into Arduino Mega
The BNO055 uses the i2c protocol to communicate, and Adafruit provided a convenient library for processing the signal, so very little transformation is necessary. I’m outputting the X, Y, and Z axes as well as the linear acceleration values for X, Y, and Z; right now I’m not using the linear acceleration, but I intend to later so I’m leaving it there. The format of the output is similar:
VEC x y z acc_x acc_y acc_z
VEC signifying that this comes from the vector glove, and six floating point (decimal) values. X, Y and Z have values between 0 and 1; the acceleration values can go above 1 (I don’t remember the upper limit).
Code is here.
Step 2: PureData
The PureData step is only necessary because SuperCollider’s built-in SerialPort class doesn’t work on Windows, so I need something to translate the Serial output into something that can be processed by SuperCollider. PureData has built-in objects that can handle COM communication, so I just found the COM port of my Arduinos and opened those — port 3 for the Uno(Scalar Glove), port 4 for the Mega (Vector Glove). Someone else wrote a library, serial_print_extended, that handles this, so I’m using that rather than reinventing the wheel.
After reading the serial input, I’m using mrpeach, which provides a udpsend object that allows me to send a udp packet to localhost on a particular port. SuperCollider creates a server that communicates on that port and will receive these messages.
That image is a little out of date now because the Scalar Glove now only outputs 0.0 or 1.0, but the structure is the same. Code from this patch can be found in the previous SuperCollider posts.
Step 3: SuperCollider
As mentioned in step 2, SuperCollider creates a server (on localhost in my example, but you can create it elsewhere) that can receive the messages from PureData. The messages are received using the OSC (“Open Sound Control”, not oscillator, which is what I definitely thought at first) protocol. In PureData I’m appending a new header, “\scalar” or “\vector”, to the beginning of the message; then, in SuperCollider, I parse out the contents of that message and assign it to some global variables.
In SuperCollider, any one-letter variable is automatically global, so my i, m, r, and p, and x, y, and z are all global by default. Any of the additional ones need to be defined as global using a tilde (~x_bin). I decided to make all of the sensor information global because I want to access it in multiple places, most notably…
I’ve defined three synthesizers in SuperCollider and assigned them to my index, middle finger, and pinky. They each do different stuff and sound kinda weird. The one on my pinky triggers differently — I bend my pinky and release it once to turn it on, then it stays on until I bend my pinky again. The ones attached to my index and middle fingers are conventional gate-triggered envelope generators.
The way I keep these synths updated is with a SuperCollider “routine” that basically runs an endless loop. The global variables, like “z”, are being constantly updated with new data from the OSC messages; the job of the loop is to update the synths using the new data. This is accomplished using the .set function. In my best-programmed synth (I need to clean some of these up), I have input parameters of p_x, p_y, and p_z, aka parameter x, parameter y, and parameter z (I named them differently so I wouldn’t have the option of accidentally overwriting the global variables). Then every tenth of a second I call “set” on my synth, updating p_x with the current value of x, p_y with the value of y, and p_z with z:
n.set("p_x", x, "p_y", y, "p_z", z);
That routine runs indefinitely until I stop it.
Code is here.
And that’s how you do it! Sorry this is still a super dense information dump and I didn’t get into anything like how the client/server architecture of SC works — I don’t understand it all that well myself and there are other tutorials that go into more depth on that.
Here’s a taste of how it sounds.
I’m sure this isn’t the last update I’ll make to the patch;