Ideation: Peter and I agreed on doing an instrument simulation using Leap Motion after a brief discussion, but later I found this fun project in Chrome music lab and came up with the idea of a choir conductor with Leap Motion. When meeting with Moon, he suggested that we combine two ideas together with the help of oscP5 library, so we ended up creating a band in which a conductor controls several musical instruments, each conductor/instrument running on a processing sketch with a Leap Motion.
Implementation: Due to time constraints, we only had one kind of instrument — guitar communicate with the conductor. Peter was in charge of the implementation of conductor, and I worked on the guitar. We both did some research and got a basic understanding of how oscP5 works.
I got the inspiration from Garageband and tried to draw a horizontal guitar interface as shown in the picture above, but during testing I found that the distance between strings were too close and the accuracy was very low. It was difficult to pluck certain string(s). Therefore I decided to switch to a vertical interface:
Since Leap Motion has a wide detection range (is more sensitive) on x-axis, the accuracy becomes higher and user experience is better. Strings have different stroke weights to imitate the real guitar.
Above was my first version of string triggering code. However, under this circumstance, messages will continuously be printed out (and sound file will be played again and again) if my finger stays in the range, which wouldn’t happen if we play a real guitar. To solve this problem, I changed the algorithm of determining string plucking.
The main idea is to figure out which strings are in between previous and current finger positions and play corresponding sounds in order. I denote six strings as index 0 to 5, then floats “start” and “end” are the corresponding indices of previous and current finger positions. If start is greater than end, it means the user swiped to the left, and strings in between are triggered from right to left, and vice versa. With this algorithm, holding the finger on a string doesn’t trigger the sound, making the experience more realistic.
I introduced z-index of Leap Motion to simulate the effect of user’s finger on/above the string. When z is less than the threshold (user’s finger moving closer to his/her body), the dot that shows the position of user’s index finger is white, indicating it’s above the string and won’t trigger any sound as it moves. When z is greater that the threshold, the dot becomes red and sounds will be triggered.
String vibration effect and sine waves were added to enhance the visual experience.
Video demo of the leap motion guitar:
Then I started to study oscP5 library. Thanks to “oscP5sendreceive” example by andreas schlegel and noguchi’s example, I created a very basic data communication program. mouseX and mouseY in osc1 (the left sketch) were sent to osc2 (the right sketch) and used to draw an ellipse on canvas.
Later I met Peter and he used oscP5tcp example for the conductor, so we decided to use tcp for both. Initially we planed to pass three parameters: volume modifier, frequency modifier and mute boolean value, but we met two problems. Because of the limitation of minim library, we couldn’t change the volume and frequency of a sound file. After several trials we managed to modify the volume using setGain() instead of setVolume(), but unfortunately we could do nothing about the frequency.
Index finger: swipes horizontally above the Leap Motion to pluck the strings
moves back towards body to move away from the strings
Dot on the screen: red – available, will trigger sounds if moving across strings
white – unavailable, either muted by conductor or too close to the body
Hand: moves up and down in certain instrument’s part of screen to increase/decrease the volume
grab to mute certain instrument
Feedback: Professor Chen brought up the “why” question, which I think is quite important and deserves further reflection. I agree with her that the idea that the conductor can actually have control over other users is great, and I can’t really answer the question why we need to have a leap motion simulating real instrument while the experience of playing physical instruments is good enough. I’m thinking of keeping the tech part but wrapping it with a different idea that is more interesting or more meaningful (though I have no idea for now).