NIME Final Instrument(Ronan)


For my final project, I wanted to make an instrument which can generate different noises based on the movements. I named it Music to the dance. As for the movements, technically, teh performer can move in whatever way they want. However, I choose the 2nd Radio Calisthenics For Primary School in China and it’s mandatory for all students from primary schools to high schools to do it every weekday. Personally, I think it’s quite special because not every country will require that. Besides, since I used to do it everyday, I’m familiar and feel more comfortable with it. And the movements are not really complicated, so it’s also easy for me to invite the audience to interact with me. When a movement is detected, the noise will be generated.


When I was presenting this idea, I thought I would use Makey Makey to trigger or loop the sound and make interactions with the audience. (For example, when teh performer and the audience touch, a sound will be played.) Besides, I also thought that I will use arduino and Max/MSP because we can easily control teh arduino with Max. However, after talking to professor Antonius, I decided to use my webcam and Max. I think it will be interesting to have no wires connected and I am amazed by what a webcam and a software can create together.


  1. I wanted to open and close my webcam with Max. Therefore, I used the object jit.grab to create a 320*240 matrice to contain my camera. The user is able to create any size of the container as they want. Then I passed two messages — open and close — to tell Max to either open or close the webcam.
  2. In order to detect the motion, I need to cut my whole camera into different pieces. I used git.scissors and set both rows and columns to 3, so I get a 3*3 = 9 pieces of camera. The user can also cut into any number of pieces as they want by simply changing the rows and columns.
  3. As for the motion detection, Jitter has an object called jit.op which can tell the difference between two matrices. This is really important because it will report the differences to the user. I forgot to use it the first time I tried it, so I could only get the real ARGB value instead of the differences.
  4. For more accurate detecting, I choose to use jit.3m, an object which can report the minimum, mean and maximum ARGB value (the order is ARGB as well). Then, I used unpack object to get the seperate ARGB mean value as a float number.
  5. If we directly use the number we get from the unpack object to generate noise, we will quickly find that the whole sound is too noisy, because as long as there is a motion, no matter how minor it is, the noise will be generated. Therefore, I used > object to filter out some small numbers so that the performer can only trigger the noise when they intend to.
  6. Instead of using the sound that I recorded, I used cycle, saw, tri and rect as different oscillators to generate different noise. At the beginning, I also used the keyslider to make random piano keys according to different movement. However, during the rehearsal, I was suggested to clean the sound a little bit and put the sound in more diverse areas so that when I moved, I could hear a clearly different sound. Therefore, I removed the keyslider.
  7.   I choose the 2nd, 4th, 6th, 8th pieces of the camera (shown in the image below). And I didn’t attach any oscillator to the center piece because basically every movement will trigger that part and that will make the whole sound a bit noisy.

Self-reflection of the NIME concert:

To be honest, I was really nervous before the concert because I’m afraid that 4 classes is not enough for me to develop an instrument that I can perform at the show. However, during the rehearsal on Thursday, my classmates and professors and fellows sincerely brought up suggestions for me. At the beginning, I planned to face the audience and make some dance movements. Because I am clear that I divided the whole camera into several parts, I had this thought in mind that I have to move within the range and make all four noises that I have. The problem with this thought is that I would keep looking at the camera and feel fear of moving freely. Therefore, Professor Antonius suggested me to turn around and not face the audience. This inspired me to use the blind-fold to cover my eyes so that I won’t be able to see anything. Also, I could just grab audience and give them the blindfold. In my opinion, this is the crucial part of my performance.

Actually, by covering my eyes, I get rid of two boundaries in my mind. The first is the actual boundary of my camera. Without this restriction, I won’t feel fear of getting out of the camera or not making every sound that I have anymore while in the rehearsal I kept looking at the computer or camera and didn’t have eye contact with my audience. The second boundary is the one between me as a performer and my audience. In the rehearsal, I was worried that the audience would refuse my invitation to try the instrument. However, with my eyes covered, I can grab anyone I touch and people definitely won’t criticize me because I’m “blind”! As a new NIMEr, I used to think that if I want my performance to stand out, I had to include as much technology as possible. However, after this concert, I start to think that we shouldn’t be limited to technology. And a performance is not necessarily less fun without a lot fancy techniques. This can be a really important lesson for me as an IMA student.

When reflecting my final performance, I remembered that Professor Antonius said that the technology is not a problem because we can get help from all the fellows. However, the interaction is more important because it gives the project a meaning and it determines who is this project built for and how other people can enjoy it as well. Therefore, although this simple piece of instrument (or maybe it couldn’t even be called as in instrument) is not that technological, I’m still proud of myself that my performance brought the audience fun. In general, as an IMA student, I always wonder what job i can get after graduation and which direction I should focus on. I’m grateful that this class gave me so much fun and confidence to make sure that I want to do visual designs and more things about music! In a word, this may be the most productive IMA semester I have: I built my Nature of Code project to be a sound visualizer and I created several logos for NIME organization (check this link to see all the logos) and finally I had my NIME instrument. It’s really interesting to see how a simple idea can generate so many results — I’m looking forward to more!


Source Code: (Some how I couldn’t copy and paste the patch so…)

Leave a Reply