Interaction Lab Final Project Documentation (Marcela Godoy) by Zersh

Date  5/20/2017

Instructor  Marcela Godoy

Student Name  Zexing Li Zersh

Project Name Coke Drum

Core Component  Capacitive Sensor

Software Arduino, Processing

Presentation Link 

http://prezi.com/hg_pgmz5c4fn/?utm_campaign=share&utm_medium=copy\

Description

My project is called coke drum. It is easy to understand by the name that my project is a self-designed drum consist of coke cans. Basically, to realize function, I used both Arduino and Processing, with the CapacitiveSensor library in Arduino and Sound library in Processing.

My project works in this way: when the user touches the coke cans, the Processing on the computer will play a sound of a drum beat. The drum has 6 coke cans in total, in other words, by hitting different coke cans, the user can generate 6 different sounds.

I designed 2 modes that the users can choose in the Processing. The first one is called Freestyle Mode. A background music will be played, and then the user can compose their own drum rhythm to match the background music. Even more, they could choose the background music as they wish.

The other mode is called Game Mode. So basically the mode works in this way: There are falling drum beats from the top of the screen. When it reaches the line, the user should hit the right can. If the user does not hit, or not hit at the proper time, or hit the wrong can, he will not get the score. If he hit the right can, he will get a point. When the game ends, the user would receive the score and compare the score with the full score.

Idea and Motivation

The initial idea of building an interactive drum was from one of my favorite games in the game center called Drum Master. But also, when I was playing the game, I found shortages in that game. One user only has one drum, and by hitting the center or the edge of the drum, it could only generate 2 sounds. This was quite a restriction for drum learners because the real traps have a number of drums. So for my project, I decided to make more than one drum, but 6 drums to similarize the situation for drum lovers. Moreover, I also consider another situation. Sometimes people just want to jump out of the ordinary path. So is this drum game, sometimes they may not have the mode to play the game or follow what the game tell them t o do. As a result, I created the freestyle mode, where the user can play whatever they what, even in a mad way, and generate loud noise!

Process

Build up Capacitive Sensor

The core component for my Arduino part is the capacitive sensor. Actually, when I have the idea of making an interactive drum, I considered a lot of about what sensors to use. I thought about using the touch sensor or pressure sensor. However, they are not perfect in my design because I knew that they are not sensitive enough to make an instrument. Then after I wrote about my final project proposal, My professor Marcela, she shared several videos about some creative and interactive instruments and told me the concept of the capacitive sensor. The capacitive sensor is using the physics principle of capacitive sensing in electrical enginnering. Basically, it is based on the technology of capacitive coupling, that can detect and measure anything that is conductive or has a dielectric different from air.

So in my case, the function of the capacitive sensors is to detect whether the user touches the conductive materials. Because the user’s hands are conductive, and the coke cans used in my project as drums are conductive as well, as a result, when the user touches the can, the capacitive coupling technology could detect this action, and correspond with the value changes in Arduino.

For my project, I began with building up one capacitive sensor and then three. It worked perfect at the first two try, However, when I wanted to make 3 to 6, problems appear, that in the Arduino, the sensors could not show correct values. I turned to Marcela for help, and she advised me to use the Grove Q touch sensor. We tried the Grove Q touch sensor, but somehow it still did not work. I tried different methods, and finally, I solved it by dividing the sensors into 2 inputs and 6 outputs. Initially, I made only 1 input and 6 outputs, perhaps this caused the sensors to be overloaded and could not sense and detect the changes. What I did is shown in the demo video that I divided the only one input to 2 inputs in the same breadboard and the same Arduino Kit. This time it worked perfectly, and it also solved another problem that it can detect the changes and action when the user wanted to hit 2 of the sensors at the same time, and the values showed correctly about the user actions.

Drum

For this part, it is not very hard to add the drum sounds. But the difficulty here is to make the sounds sound like real drum beats because sometimes the sound would be different when it played by the system player and by the processing sound library. Moreover, when I am going to show my project in the IMA show, I would use a speaker because of the rather noisy atmosphere, I should alter the sounds accordingly. This took me a very long time to make the slight changes to the volume, the pitch, the rate of the sounds to make them really sound like the sound generated from the real drums.

Game Development

Initially, the game is the goal for me to achieve, because I wanted to make the interactive game for both game lovers as well as drum/music lovers. But when I really encountered the situation, I had problems dealing with the data collection from the music. I was thinking of generating the game by the programming from the pitch or the initial drum beats from the audio. However, I thought I could do this by detecting the volume of the music, but the problem is that when the volume goes louder, it could not detect whether this is by the singer’s voice, or by the instrument or by the drum, and it could not automatically generate the game or it would become like a randomly generated game without following the beats in the music. The I was thinking of analyzing the data within the music, but it also has the problem.  Moreover, because the game should allow the user to react with some preparation, and that is why it needs some time for the beat to fall from the top screen to the line. The user really needs that time to find out which is the correct drum that the user should hit, and they should also find where the correct drum is.

As a result, I decided to make the game entirely by myself, and my understanding of the music, because I am the designer of the game. I listened to the sample piece so many time to find out where is the perfect time for the user to hit the drum, and I altered them into the correct time. This is a hard but fun moment because every time I hear the music again, I could make a slight progress to finish the entire piece. Finally, I succeeded making my game demo with the BGM from a cartoon called Konan.

Improvement 

First of all, I want to critique about the design of the appearance. Initially, I wanted to use different fruits. The reason I gave up the idea was because the fruits were easy to get expired, and they might no have a good looking after several days, or after the strong hit by the user, and become less attractive. I changed them into six coke cans, but at the same time, there is a problem when the users paly the game (which I observed in the IMA Show) that the user may be not very sensitive to find out which one is the first or which one is the number 4 can, and they need more time to find out because they are the same can. So after the IMA show, I thought that I might either put some clear and obvious signals to show the user of the numbered cans so that they can easily find out the match between the real cans and the drums in the game, or either I changed the cans into different cans so that the color and size and shape would be different, so they would become more distinguished for the users.

Second, I want to improve the “background music” in the freestyle mode. In the IMA show, because my “assistant” Max is a fan of BigBang, so she advised me to use their song “Bang Bang Bang” as the background music. However, not everyone knew the song, and not everyone liked the song, so if I have more time, I would develop for more choices of the background music for the freestyle so that everyone is able to choose their favorite song or the music played by their favorite band. And then, the users would have more passion for playing with their stars, and enjoy my drum.

Third, I wanted to solve the problems I met during the development of the game mode. As I said, it was very hard to generate the beats in the game, but I wanted to explore more with maybe machine learning to figure out a solution.

Reflection

The first thing I want to talk about is the use of the capacitive sensor. It is easy to use one capacitive sensor, but it is extremely hard to collaborate 6 of them. I tried so many times and spent so much time to make 6 of them work together. I was so excited at the time the Arduino received the correct values. In addition to my project, the capacitive sensors are a very interactive and magical communication approach. The capacitive sensor can take the place of other switches, buttons, and other controllers and it could be anything conductive. So for me, the capacitive sensor could become the extension of traditional controllers.

Then, it is very important to manage the time. For this semester, I took three IMA courses, and at the end of the semester, I need to deal with 3 final projects. I have not been through this situation before, but Marcela told me to start early when advised me after the midterm project. I think for my final project, I really started at the very beginning to think and rethink about my final project idea and change it along the way when I was designing the physical part. If I started late, I would not have enough time to build up the 6 capacitive sensors and challenge the game problems, and at the same time finishing the three projects. Moreover, IMA is a very collaborated major, not only with the classmates and teammates in the same class but also with the other classes within the major. When I was preparing for the background soundtrack for my animation course, it is just like I am composing my own music with the resources. This inspired me that for the freestyle mode I created, it would be more interesting to not only restricted to the drum beats but also to other interesting sounds.This is so common in the IMA field because all of the classes I took were so creative and inspirational, and this is why I loved this major, that it seems there is no borders, restrictions, limitations, and everyone can use their craziest imagination and wildest dreams to create whatever we what, and this is the charm for me to pursue the Interactive Media Arts, but also the goal for me to achieve.

IMA Show

1 2 3 4 5

Girls got the full score! Congrats!

Introducing my project

Freestyle Tryout

Game Tryout

Arduino Code:

#include <CapacitiveSensor.h>

/*
 * CapitiveSense Library Demo Sketch
 * Paul Badger 2008
 * Uses a high value resistor e.g. 10M between send pin and receive pin
 * Resistor effects sensitivity, experiment with values, 50K - 50M. Larger resistor values yield larger sensor values.
 * Receive pin is the sensor pin - try different amounts of foil/metal on this pin
 */


CapacitiveSensor   cs_2_3 = CapacitiveSensor(2,3);        // 10M resistor between pins 4 & 2, pin 2 is sensor pin, add a wire and or foil if desired
CapacitiveSensor   cs_2_4 = CapacitiveSensor(2,4);        // 10M resistor between pins 4 & 6, pin 6 is sensor pin, add a wire and or foil
CapacitiveSensor   cs_2_5 = CapacitiveSensor(2,5);        // 10M resistor between pins 4 & 8, pin 8 is sensor pin, add a wire and or foil
CapacitiveSensor   cs_9_6 = CapacitiveSensor(9,6);        // 10M resistor between pins 4 & 2, pin 2 is sensor pin, add a wire and or foil if desired
CapacitiveSensor   cs_9_7 = CapacitiveSensor(9,7);        // 10M resistor between pins 4 & 6, pin 6 is sensor pin, add a wire and or foil
CapacitiveSensor   cs_9_8 = CapacitiveSensor(9,8); 
void setup()                    
{   
   pinMode (11,OUTPUT);
   
   Serial.begin(9600);
}

void loop()                    
{
    long start = millis();
    long total1 =  cs_2_3.capacitiveSensor(30);
    long total2 =  cs_2_4.capacitiveSensor(30);
    long total3 =  cs_2_5.capacitiveSensor(30);
    long total4 =  cs_9_6.capacitiveSensor(30);
    long total5 =  cs_9_7.capacitiveSensor(30);
    long total6 =  cs_9_8.capacitiveSensor(30);
    if (total1 >2 || total2 >2 ||total3 >2 ||total4 >2 || total5 >2 ||total6 >2    ){
      digitalWrite (11,HIGH);
      }
    else {
      digitalWrite (11, LOW);}

    Serial.print(millis() - start);        // check on performance in milliseconds
    Serial.print("t");                    // tab character for debug windown spacing

    Serial.print(total1);                  // print sensor output 1
    Serial.print("t");
    Serial.print(total2);                  // print sensor output 2
    Serial.print("t");
    Serial.print(total3);                  // print sensor output 1
    Serial.print("t");
    Serial.print(total4);                  // print sensor output 2
    Serial.print("t");
    Serial.print(total5);                  // print sensor output 1
    Serial.print("t");
    Serial.println(total6);                // print sensor output 3

    delay(20);                             // arbitrary delay to limit data to serial port 
}

Processing Code

import processing.sound.*;
SoundFile soundfile;
SoundFile soundfile2;
SoundFile soundfile3;
SoundFile soundfile4;
SoundFile soundfile5;
SoundFile soundfile6;
SoundFile soundfile7;
SoundFile soundfile8;


int current;
int fall;
int timer;
int screen = 0;
boolean played = false;
int score = 0;



import processing.serial.*;

String myString = null;
Serial myPort;


int NUM_OF_VALUES = 7;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;      /** this array stores values from Arduino **/
boolean playSound = false;
boolean playSound2 = false;
boolean playSound3 = false;
boolean playSound4 = false;
boolean playSound5 = false;
boolean playSound6 = false;
boolean beat1= false;
boolean beat2= false;
boolean beat3= false;
boolean beat4= false;
boolean beat5= false;
boolean beat6= false;
boolean beat7= false;
boolean beat8= false;
boolean beat9= false;
boolean beat10= false;
boolean beat11= false;
boolean beat12= false;
boolean beat13= false;
boolean beat14= false;
boolean beat15= false;
boolean beat16= false;
boolean beat17= false;



void setup() {
  size(1050,600);
  background(100);
  background(0);
  soundfile = new SoundFile(this, "beat6.aiff");
  soundfile2 = new SoundFile(this, "beat2.flac");
  soundfile3 = new SoundFile(this, "beat7.mp3");
  soundfile4 = new SoundFile(this, "beat9.mp3");
  soundfile5 = new SoundFile(this, "bang.mp3");
  soundfile6 = new SoundFile(this, "beat3.mp3");
  soundfile7 = new SoundFile(this, "kenan.mp3");
  soundfile8 = new SoundFile(this, "beat5.wav");

  setupSerial();
}


void draw() {
  
  
  
  
  updateSerial();
  printArray(sensorValues);

  // use the values like this!
  
  
     background(100);
   fill(255);
   line(150,0, 150, 600);
   line(300,0, 300, 600);
   line(450,0, 450, 600);
   line(600,0, 600, 600);
   line(750,0, 750, 600);
   line(900,0, 900, 600);
   line(0,450,1050,450);
   
   ellipse(150,450,50,50);
   ellipse(300,450,50,50);
   ellipse(450,450,50,50);
   ellipse(600,450,50,50);
   ellipse(750,450,50,50);
   ellipse(900,450,50,50);
   stroke(200);
   fill(0);
   text ("1",135,465);
   text ("2",285,465);
   text ("3",435,465);
   text ("4",585,465);
   text ("5",735,465);
   text ("6",885,465);
   if (screen==0){
    fill(0);
    textSize(45);
    pushStyle();
    fill(255, 0, 0);
    rect(460, 365, 135, 50);
    rect(405,255,235,50);
    popStyle();
    textSize(45);
    
    text("GAME",465,407);
    text("FREESTYLE",410,300);
    current=millis();
    if(mousePressed){
      if(mouseX>=445 && mouseX<=595 && mouseY<=420 && mouseY >=350){
        screen = 1;
    }
      if(mouseX>=445 && mouseX<=595 && mouseY<=310 && mouseY >=240){
        screen = 2;
    }
    
    }
  }
  
  if (screen ==2){
    if (sensorValues[1] > 40 && playSound == false) {  
    ellipse(150,450,150,150);
    soundfile.amp(30);
    soundfile.play(2);
    playSound = true;
  }
  if ( sensorValues[1]<40 && playSound == true) {
    playSound = false;
  } 
  
  
  if (sensorValues[2] > 40 && playSound2 == false) {
    ellipse(300,450,150,150);
    soundfile2.amp(30);
    soundfile2.play();
    playSound2 = true;
  }
  if ( sensorValues[2]<40 && playSound2 == true) {
    playSound2 = false;
  } 
  
  
  if (sensorValues[3] > 40 && playSound3 == false) {   
    ellipse(450,450,150,150);
    soundfile3.amp(30);
    soundfile3.play();
    playSound3 = true;
  }
  if ( sensorValues[3]<40 && playSound3 == true) {
    playSound3 = false;
  } 
  
  if (sensorValues[4] > 40 && playSound4 == false) {
    ellipse(600,450,150,150);
    soundfile4.amp(30);
    soundfile4.play();
    playSound4 = true;
  }
  if ( sensorValues[4]<40 && playSound4 == true) {
    playSound4 = false;
  } 
  
  
  if (sensorValues[5] > 40 && playSound5 == false) {
    ellipse(750,450,150,150);
    soundfile5.amp(2);
    soundfile5.play();
    
    playSound5 = true;
  }
  if ( sensorValues[5]<40 && playSound5 == true) {
    //playSound5 = false;
  } 
  
  
  if (sensorValues[6] > 40 && playSound6 == false) {
    ellipse(900,450,150,150);
    soundfile8.amp(30);
    soundfile8.play();
    playSound6 = true;
  }
  if ( sensorValues[6]<40 && playSound6 == true) {
    playSound6 = false;
  } 
  }
  
  
  
  if( screen==1){
    
     text(timer,40,40);
     text(score, 515,100);
     
     timer=millis()-current;
     if (timer >1000 && played == false){
   soundfile7.play();
   soundfile7.amp(2);
   soundfile7.cue(0.5);
   played =true;}
   fill (0);
   ellipse(900,(timer-1700)*0.6,50,50);
   ellipse(900,(timer-2600)*0.6,50,50);
   ellipse(900,(timer-3500)*0.6,50,50);  
   ellipse(900,(timer-4400)*0.6,50,50);
   ellipse(900,(timer-5300)*0.6,50,50);
   ellipse(900,(timer-6200)*0.6,50,50);
   
   
   ellipse(750,(timer-7100)*0.6,50,50);
   ellipse(450,(timer-7550)*0.6,50,50);
   
   
   ellipse(150,(timer-8200)*0.6,50,50);
   ellipse(150,(timer-8900)*0.6,50,50);
   ellipse(150,(timer-9800)*0.6,50,50);
   ellipse(150,(timer-10700)*0.6,50,50);
   ellipse(150,(timer-11600)*0.6,50,50);
   ellipse(150,(timer-12500)*0.6,50,50);
   ellipse(150,(timer-13400)*0.6,50,50);
   ellipse(600,(timer-14100)*0.6,50,50);
   ellipse(450,(timer-14725)*0.6,50,50);
   
   //fill(255,0,0);
   //ellipse(750,(timer-15400)*0.6,50,50);
   //ellipse(750,(timer-15800)*0.6,50,50);
   //ellipse(600,(timer-16150)*0.6,50,50);
   //ellipse(900,(timer-16700)*0.6,50,50);
   //ellipse(600,(timer-17100)*0.6,50,50);
   //ellipse(300,(timer-17650)*0.6,50,50);
   
   //ellipse(450,(timer-18000)*0.6,50,50);
   //ellipse(450,(timer-18325)*0.6,50,50);
   //ellipse(450,(timer-18650)*0.6,50,50);
   
   
   //ellipse(300,(timer-19000)*0.6,50,50);
   //ellipse(150,(timer-19050)*0.6,50,50);
   //ellipse(300,(timer-19550)*0.6,50,50);
   
   
   
   
   
   
  fill(0);
  
  
  if (sensorValues[1] > 40 && playSound == false) {
    soundfile.amp(30);
    soundfile.play(3);
    if (timer>= 8900 && timer <= 9300 && beat9 == false ){
     score++;
     beat9 = true;}
     if (timer>= 9600 && timer <= 10000 && beat10 == false ){
     score++;
     beat10 = true;}
     if (timer>= 10500 && timer <= 10900 && beat11 == false ){
     score++;
     beat11 = true;}
     if (timer>= 11400 && timer <= 11800 && beat12 == false ){
     score++;
     beat12 = true;}
     if (timer>= 12300 && timer <= 12700 && beat13 == false ){
     score++;
     beat13 = true;}
     if (timer>= 13200 && timer <= 13600 && beat14 == false ){
     score++;
     beat14 = true;}
     if (timer>= 14100 && timer <= 14500 && beat15 == false ){
     score++;
     beat15 = true;}
    ellipse(150,450,100,100);
    playSound = true;
    
  }
  if ( sensorValues[1]<40 && playSound == true) {
    playSound = false;
  } 
  
  
  if (sensorValues[2] > 40 && playSound2 == false) {
    soundfile2.amp(30);
    soundfile2.play();
    ellipse(300,450,100,100);
  
    playSound2 = true;
  }
  if ( sensorValues[2]<40 && playSound2 == true) {
    playSound2 = false;
  } 
  
  
  if (sensorValues[3] > 40 && playSound3 == false) {
    soundfile3.amp(30);
    soundfile3.play();
    if (timer>= 8250 && timer <= 8650 && beat8 == false ){
     score++;
     beat8 = true;}
     if (timer>= 15425 && timer <= 15825 && beat17 == false ){
     score++;
     beat17 = true;}
    ellipse(450,450,100,100);
    playSound3 = true;
  }
  if ( sensorValues[3]<40 && playSound3 == true) {
    playSound3 = false;
  } 
  
  if (sensorValues[4] >40 && playSound4 == false) {
    soundfile4.amp(30);
    soundfile4.play();
    if (timer>= 14800 && timer <= 15200 && beat16 == false ){
     score++;
     beat16 = true;}
    ellipse(600,450,100,100);
    
    playSound4 = true;
  }
  if ( sensorValues[4]<40 && playSound4 == true) {
    playSound4 = false;
  } 
  
  
  if (sensorValues[5] > 40 && playSound5 == false) {
    soundfile8.amp(30);
    soundfile8.play();
    if (timer>= 7800 && timer <= 8200 && beat7 == false ){
     score++;
     beat7 = true;}
    ellipse(750,450,100,100);
   
    playSound5 = true;
  }
  if ( sensorValues[5]<40 && playSound5 == true) {
    playSound5 = false;
  } 
  
  
  if (sensorValues[6] > 40 && playSound6 == false) {
    soundfile6.amp(30);
    soundfile6.play(1.5);
    if (timer>= 2400 && timer <= 2800 && beat1 == false ){
     score++;
     beat1 = true;}
     if (timer>= 3300 && timer <= 3700 && beat2 == false){
     score++;
     beat2 = true;}
     if (timer>= 4200 && timer <= 4600 && beat3 == false){
     score++;
     beat3 = true;}
     if (timer>= 5100 && timer <= 5500 && beat4 == false){
     score++;
     beat4 = true;}
     if (timer>= 6000 && timer <= 6400  && beat5 == false){
     score++;
   beat5 = true;}
     if (timer>= 6900 && timer <= 7300 && beat6 == false){
     score++;
   beat6 = true;}
    ellipse(900,450,100,100);
    playSound6 = true;
  }
  if ( sensorValues[6]<40 && playSound6 == true) {
    playSound6 = false;
  } 
  
  if (timer >= 17500){
    text("YOUR SCORE IS         OUT OF 17",160,100);
  }
  
//screen one  
}



//void draw
}


void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 1 ], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = 'n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}



void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = 'n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), "t");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

One thought on “Interaction Lab Final Project Documentation (Marcela Godoy) by Zersh

  1. it was great you figured out about the capacitive sensor, and how to make it work by yourself Zersh!
    I think you still need to work more about the drums, but you know about that and I’m glad you noticed and pointed it out in your reflections. It was great that you participated of the show and got the feedback though!
    I’m happy that you didn’t give up and you worked harder for the final to improve all the things we talked about your midterm.
    Great job Zersh!

Leave a Reply