IMA Final Project: Taiko v2

Taiko v2


Author: Andrew

Professor: Dan

Date: May 17th

Abstract

A newly virtual drumming experience

A continuation of my midterm project, Taiko, a rhythm game where instead of using physical drums (cardboard boxes) I use the leap motion sensor to detect the user’s hand motions to evoke a response. In addition, music and beat timings were added to the game. I got most of my inspirations from the rhythm games I played before, such as DDR.


Process

I did not really have a brainstorming session, as most of the outlining was done during the midterm project, but I did not have the programming expertise yet to build the product just yet. After I attended the computer vision workshop, I figured out that the Leap Motion Sensor might prove to be immensely helpful in designing a drumming experience where the users isn’t limited by the just drumming on a surface.


Logistics:

IMG_20170520_003435 Screen+Shot+2017-05-20+at+12.34.43+AM

Most of the work was involved in the Arduino code. It proved tricky to obtain the timings for the using the minim BeatDetect Library because the beats were generated live and to play the music. In order to solve this problem, I saved the timings in csv format, thus requiring the song being played at least once in order to obtain the timings. After that each note is moved downward using translate and it’s state changed depending on one hits the notes. I added the code for the leap motion sensor to both detect the finger tap gesture or the z position in order to detect whether the user was in the hit-zone using relative to the laptop position. The values of BeatDetect itself were also sort of finicky since it largely depends on the peaks of the song.

I then sent the values of the Excellent score x 10 to Arduino, where it’s value was displayed on a 4 digit display.


Trials and tribulations

The most difficult of the project was getting a decent playing experience with the Leap Motion sensor. The midair drumming felt finicky at best, and most of the time, didn’t evoke a response within processing. It proved to not be as good of a experience as a physical drum! In addition, the drumming experience didn’t really feel like drumming at all.

Arduino also didn’t really play an significant role in this project at all, as the scoring system felt superfluous, I suppose leap motion provided most of the interaction the user needed to use the product effectively.


Conclusions

Overall, this final project was pretty rewarding. I developed a good understanding of Processing and Arduino over the course of this semester, and it was certainly a good primer into IMA. I think Interaction Lab provided me a good set of skills for me to possibly continue in IMA. I enjoyed my time this semester coding creative and interactive projects which wouldn’t normally be in computer science.

My code for the project is attached below.

import ddf.minim.*;
import ddf.minim.analysis.*;
import de.voidplus.leapmotion.*;
import processing.serial.*;

Serial myPort;

int value = 0;
int num = 0;
Minim minim;
AudioPlayer song;
BeatDetect beat;
ArrayList systems;

Table table;

float eRadius;

Score score;
Note[] notes;

LeapMotion leap;
boolean KickZone;

void setup()
{
  size(1200, 700);
  //fullScreen();
  minim = new Minim(this);
  song = minim.loadFile("anime.mp3", 2048);
  song.play();
  
  systems = new ArrayList();
  beat = new BeatDetect();

  //beat.detectMode(BeatDetect.FREQ_ENERGY);
    beat.setSensitivity(300);  

  //table = new Table();
  //table.addColumn("id");
  //table.addColumn("time");
  
  table = loadTable("new2.csv", "header");
  for (TableRow row : table.rows()) {
    
    int id = row.getInt("time");
   
    println(id);
  }
  score = new Score(table);
  
  ellipseMode(RADIUS);
  eRadius = 20;
  
  // Arduino
  printArray(Serial.list());
  
  myPort = new Serial(this, Serial.list()[2], 9600);
  
   leap = new LeapMotion(this).allowGestures();  // All gestures
}


void draw()
{
  background(0, 0, 0);
  
   //Particle System Creation
  
  for(int i = systems.size() -1 ; i >= 0 ; i--) {
   ParticleSystem ps = (ParticleSystem)systems.get(i);
   if(ps.done()) {
      systems.remove(i); 
   } else {
     ps.draw();
   }
  }
 
 // Drawing Buttons and Line 
 
  noStroke();
  fill(255, 255, 255);
  rect( width / 2 - 165, height - 140, 30, 30); // 125
  ellipse( width / 2 + 150, height - 125, 15, 15); // 125
  stroke(0); 
  noFill();  
  rect( width / 2 - 175, height - 150, 50, 50); // 125
  ellipse( width / 2 + 150, height - 125, 25, 25); // 125
  
  stroke(255);
  strokeWeight(5);
  fill(255, 255, 255);
  line(0, height - 125, width, height - 125);
  noStroke();

  // Update and display notes.
  
  score.update();
  score.display();
  
  int[] scores = score.getScores();
  textSize(26);
  fill(255, 0, 0, 100);
  text("PERFECT: " + scores[1], 10, 30);
  text("OKAY: " + scores[2], 10, 55);
  text("POOR: " + scores[3], 10, 80);
  text("MISSED: " + scores[0], 10, 105);
  
  
  stroke(255);
  
  // For beat detection
  beat.detect(song.mix);
  float a = map(eRadius, 20, 80, 60, 255);
  fill(60, 255, 0, a);
  if ( beat.isOnset() ) {
    background(60, 255, 0, a);
    eRadius = 80;
    num += 1;
    //println(num);
    //TableRow newRow = table.addRow();
    //newRow.setInt("time", song.position());
    //newRow.setInt("id", table.getRowCount() - 1);
  }
  
  ellipse(width/2, height/2, eRadius, eRadius);
  eRadius *= 0.95;
  if ( eRadius < 20 ) eRadius = 20;
  
  //===================================================
    
  // Leap logic
    
  for (Hand hand : leap.getHands()) {
    //hand.draw();
    PVector hand_position = hand.getPosition();
    
    println(hand_position.x);
    println(hand_position.y);
    println(hand_position.z);

    if (hand_position.z<50) {  
      KickZone = false;
    } else {
      KickZone = true;
    }
  }
  if(KickZone){
    println("hit");
    hit();
  } else {
    println("not hit");
  }  
}

void keyPressed() {
  if (key == CODED) { 
    if (keyCode == UP) {
      saveTable(table, "data/new.csv");
    }
  } 
  
  if (key == ' ') {
   hit();
  }
}

void hit() {
  
   println(score.notes[score.currNote].pos.y - score.totalDiff - height);
   Note currNote = score.notes[score.currNote];
   // If note hasn't been played, check for hit.
   if (!currNote.played) {
     float distFromLine = currNote.pos.y - score.totalDiff - height;
     if ( distFromLine > -100 && distFromLine < 50) {
       println("RIGHT ON");
       currNote.played = true;
       currNote.status = 1;
        int size = round(random(500,1000));
        systems.add( new ParticleSystem(size,random(.004*size,.006*size),random(0, width) ,random(0, 150)) ); 
     }
     if ( distFromLine < -100 && distFromLine > -200) {
       println("OKAY");
       currNote.played = true;
       currNote.status = 2;
     }
     if ( distFromLine < -200) {
       println("POOR");
       currNote.played = true;
       currNote.status = 3;
     }
   }
}


class Note {
   PVector pos;
   int id;
   float time;
   boolean played;
   boolean expired;
   color c;
   int status; // 0 - unhit, 1 - hit perfect, 2 - hit okay, 3 - hit poor, -1, not hit
   int permute;
   
   Note(int _id, float _time) {
     id = _id;
     time = _time;
     played = false;
     expired = false;
     c = color(60, 255, 0, 100);
     pos = new PVector(0, 0);
     permute = (int) random(50);
   }
   
   void update() {
     
   }
   
   void display() {
        //int timeOffset = 0;
        float mapped = map(time, 0, 2000, height, 0);        
        if (status == 0) {
          fill(c);
        } 
        switch(status) {
         case -1: fill(255, 0, 0);
                  break;
         case 0: fill(c);
                  break;
          case 1: fill(255, 255, 255);
                  break;
         case 2: fill(255, 255, 0);
                 break;
         case 3: fill(255, 0, 0);
                 break;
        }
        
        if (score.totalDiff < pos.y - height - 50) {
          expired = true;
          if (!played) {
            status = -1;
          }
        }
        if (id == 0) {
          println("Y POS:", pos.y);
          //println(song.position());
          println("TOTAL DIFF:", score.totalDiff);
          //println(time);          
        }
        
        //noStroke();
         println(permute);
        
       // if (id % 2 == 0) {
         if (permute % 2 == 0){
           //ellipse(width / 2 - 150, mapped, 15, 15);
           rect(width / 2 - 150 - 15, mapped - 15, 30, 30);
           pos.x = width / 2 - 150;
           pos.y = mapped;
        }
        else {
           ellipse(width / 2 + 150, mapped, 15, 15);
           pos.x = width / 2 - 150;
           pos.y = mapped;
        }
   }
}


// Class to create a Particle System
class Particle {
  int t;
  PVector p;
  PVector v;
  float r = 3.0;
  float g = 0.05;
  color c;
  
  Particle(float max_v, color inC) {
    p = new PVector(0,0);
    float a = random(0,TWO_PI);
    float vel = random(0,max_v);
    v = new PVector(vel*cos(a),vel*sin(a));
   
    int r = (inC >> 16) & 0xFF;  // Faster way of getting red(argb)
    int g = (inC >> 8) & 0xFF;   // Faster way of getting green(argb)
    int b = inC & 0xFF;          // Faster way of getting blue(argb)
    c = color( constrain( r+round(random(-30,30)) , 0 , 255),
               constrain( g+round(random(-30,30)) , 0 , 255),
               constrain( b+round(random(-30,30)) , 0 , 255));
  }
  
  void draw() {
    update();
    noStroke();
    fill(c);
    ellipse(p.x,p.y,r,r);
  }
  
  void update() {
    t++;
    v.add(new PVector(0,g)); // Accelerate
    p.add(v);                // Velocitate
  }
  
  boolean done() {
    if(random(0,t) > 50) {
       return true; 
    }
    
    return false;
  }

  
}

class ParticleSystem {
  ArrayList particles;
  PVector p; //position
  color c;
  
  
  ParticleSystem(int size, float spread, float x, float y) {
    p = new PVector(x,y);
    particles = new ArrayList();
    c = color(random(220)+35,random(220)+35,random(220)+35);
    
    for(int i = 0; i < size; i++) {
       particles.add(new Particle(spread,c)); 
    }
    
  }
  
  void draw() {
    pushMatrix();
    translate(p.x,p.y);
    
    for(int i=particles.size()-1; i >= 0 ; i--) {
      Particle p = (Particle)particles.get(i);
      if( p.done() ) {
         particles.remove(i);
      } else {
         p.draw();
      }
    }
    
    popMatrix();
  }
  
  boolean done() {
    if(particles.size() == 0 ) {
       return true; 
    }
    return false;
  }
  
}

class Score {
  Table t;
  float prevH = 0;
  float currH = 0;
  float totalDiff = 0;
  float diff = 0;
  Note[] notes;
  int currNote = 0;
  
  Score(Table _t) {
    t = _t;
    notes = new Note[t.getRowCount()];
    int rowNum = 0;
     for (TableRow row : table.rows()) {
      int time = row.getInt("time");
      notes[rowNum] = new Note(rowNum, time); 
      rowNum += 1;
    }
  }
  
  int[] getScores() {
    // get scoring
    int[] scores = {0, 0, 0, 0};

    for (int i = 0; i < notes.length; i++) {
      switch(notes[i].status) {
         case -1: scores[0] += 1;
                  break;
         case 1: scores[1] += 1;
                  break;
         case 2: scores[2] += 1;
                 break;
         case 3: scores[3] += 1;
                 break;
        }
    }
    
    myPort.write(scores[1]);
    
    return scores;
  }
  
  void update() {
    for (int i = 0; i < notes.length; i++) {
     if (!notes[i].expired) {
       currNote = i;
       break;
     }
    }
  }
  
  void display() {
    pushMatrix();
      translate(0, -125 - totalDiff);
      float h = map(song.position(), 0, 2000, height, 0);
  
     if (prevH == 0 && currH == 0) {
       prevH = h;
       currH = h;
      } else {
        if (currH != prevH) {
          diff = currH - prevH; // minim stupid
        }
        prevH = currH;
        currH = h;
        totalDiff += diff / 3;
      }
      
      for (int i = 0; i < notes.length; i++) {
       Note n = notes[i];
       n.update();
       n.display();
      }
    popMatrix();
  }
}

//==========================

//FINAL CODE

//TOdo add game over state, 
// add current streak

#include <Arduino.h>
#include <TM1637Display.h>

// Module connection pins (Digital Pins)
#define CLK 2
#define DIO 3

// The amount of time (in milliseconds) between tests
#define TEST_DELAY   10

//4 dig- 7 Segment Display
TM1637Display display(CLK, DIO);
void setup() {
  Serial.begin(9600);
}
int val = 0;

void loop() {
  int k;
    uint8_t data[] = { 0xff, 0xff, 0xff, 0xff };
  display.setBrightness(0x0f);

  // All segments on
//  display.setSegments(data);
//  delay(TEST_DELAY);

  // Selectively set different digits
//  data[0] = 0b01001001;
//  data[1] = display.encodeDigit(1);
//  data[2] = display.encodeDigit(2);
//  data[3] = display.encodeDigit(3);

//  for(k = 3; k >= 0; k--) {
//  display.setSegments(data, 1, k);
//  delay(TEST_DELAY);
//  }
//
//  display.setSegments(data+2, 2, 2);
//  delay(TEST_DELAY);
//
//  display.setSegments(data+2, 2, 1);
//  delay(TEST_DELAY);
//
//  display.setSegments(data+1, 3, 1);
//  delay(TEST_DELAY);
  
  if (Serial.available()>0){
    val = Serial.read();
  }
  display.showNumberDec(val*10, false);
  delay(TEST_DELAY);
  
  
//  bool lz = false;
//  for (uint8_t z = 0; z < 2; z++) {
//  for(k = 0; k < 10000; k += 10) {
//    display.showNumberDec(k, lz);
//    delay(TEST_DELAY);
//  }
//  lz = true;
//  }
}

Leave a Reply