Kinetic Interfaces – Midterm (Moon) Andrew Huang

Author: Andrew Huang

Project name: Underfire! A bullet hell game

Professor: Moon

Date: Nov 3 2018

Goal: To Create a Upper body immersive bullet hell / dodging game.

Description:

After playing some Undertale and bullet hell games over the last couple of weeks, I was inspired to create a more kinetic and interactive version of the game. I used OpenCV to track the players face and used their X and Y positions in order to control the main character.  The player has to survive by dodging all of the homing fireballs that would come near the user. A camera feed at the bottom of the screen would let the user know their webcam is working and where it is relative to the game. When the user runs out of hp (they also periodically get an hp bump), the game displays GAME OVER, and the users score is displayed ( the number of total balls that the user has been hit by). The main character is a frisk sprite from Undertale and and the fireball is a fireball sprite from google.

 

Problems:

In the beginning, the fireballs would track very poorly because I was adjusting the xy position using linear velocity adjustments rather than using ease. By adding a dampening function, I was easily able to make sure that the velocity of the fireballs slowed down as they approached the player.

Even though some people complained that the bottom of my game had a camera picture and ruined the immersion of the game, I reasoned that it was better for the user to know where their face was relative to the game screen than the for the user to not know anything about the current state of the game.

Another problem I have is that the refresh rate of OpenCV is very poor and that the tracking isn’t good no matter what kind of program I run on my machine. This could be a limit on my machine, or is an inherent problem with OpenCV, further research would need to be conducted in order to address this.

Feedback & Future Development

Perhaps I can add better tracking / cameras. Some types of portable cameras can be integrated with OpenCV and a higher resolution experience using full body and limb tracking would result in a better playing experience. The user could probably use their hands to activate some sort of power-up which would either disperse all of the balls or they can use all of the balls as hp (similar to pacman). I was also thinking about adding a bezier track for a type of tracking that would be more realistic and fun to dodge than just using easing. I could also add a menu and better assets to polish the gameplay experience. Overall, I could see this project have promise, and would consider working on it more for the final.

// ANDREW
// KINETIC INTERFACES MIDTERM
// make tracking easier and add endgame
import gab.opencv.*;
import processing.video.*;
import java.awt.*;

Capture video;
OpenCV opencv;

ArrayList<Ball> balls = new ArrayList<Ball>();
PImage img, flameimg;
Health hp;
void setup() {
  size(500, 500
  );
  background(255);
  noStroke();
  smooth();
  noStroke();
  hp = new Health();
  fill(0);
  rect(mouseX, mouseY, 50, 7);
  img = loadImage("frisk.png");
  flameimg = loadImage("flame.png");

  video = new Capture(this, 640/4, 480/4);
  opencv = new OpenCV(this, 640/4, 480/4);
  opencv.loadCascade(OpenCV.CASCADE_FRONTALFACE);  

  video.start();
}
float inputx = 0, inputy = 0, facex = 0, facey = 0;
void draw() {
  opencv.loadImage(video);
  Rectangle[] faces = opencv.detect();
  background(#354c7c);
  text(frameRate, 20, 20);
  text("Balls added: " + balls.size(), 20, 40); 
  rectMode(CENTER);
  if (faces.length > 0) {
    inputx = map(faces[0].x, 0, 640/4, 500, -120);
    inputy = map(faces[0].y, 0, 480/4, 50, 400);
    facex = faces[0].width;
    facey = faces[0].height;
  }
  pushMatrix();
  image(img, inputx - img.width/20, inputy - img.height/20, img.width/10, img.height/10);
  stroke(100);
  noFill();
 // println(inputx, inputy);
  rect( inputx, inputy, img.width/10, img.height/10);
  popMatrix();
  hp.display();
  for (int i=0; i<balls.size(); i++) {
    balls.get(i).track(inputx, inputy);
    balls.get(i).bounce();
    balls.get(i).display();
    if (balls.get(i).removeBall(i)) {
      hp.hit();
    }
    //balls.get(i).age++;
  }
  if (frameCount % 100 == 0) {
    float r = random(0, 1);
    if (r > 0.5) {
      balls.add(new Ball(random(0, width), 10));
    } else {
      balls.add(new Ball(random(0, width), height));
    }
  }
  if (frameCount % 500 == 0 && hp.health < 100) {
    hp.health = ( hp.health + 20 ) > 100 ? 100 : hp.health + 20;
    for (int i = 0; i < balls.size(); i++) {
      balls.get(i).removeBall(i);
    }
  }
  if (hp.getHealth() <= 0) {
    textSize(40);
    fill(255, 0, 0);
    text("GAME OVER", width/3, height/2);
    text("SCORE: "  + hp.getHits(), width/3, height/2 + 50);
    noLoop();
    //delay(9999999);
  }
  pushMatrix();
  image(video, width/2 - 640/8, height-125);
  popMatrix();
}
void captureEvent(Capture c) {
  c.read();
}

void mouseClicked() {
  float r = random(0, 1);
  if (r > 0.5) {
    balls.add(new Ball(random(0, width), 10));
  } else {
    balls.add(new Ball(random(0, width), height));
  }
}

class Ball {
  float x, y, size;
  color clr;
  float xspeed, yspeed;
  float age = 0;

  Ball(float tempX, float tempY) {
    x = tempX;
    y = tempY;
    size = 20;
    clr = color(random(255), random(255), random(255));

    xspeed = random(-5, 5);
    yspeed = random(3, 5);
  }

  void display() {
    //fill(clr);
    tint(clr);
    image(flameimg, x,y,size*2,size*2);
    noTint();
    //ellipse(x, y, size, size);
  }

  void move() {
    x += xspeed;
    y += yspeed;
  }
  
  void track(float xx, float yy){
    float easing = 0.01;
     x += (xx - x) * easing;
     y += (yy - y) * easing;
  }

  void bounce() {

    if ((x>mouseX-25) && (x<mouseX+25) && (y > mouseY-3.5)) {
      yspeed = -yspeed;
    } 


    if (y < 0) {
      yspeed = -yspeed;
    } 
    if (x < 0) {
      xspeed = -xspeed;
    } else if (x > width) {
      xspeed = -xspeed;
    }
  }

  boolean removeBall(int i) { // hitdetection
    if (dist(inputx, inputy, x, y) < 50 || age > 400) {
      balls.remove(i);
      return true;
    }
    return false;
  }
}
class Health {
  float health = 100;
  float MAX_HEALTH = 100;
  float rectWidth = 200;
  float hits;
  
  Health(){
    return;
  }
  void display(){
    if (health < 25){
    fill(255, 0, 0);
    }  else if (health < 50) {
    fill(255, 200, 0);
    } else {
    fill(0, 255, 0);
    }
  
    noStroke();
    // Get fraction 0->1 and multiply it by width of bar
    float drawWidth = (health / MAX_HEALTH) * rectWidth;
    rectMode(CORNER);
    rect(200, 10, drawWidth, 10); 
    
    // Outline
    stroke(255);
    noFill();
    rect(200, 10, rectWidth, 10);
    stroke(0);
  }
  float getHealth(){
    return health;
  }
  float getHits(){
    return hits;
  }
  void hit(){
    health-= 5;
    hits++;
  }
  
}

Kinetic Interfaces – Programming fundamentals – Week 2

Description:

A modification of the original ball moving animation that spawned with mouse holding, but a twist. The balls now wrap around and they get progressively faster as they wrap around the edges. I think this is a good twist on the original creation where the balls just bounced around. The template already had an ArrayList of balls for storing them so I simply used the existing architecture. As I figured out how to make the balls warp, I went to see if there was a way I could smoothly make it warp across the border, but it would see that it would require a lot more modification to the code. Including how balls are rendered when they encounter a border. Overall, I enjoyed making this animation.

// Ball.pde

// you can organize your code better by using the tabs!
// modified by Andrew Huang

class Ball {
  float x, y, size;
  color clr;
  float xspeed, yspeed;

  Ball(float tempX, float tempY) {
    x = tempX;
    y = tempY;
    size = random(10, 100);
    clr = color(random(255), random(255), random(255));

    xspeed = random(-2, 5);
    yspeed = random(-2, 5);
  }

  void display() {
    fill(clr);
    ellipse(x, y, size, size);
  }
  void move() {
    x += xspeed;
    y += yspeed;
  }
  void bounce() { // change to wrap instead
    if (x < 0) {
      //xspeed = -xspeed;
      x = width;
      xspeed -= 5;
    } else if (x > width) {
      //xspeed = -xspeed;
      x = 0;
      xspeed -= 5;
    }
    if (y < 0) {
     // yspeed = -yspeed;
      y = height;
       yspeed -= 5;
    } else if (y > height) {
      //yspeed = -yspeed;
      y = 0;
       yspeed -= 5;
    }
  }
}

// main. pde

ArrayList<Ball> balls = new ArrayList<Ball>();


void setup() {
  size(500, 600);
  noStroke();
}


void draw() {
  background(100);

  for (int i=0; i<balls.size(); i++) {
    // I suggest this way!
    // much more convenient, right?
    Ball b = balls.get(i); 
    b.move();
    b.bounce();
    b.display();
  }

  // let's add balls at the mouseX/Y position when mousePressed !
  if (mousePressed) {
    balls.add( new Ball(mouseX, mouseY) );
  }
}

Kinetic Interfaces – Tilt Brush (Andrew Huang)

One particular product that has stuck out to me over the years is VR. One clear example of this is the Tilt Brush, demo’d in this Google Ad.

The technologies behind this are mainly the occulus rift/ htc vive. Even though this particular application is nothing groundbreaking now that it has been out for several years, the potential for VR/AR is still massive I think. The concept of the 3d brush is very useful because it allows artists to leverage one additional dimension, whereas previously, artists were constrained to 2 dimensional canvases. I believe the future of human / device interaction will become more and more closely integrated, with the device being less of using something and more of an extention of one’s body. One possible feature that VR hasnt done is possible user feedback, such as vibration / haptic feedback or other responses to the user to let them know changes to their environment. I think this really brings about the concept of ZeroUI, and as we already see Google home/ and Amazon Echo become more seamlessly connected to benefit our lives, we will definitely see these technologies pop up more in the future. I hope to make a project in kinetic interfaces that will give users a fine and seamless control over what they’re doing.

IMA Final Project: Taiko v2

Taiko v2


Author: Andrew

Professor: Dan

Date: May 17th

Abstract

A newly virtual drumming experience

A continuation of my midterm project, Taiko, a rhythm game where instead of using physical drums (cardboard boxes) I use the leap motion sensor to detect the user’s hand motions to evoke a response. In addition, music and beat timings were added to the game. I got most of my inspirations from the rhythm games I played before, such as DDR.


Process

I did not really have a brainstorming session, as most of the outlining was done during the midterm project, but I did not have the programming expertise yet to build the product just yet. After I attended the computer vision workshop, I figured out that the Leap Motion Sensor might prove to be immensely helpful in designing a drumming experience where the users isn’t limited by the just drumming on a surface.


Logistics:

IMG_20170520_003435 Screen+Shot+2017-05-20+at+12.34.43+AM

Most of the work was involved in the Arduino code. It proved tricky to obtain the timings for the using the minim BeatDetect Library because the beats were generated live and to play the music. In order to solve this problem, I saved the timings in csv format, thus requiring the song being played at least once in order to obtain the timings. After that each note is moved downward using translate and it’s state changed depending on one hits the notes. I added the code for the leap motion sensor to both detect the finger tap gesture or the z position in order to detect whether the user was in the hit-zone using relative to the laptop position. The values of BeatDetect itself were also sort of finicky since it largely depends on the peaks of the song.

I then sent the values of the Excellent score x 10 to Arduino, where it’s value was displayed on a 4 digit display.


Trials and tribulations

The most difficult of the project was getting a decent playing experience with the Leap Motion sensor. The midair drumming felt finicky at best, and most of the time, didn’t evoke a response within processing. It proved to not be as good of a experience as a physical drum! In addition, the drumming experience didn’t really feel like drumming at all.

Arduino also didn’t really play an significant role in this project at all, as the scoring system felt superfluous, I suppose leap motion provided most of the interaction the user needed to use the product effectively.


Conclusions

Overall, this final project was pretty rewarding. I developed a good understanding of Processing and Arduino over the course of this semester, and it was certainly a good primer into IMA. I think Interaction Lab provided me a good set of skills for me to possibly continue in IMA. I enjoyed my time this semester coding creative and interactive projects which wouldn’t normally be in computer science.

My code for the project is attached below.

import ddf.minim.*;
import ddf.minim.analysis.*;
import de.voidplus.leapmotion.*;
import processing.serial.*;

Serial myPort;

int value = 0;
int num = 0;
Minim minim;
AudioPlayer song;
BeatDetect beat;
ArrayList systems;

Table table;

float eRadius;

Score score;
Note[] notes;

LeapMotion leap;
boolean KickZone;

void setup()
{
  size(1200, 700);
  //fullScreen();
  minim = new Minim(this);
  song = minim.loadFile("anime.mp3", 2048);
  song.play();
  
  systems = new ArrayList();
  beat = new BeatDetect();

  //beat.detectMode(BeatDetect.FREQ_ENERGY);
    beat.setSensitivity(300);  

  //table = new Table();
  //table.addColumn("id");
  //table.addColumn("time");
  
  table = loadTable("new2.csv", "header");
  for (TableRow row : table.rows()) {
    
    int id = row.getInt("time");
   
    println(id);
  }
  score = new Score(table);
  
  ellipseMode(RADIUS);
  eRadius = 20;
  
  // Arduino
  printArray(Serial.list());
  
  myPort = new Serial(this, Serial.list()[2], 9600);
  
   leap = new LeapMotion(this).allowGestures();  // All gestures
}


void draw()
{
  background(0, 0, 0);
  
   //Particle System Creation
  
  for(int i = systems.size() -1 ; i >= 0 ; i--) {
   ParticleSystem ps = (ParticleSystem)systems.get(i);
   if(ps.done()) {
      systems.remove(i); 
   } else {
     ps.draw();
   }
  }
 
 // Drawing Buttons and Line 
 
  noStroke();
  fill(255, 255, 255);
  rect( width / 2 - 165, height - 140, 30, 30); // 125
  ellipse( width / 2 + 150, height - 125, 15, 15); // 125
  stroke(0); 
  noFill();  
  rect( width / 2 - 175, height - 150, 50, 50); // 125
  ellipse( width / 2 + 150, height - 125, 25, 25); // 125
  
  stroke(255);
  strokeWeight(5);
  fill(255, 255, 255);
  line(0, height - 125, width, height - 125);
  noStroke();

  // Update and display notes.
  
  score.update();
  score.display();
  
  int[] scores = score.getScores();
  textSize(26);
  fill(255, 0, 0, 100);
  text("PERFECT: " + scores[1], 10, 30);
  text("OKAY: " + scores[2], 10, 55);
  text("POOR: " + scores[3], 10, 80);
  text("MISSED: " + scores[0], 10, 105);
  
  
  stroke(255);
  
  // For beat detection
  beat.detect(song.mix);
  float a = map(eRadius, 20, 80, 60, 255);
  fill(60, 255, 0, a);
  if ( beat.isOnset() ) {
    background(60, 255, 0, a);
    eRadius = 80;
    num += 1;
    //println(num);
    //TableRow newRow = table.addRow();
    //newRow.setInt("time", song.position());
    //newRow.setInt("id", table.getRowCount() - 1);
  }
  
  ellipse(width/2, height/2, eRadius, eRadius);
  eRadius *= 0.95;
  if ( eRadius < 20 ) eRadius = 20;
  
  //===================================================
    
  // Leap logic
    
  for (Hand hand : leap.getHands()) {
    //hand.draw();
    PVector hand_position = hand.getPosition();
    
    println(hand_position.x);
    println(hand_position.y);
    println(hand_position.z);

    if (hand_position.z<50) {  
      KickZone = false;
    } else {
      KickZone = true;
    }
  }
  if(KickZone){
    println("hit");
    hit();
  } else {
    println("not hit");
  }  
}

void keyPressed() {
  if (key == CODED) { 
    if (keyCode == UP) {
      saveTable(table, "data/new.csv");
    }
  } 
  
  if (key == ' ') {
   hit();
  }
}

void hit() {
  
   println(score.notes[score.currNote].pos.y - score.totalDiff - height);
   Note currNote = score.notes[score.currNote];
   // If note hasn't been played, check for hit.
   if (!currNote.played) {
     float distFromLine = currNote.pos.y - score.totalDiff - height;
     if ( distFromLine > -100 && distFromLine < 50) {
       println("RIGHT ON");
       currNote.played = true;
       currNote.status = 1;
        int size = round(random(500,1000));
        systems.add( new ParticleSystem(size,random(.004*size,.006*size),random(0, width) ,random(0, 150)) ); 
     }
     if ( distFromLine < -100 && distFromLine > -200) {
       println("OKAY");
       currNote.played = true;
       currNote.status = 2;
     }
     if ( distFromLine < -200) {
       println("POOR");
       currNote.played = true;
       currNote.status = 3;
     }
   }
}


class Note {
   PVector pos;
   int id;
   float time;
   boolean played;
   boolean expired;
   color c;
   int status; // 0 - unhit, 1 - hit perfect, 2 - hit okay, 3 - hit poor, -1, not hit
   int permute;
   
   Note(int _id, float _time) {
     id = _id;
     time = _time;
     played = false;
     expired = false;
     c = color(60, 255, 0, 100);
     pos = new PVector(0, 0);
     permute = (int) random(50);
   }
   
   void update() {
     
   }
   
   void display() {
        //int timeOffset = 0;
        float mapped = map(time, 0, 2000, height, 0);        
        if (status == 0) {
          fill(c);
        } 
        switch(status) {
         case -1: fill(255, 0, 0);
                  break;
         case 0: fill(c);
                  break;
          case 1: fill(255, 255, 255);
                  break;
         case 2: fill(255, 255, 0);
                 break;
         case 3: fill(255, 0, 0);
                 break;
        }
        
        if (score.totalDiff < pos.y - height - 50) {
          expired = true;
          if (!played) {
            status = -1;
          }
        }
        if (id == 0) {
          println("Y POS:", pos.y);
          //println(song.position());
          println("TOTAL DIFF:", score.totalDiff);
          //println(time);          
        }
        
        //noStroke();
         println(permute);
        
       // if (id % 2 == 0) {
         if (permute % 2 == 0){
           //ellipse(width / 2 - 150, mapped, 15, 15);
           rect(width / 2 - 150 - 15, mapped - 15, 30, 30);
           pos.x = width / 2 - 150;
           pos.y = mapped;
        }
        else {
           ellipse(width / 2 + 150, mapped, 15, 15);
           pos.x = width / 2 - 150;
           pos.y = mapped;
        }
   }
}


// Class to create a Particle System
class Particle {
  int t;
  PVector p;
  PVector v;
  float r = 3.0;
  float g = 0.05;
  color c;
  
  Particle(float max_v, color inC) {
    p = new PVector(0,0);
    float a = random(0,TWO_PI);
    float vel = random(0,max_v);
    v = new PVector(vel*cos(a),vel*sin(a));
   
    int r = (inC >> 16) & 0xFF;  // Faster way of getting red(argb)
    int g = (inC >> 8) & 0xFF;   // Faster way of getting green(argb)
    int b = inC & 0xFF;          // Faster way of getting blue(argb)
    c = color( constrain( r+round(random(-30,30)) , 0 , 255),
               constrain( g+round(random(-30,30)) , 0 , 255),
               constrain( b+round(random(-30,30)) , 0 , 255));
  }
  
  void draw() {
    update();
    noStroke();
    fill(c);
    ellipse(p.x,p.y,r,r);
  }
  
  void update() {
    t++;
    v.add(new PVector(0,g)); // Accelerate
    p.add(v);                // Velocitate
  }
  
  boolean done() {
    if(random(0,t) > 50) {
       return true; 
    }
    
    return false;
  }

  
}

class ParticleSystem {
  ArrayList particles;
  PVector p; //position
  color c;
  
  
  ParticleSystem(int size, float spread, float x, float y) {
    p = new PVector(x,y);
    particles = new ArrayList();
    c = color(random(220)+35,random(220)+35,random(220)+35);
    
    for(int i = 0; i < size; i++) {
       particles.add(new Particle(spread,c)); 
    }
    
  }
  
  void draw() {
    pushMatrix();
    translate(p.x,p.y);
    
    for(int i=particles.size()-1; i >= 0 ; i--) {
      Particle p = (Particle)particles.get(i);
      if( p.done() ) {
         particles.remove(i);
      } else {
         p.draw();
      }
    }
    
    popMatrix();
  }
  
  boolean done() {
    if(particles.size() == 0 ) {
       return true; 
    }
    return false;
  }
  
}

class Score {
  Table t;
  float prevH = 0;
  float currH = 0;
  float totalDiff = 0;
  float diff = 0;
  Note[] notes;
  int currNote = 0;
  
  Score(Table _t) {
    t = _t;
    notes = new Note[t.getRowCount()];
    int rowNum = 0;
     for (TableRow row : table.rows()) {
      int time = row.getInt("time");
      notes[rowNum] = new Note(rowNum, time); 
      rowNum += 1;
    }
  }
  
  int[] getScores() {
    // get scoring
    int[] scores = {0, 0, 0, 0};

    for (int i = 0; i < notes.length; i++) {
      switch(notes[i].status) {
         case -1: scores[0] += 1;
                  break;
         case 1: scores[1] += 1;
                  break;
         case 2: scores[2] += 1;
                 break;
         case 3: scores[3] += 1;
                 break;
        }
    }
    
    myPort.write(scores[1]);
    
    return scores;
  }
  
  void update() {
    for (int i = 0; i < notes.length; i++) {
     if (!notes[i].expired) {
       currNote = i;
       break;
     }
    }
  }
  
  void display() {
    pushMatrix();
      translate(0, -125 - totalDiff);
      float h = map(song.position(), 0, 2000, height, 0);
  
     if (prevH == 0 && currH == 0) {
       prevH = h;
       currH = h;
      } else {
        if (currH != prevH) {
          diff = currH - prevH; // minim stupid
        }
        prevH = currH;
        currH = h;
        totalDiff += diff / 3;
      }
      
      for (int i = 0; i < notes.length; i++) {
       Note n = notes[i];
       n.update();
       n.display();
      }
    popMatrix();
  }
}

//==========================

//FINAL CODE

//TOdo add game over state, 
// add current streak

#include <Arduino.h>
#include <TM1637Display.h>

// Module connection pins (Digital Pins)
#define CLK 2
#define DIO 3

// The amount of time (in milliseconds) between tests
#define TEST_DELAY   10

//4 dig- 7 Segment Display
TM1637Display display(CLK, DIO);
void setup() {
  Serial.begin(9600);
}
int val = 0;

void loop() {
  int k;
    uint8_t data[] = { 0xff, 0xff, 0xff, 0xff };
  display.setBrightness(0x0f);

  // All segments on
//  display.setSegments(data);
//  delay(TEST_DELAY);

  // Selectively set different digits
//  data[0] = 0b01001001;
//  data[1] = display.encodeDigit(1);
//  data[2] = display.encodeDigit(2);
//  data[3] = display.encodeDigit(3);

//  for(k = 3; k >= 0; k--) {
//  display.setSegments(data, 1, k);
//  delay(TEST_DELAY);
//  }
//
//  display.setSegments(data+2, 2, 2);
//  delay(TEST_DELAY);
//
//  display.setSegments(data+2, 2, 1);
//  delay(TEST_DELAY);
//
//  display.setSegments(data+1, 3, 1);
//  delay(TEST_DELAY);
  
  if (Serial.available()>0){
    val = Serial.read();
  }
  display.showNumberDec(val*10, false);
  delay(TEST_DELAY);
  
  
//  bool lz = false;
//  for (uint8_t z = 0; z < 2; z++) {
//  for(k = 0; k < 10000; k += 10) {
//    display.showNumberDec(k, lz);
//    delay(TEST_DELAY);
//  }
//  lz = true;
//  }
}

Lab 12: Media Controller

Author: Andrew Huang

Date: May 5 2017

Partners: Jacob Park

Materials: Arduino, Potentiometer, Processing

Today, after learning about images and videos, we used Arduino and Processing to make an interaction between the two. Our Arduino Potentiometer code allowed us to control the position of the picture, and the Processing code allowed us to draw the picture of a rocket . Afterward, we decided the sound of a rocket. Some difficulties we had was accessing the right values of the values of Arduino, causing the baud rate to become unstable, but we quickly realized that our baud rate was not correct because we accessed the values outside of the while loop, and this caused the reading to be unstable. Finally, we got both the Arduino and the Processing side of the code to work properly.

The code for the lab is attached below.

Links / sources: 

https://www.youtube.com/watch?v=KfYto9Szei8https://www.extremetech.com/wp-content/uploads/2017/02/pslv-640×353.jpg


 


/**
 * Load and Display 
 * 
 * Images can be loaded and displayed to the screen at their actual size
 * or any other size. 
 */
import processing.serial.*;
import processing.sound.*;

SoundFile file;

PImage img;  // Declare variable "a" of type PImage


Serial myPort;
int valueFromArduino;

void setup() {
  size(640, 360);
  // The image file must be in the data folder of the current sketch 
  // to load successfully
   printArray(Serial.list());
  // this prints out the list of all available serial ports on your computer.
  
  myPort = new Serial(this, Serial.list()[ 2 ], 9600);
  img = loadImage("pslv-640x353.jpg");  // Load the image into the program 
  file = new SoundFile(this, "Rocket Launch Sound Effect.mp3");
  file.play();
  file.loop();
}

void draw() {
  // Displays the image at its actual size at point (0,0)
   while ( myPort.available() > 0) {
    valueFromArduino = myPort.read();
    println(valueFromArduino);
  }
  background(0);
  //image(img, 0, 0);
  // Displays the image at point (0, height/2) at half of its size
  image(img, 0, valueFromArduino, img.width, img.height);
}

==================================

// Arduino

// the setup routine runs once when you press reset:
void setup() {
  // initialize serial communication at 9600 bits per second:
  Serial.begin(9600);
}

// the loop routine runs over and over again forever:
void loop() {
  // read the input on analog pin 0:
  int sensorValue = analogRead(A0);
  // print out the value you read:
  Serial.write(sensorValue);
  delay(10);        // delay in between reads for stability
}

Lab 11 – Drawing Machines

Date: April 28th

Author: Andrew Huang

Partner(s): Jacob Park

Materials: Arduino, Stepper motor, wires, plastic arms, Transistor code

Today, we made a drawing arm using two stepper motors and the SH75440NE chip. The hardest part of the lab was hooking up the assembly, as plugging in the pins for every single pin was tedious and I messed up plugging in the ground for part of the lab. Finally, we hooked up the arm assembly, and it worked the first time we put it together. When we put it together, we uploaded the code for continuous and speed control which allowed us to control the speed of the stepper motor through the potentiometer. This allowed us to use the arm and control the functional spinning speed. When we put it all together with another group, both of the arms worked!

What we learned: we learned how to control the stepper motor and it’s speed in arduino, and how to use the transistor to control logic by hooking it up to the pins in arduino.

bipolarStepperMotor-300x254IMG_20170428_134634

Lab 10: Tinkercad 3d printing

Author: Andrew Huang

Instructor: Professor Dan

Lab Date: April 21st

Today we learned how to make a 3d model in Tinkercad in order to learn how to make something on the 3d printer. The assignment was to make a mount which would house two parts in order to combine them. I made a mount for the stepper motor in order to hold the security camera / webcam but in reality, it could be adapted to hold anything rectangular of similar dimensions. After I had tediously figured out how to extrude properly in in Tinkercad with help from Marcela, I then used the caliper to measure out the rod dimensions of the Stepper motor and an approximation of the size of the webcam. Afterwards I extruded a hole accomodate the axel, the square shaped hold for the webcam, and two triangle shaped things for style and support. Overall, I found this lab a good challenge for figuring out how to create a concept in my mind in a 3d space. And finding out the orthogonal view made it very helpful to snap my pieces together, making sure everything is properly centered and aligned.

My final product is below:

Fantabulous Borwo-Sango (1) Fantabulous Borwo-Sango Fantabulous Borwo-Sango (2)

Field Trip to Autodesk and Stamp Documentation

Instructor: Professor Dan

Author: Andrew

Today we took a field trip to Autodesk where we learned about the development process and the importance of 3d cad in the modern software developing world. The 3d cad models used for printing and animation seem very important, especially at the rate that products are being produced and fabricated every year. As I begin to plan for my future projects, I see the usefulness of being able to make a mockup quickly on the computer and a 3d model being able to test my creations quickly. I have previously used Autodesk Inventor before to make 3d cad mockups, so being able to see the logic behind the business itself was very interesting.

17968770_10210653926217004_1185247383_o


Illustrator Stamp

Using Illustrator was similar to using Photoshop, however, there were several features that I was unfamiliar with and the object and vector differences proved to be kind of confusing. What I found most impressive was the fact one can import a picture and illustrator can generate a vector image based on the qualities of the picture. However, as I was doing this I could not remove the white parts of the image; I later found out that the white parts of the image could be removed through the advanced features located in the sidebar under Image Trace >> Advanced. However I did find it pretty useful to generate clean vector graphics based off previously blurry pngs.

Screen Shot 2017-04-18 at 3.03.20 AM Screen Shot 2017-04-18 at 2.46.32 AM

IMA Midterm Documentation: Taiko

Taiko: An immersive drumming experience.

Author: Andrew Huang

Instructor: Professor Dan Mikesell

Description: A drum pad  experience made with Arduino and processing for visualization. I created this because of inspiration of midi instruments, and games I have played such as DDR or other rhythm games in arcades. I wanted to build off that and simply make a processing / Arduino version. In some sense, this isn’t an original idea, but the idea of creating an immersive drumming experience resonated with me. Remembering the vibration sensor from previous projects, I remembered what kind of meaningful interfaces I could make with Arduino.

Demo:

taiko1

Conceptual Development: 

I originally wanted to build a type of midi like instrument such as a wind instrument, but that proved to be too ambitious as I found out from the previous assignment that the noise sensors prove not to be very accurate regarding it’s surroundings. So I settled for drums. Making this proved to be very intuitive, but I did not have the time before to plan and gather the materials, so I settled for cardboard boxes. I took most of my inspiration from games like Guitar Hero or Rock band, where the goal of the game is to maximize your score while keeping a sense of rhythm at hand. Now that I had a concrete idea of what to make for the game, I decided to get started with making my project on the Arduino side first.

Technical Development:

I had a good idea in mind already, so I did not make a mock-up and went straight to trying to create a presentable project. Most of the heavy lifting of the project is Processing code, since the actual game has several complex components such as scoring, and the types of gameplay augmentation that I was looking for. I tried to get a 3d implementation of the program working in the end with P3D but the assignment due date was nearing. So I settled with a 2d version of the program.

The hardware portion of the project proved simple, as I simply just did simple analogRead and sent the values I got from Arduino into Processing, where I detected during the hit, and applied my scoring function.

Lessons learned

I’ve learned that sometimes, presentation matters more than the hidden features that does not really affect the final product. The cardboard boxes I used for my midterm projects did not really demonstrate the feasibility of the project, and it was a stretch to even call them as “drums”. Next project, I will seek to gather the appropriate materials in order to deliver the true immersive experience. Overall, the user experience using the vibration sensors is kind of buggy, as the box continues to vibrate after a hit, leading to multiple hits of the scoring function.

Perhaps next time I will make a more intuitive user interface, as the score itself was hard to see against the backdrop. Additionally, I hope to use more fonts and art to make a look that makes a difference between my project and that stock Processing shape look. Overall, I would consider my project to be somewhat of a success, the vision was clear, however, it lacked creativity and originality.

Attached is the source code I used in the project

//Processing main class:

import de.looksgood.ani.*;
import de.looksgood.ani.easing.*;

/*
*DDR MAIN CLASS FOR THE MIDTERM PROJECT
* AUTHOR: ANDREW HUANG
* DATE: MAR 24 2017
*
*/ 
final int COLS = 2;
final int wid = 1200;
final int ht = 700;
final int tolerance = 10; // no one is perfect
final int AR = 7; //ms of approach rate

import processing.serial.*;
Serial myPort;
int val, value;

void setup(){
   size(1200,700, P2D); // manually set COLS and wid,  weird bug
   printArray(Serial.list());
   myPort = new Serial(this, Serial.list()[2], 9600);
   Ani.init(this);
}

// GRID( color, alpha=-1, columns, width, height, approach rate);

Grid g = new Grid(100, -1, COLS, wid, 0, AR); // starts at top.

Grid hz = new Grid(255, 0, COLS, wid, ht-35, 0); // hitzone on bottom.

int score = 0;
int valueFromArduino;

void draw(){
  //background(0);
  noStroke();
  gradientRect(0, 0, wid, ht, #AEC6CF, #B39EB5);
  //fill(100);
  textSize(32);
  text("Score: " + score, 10, 30); 
  g.drawGrid();
  g.moverandom();
  stroke(255);
  strokeWeight(5);
  drumhit();
  hz.drawGrid();
}
void keyPressed(){
   score += g.scoring(ht - 35, tolerance);
   println(score);
}

void drumhit(){
  while ( myPort.available() > 0) {
    valueFromArduino = myPort.read();
    
    valueFromArduino = valueFromArduino / 10 / 2;
    println(valueFromArduino);
    
  }
  if (valueFromArduino > 20){
    score += g.scoring(ht - 35, tolerance);
     println(score);
    delay(5);
  }
}

void gradientRect(int x, int y, int w, int h, color c1, color c2) {
  beginShape();
  fill(c1);
  vertex(x,y);
  vertex(x+w,y);
 
  fill(c2);
  vertex(x+w,y+h);
  vertex(x,y+h);
  endShape();
}

/////////////////////////////////////////////////////////////////////////////

/*
*CELL/GRID CLASS FOR THE MIDTERM PROJECT
 * AUTHOR: ANDREW HUANG
 * DATE: MAR 24 2017
 *
 */
class Cell {
  int speed, rad, posx, posy;
  
  Cell(int rad, int posx, int posy, int speed) {
    this.rad = rad;
    this.posx = posx;
    this.posy = posy;
    this.speed = speed;
    
  }
  void draw() {
    ellipse(posx, posy, rad, rad);
  }
  void animate(int r, int g, int b){
    //radiusAni.start();
    //fill(r,g,b);
    //Ani.to( this, 0.1,  "rad", rad+30);
    
    //Ani.to( this, 0.1,  "rad", rad-30);
    
    ////if (rad >= 50){
    
    ////}
    //ellipse(posx, posy, rad, rad);
    
    
    for (int i = rad; i < rad +30; i++){
      fill(r,g,b);
      ellipse(posx, posy, i, i);
    }
    for (int i = rad+30; i < rad; i--){
      fill(r,g,b);
      ellipse(posx, posy, i, i);
    }  
  }
  void move(){
    posy += speed;
  }
  int getposx(){
    return posx;
  }
  int getposy(){
    return posy;
  }
  int getspeed(){
    return speed;
  }
  void setspeed(int spd){
    speed = spd;
  }
  void setpos(int x, int y){
    posx = x;
    posy = y;
  }
}

class Grid{
  // both the pieces api and the hitzone api will be coded from this class
  
  Cell[] cells;
  int cols;
  int permute = -1;
  int fillc=0;
  int alph=-1;
  
  Grid(int fillc, int alph, int cols, int w, int h, int AR){
    this.alph = alph;
    this.cols = cols;
    this.fillc = fillc;
    cells = new Cell[cols];
    for (int i =0; i < cols; i++){
      cells[i] = new Cell(50, (int )map(i, 0, cols, 0 , w) + w/cols/2, h, AR);
    }
  }
  void drawGrid(){
    for (int i=0; i < cols; i++){
      if (alph >= 0){
         fill(fillc, alph);
      } else {
        fill(fillc, cells[i].getposy()-20);
      }
      cells[i].draw();
    }
  }
  void moveGrid(){
    for (int i=0; i < cols; i++){
      cells[i].move();
    }
  }
  void moverandom(){
    //moves one at a time.
    if (permute == -1){
      permute = (int) random(0, cols);
    }
    if (cells[permute].getposy() >= 0 && cells[permute].getposy() < height){
      cells[permute].move();
    } else {
      //cells[permute].setspeed(-cells[permute].getspeed());
      cells[permute].setpos(cells[permute].getposx(), 0);
      permute = (int) random(0, cols);
    }
  }
  void bounce(){
    for (int i=0; i < cols; i++){
      if (cells[i].getposx() < 0 || cells[i].getposy()< 0 || cells[i].getposy()> height){
        cells[i].setspeed(-cells[i].getspeed());
      }
    }
  }
  int scoring(int ht, int tolerance){
    // scoring for the balls position, relative to the screen.
    Cell ball = cells[permute]; //current moving ball
    if (ball.getposy() <= ht + tolerance && ball.getposy() >=  ht - tolerance){
      ball.animate(51, 252, 255);
      return 300;
    }
    else if (ball.getposy() <= ht + tolerance*2 && ball.getposy() >= ht - tolerance*2){
      ball.animate(51, 255, 60);
      return 200;
    }
    else if (ball.getposy() <= ht + tolerance*3 && ball.getposy() >= ht - tolerance*5){
      ball.animate(255, 94, 51);
      return 100;
    }
    return -100;
  }
}

////////////////////////////////////////////////////////////////////////////////////////////

// Arduino Code

int analogPin = 0;
int analogPin2 = 1;// potentiometer wiper (middle terminal) connected to analog pin 3
                       // outside leads to ground and +5V
int val = 0;           // variable to store the value read
int val2 = 0;

void setup() 
{
  Serial.begin(9600);          //  setup serial
}
void loop()
{
  val = analogRead(analogPin);    // read the input pin
  Serial.println(val); 
//  Serial.write(val); 
  delay(10); // debug value 
}

//////////////////////////////////////////////////////////////////////////////////

Lab 6: Serial Communication

Date: March 17 2017

Instructor: Professor Mikesell

Author: Andrew Huang

Partner: Jacob

Materials used: Breadboard, buzzer, wires, Arduino, 10k resistor

Processing to Arduino

Our first part of the part of the project was to get the buzzer to go off when the mouse is pressed in processing. This code was somewhat trivial, since we already had the code from yesterday where we made the LED light up from clicking the square in Processing and Arduino. Modifying it to buzz was simply a matter of using the tone() function. After that, we decided to proceed onto the Arduino to processing.

Arduino to Processing

The second part of the lab involved us using the serial port to turn the square darker when the button was pressed. This part of the lab was more tricky for us, due to the fact that I omitted the crucial while loop that told me whether or not the port was available. Due to that fact, we got really weird values, such as negative ones; values that we originally did not write to the serial port. From that, we realized that our code somewhat mirrored our first exercises, and after help from an instructor, we figured the code out and got the button to work properly.

Conclusion:

This lab provided good intuition on how to use Serial to use processing to talk to Arduino, and vise versa. It also told me crucially, that the ports have to be available for either of them to read their values, otherwise, the values that come out are not coherent.

//Buzzer Code: 

//Processing:

import processing.serial.*;
Serial myPort;
int val;

void setup(){
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[2], 9600);
}

void draw(){
  if (mousePressed){
    myPort.write('H');
  } else {
    myPort.write('L');
  }
}

//Arduino:

int val;

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  pinMode(13, OUTPUT);
  

}

void loop() {
  // put your main code here, to run repeatedly:
  while (Serial.available()){
    val = Serial.read();
  }
  if (val =='H'){
    //digitalWrite(13, HIGH);
    tone(13, 1200, 20);
  } else if (val == 'L'){
    //digitalWrite(13, LOW);
  }
  delay(10);
}

//_________________________________

//Button Code:

//Processing:

import processing.serial.*;
Serial myPort;
int val, value;

void setup(){
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[2], 9600);
}

void draw(){
  if ( myPort.available() > 0) {
  value = myPort.read();
  println(value);
  }
  rect(0, 1, 50, 50);
  if (value == 1){
    fill(0, 0, 0);
  } else {
    fill(255, 255, 255);
  }
}

//Arduino:

int val;

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  pinMode(13, INPUT);
  

}

void loop() {
  // put your main code here, to run repeatedly:
//  while (Serial.available()){
//    val = Serial.read();
//  }
  int read = digitalRead(13);
  if (read == 1){
    //digitalWrite(13, HIGH);
    //tone(13, 1200, 20);
    Serial.write(1);
  } else {
    //digitalWrite(13, LOW);
    Serial.write(0);
  }
  delay(100);
}