Interaction Lab_Final Project- Catch Me If You Can (Daniel)

Instructor: Professor Daniel

Project Due Date: May 19, 2017

Partner: Nathalia Lin

Title

Catch Me If you Can

Subtitle

A playing ball created with processing 3D world.

Description

Catch Me If You Can is a playing ball in a 3D space. The joy stick component of the project allows its users navigate through the 3D world and change velocity and direction of the ball. Users can grab the ball inside the 3D space using leap motion as well. In short, it is an interactive toy that lets users to physically engage in the digital world.

Demo of Project

  • Picture of the Controller

20170516_134038

  • Video

 

Conceptual Development

  • Ideation

Our very first aim was making a 3D Tetris using leap motion. After presenting our idea to class and started working on the development of the project, we realised that it would be difficult for us to create a high quality 3D Tetris in the amount of time available for us. Also, we wanted to create something different since the 3D Tetris using leap motion already exists. Therefore, we started thinking about what we could create with the 3D space and leap motion. We were inspired by the stress ball that bounces well:

stress-ball

Therefore, we decided to create a bouncing ball in a 3D space and add the grabbing feature to it using leap motion.

There were some possible features we were thinking in the beginning

-Bouncing Ball

-Gravity

-Velocity Change

-Leap motion Grab

-Leap motion Throw

-Adding more balls

  • Research

In the beginning, we did not know much about leap motion and 3D in processing, so our research mostly focused on these two components. Also, there are some processing examples that we researched concerning the bouncing ball part.

3D Processing: https://processing.org/tutorials/p3d/

Bouncing Ball: https://processing.org/examples/bouncingball.html

Technical Development

  • Design

Our design of the project started with thinking about the 3D space. We also had to think about the design of the sphere. We first created a Box like space that has an open side for the users to see:

We made the colors of the sides different to emphasize that the space is 3D. We also included the navigating option (with mouse).

The design of the sphere started with 2D. We started with a simple circle to simple sphere. Finally, the design of the sphere was changed with the sphere detail. Lowering the sphere detail let us have the final ball that we have, a sphere that looks less full:

dome

Another part of our design was the physical component, our Joy Stick Controller. We simply used a box below the joy stick and put the Arduino inside the box. At first, we made a hole on the box to put the joy stick under the box, but that would make the controller too fragile. Thus instead, we decided to put the joy stick on top of the box and only put the chords inside the box by making a hole (controller picture is included in the demo part).

  • Implementation

Stage 1- Working with leap motion

We started approaching the project with trying to understand how to use leap motion. Since the sample leap motion code was very complicated we spent much time trying to understand it. We tested out the edited code at the workshop (with moon):

By using the leap motion’s grab function and creating if statements, we created a button in 2D that we could play with:

Stage 2- Designing a Bouncing Ball

We tried to understand the bouncing ball in 2D first, then eventually we converted it to 3D. We tried both using gravity and not using gravity, and we decided that the ball without gravity would be more dynamic since it can bounce around in every direction:

Also, we included the background color change when the ball hits the sides in the beginning, the function that we excluded in the end.

Stage 3- Converting the Ball from 2D to 3D

Converting anything from 2D to 3D was always very challenging. Since we already created a 3D space (described in the design part), we wanted to change 2D ball to a sphere and make it bounce around in the 3D space. The problem that we had was that the ball would sink to the walls half way. We solved this problem by changing the bounce if statements. e.g. if (sphX > (worldSize/2)-area) {
sphX = worldSize/2-area;
velX = -velX;

}

and the same goes for y and z (else if for opposite directions).

Stage 4- Combining Leap motion with the 3D Bouncing Ball

Now, our challenge was to integrate all the processing parts that we had separately, leap motion and 3D bouncing ball. We struggled a lot in the beginning since the integration would continuously give us errors. We found out that the error was happening because the variables were mixed up. We unified the variables relating to spheres (e.g. sphere velocity, sphere position, and so on), and that gave us the good outcome.

*Another error was the translating error:

Professor Moon helped us debugging the code and the error was due to not setting up the background color.

Stage 5- Adding Arduino component of the project

Many problems occurred on this stage of our project development. When we combined our serial communication component with the rest of the project the project lagged a lot. With Professor Moon’s help, we found out that the delay of our Arduino code was too small. We increased the delay from 10 to 100, and the problem was solved.

Stage 6- Problem Solving

The last stage of our project was polishing up everything we had. The minor problems that we wanted to fix at the end was concerning the stability of the project. Since the sensor value that we received from the project was fluctuating a little bit, we had to create if statements to make the space stable. We created statements like  if ((sensorValues[0]>506)||(sensorValues[0]<499)) {
rotateY(rotAngleY);
}

to turn the space only when the sensor value goes above or beyond the range that we set. Also, since the sensor value 1 would turn to 1023 when we press the button, we had to create another if statement to hold the space from turing when the value was larger than 763 (max value it would get with turing the joy stick).

Lessons Learned

Creating this project and taking this class in general let me see the artistic potential inside of me. I always considered myself a person who is terrible at both art and technology. This class helped me overcome that fear I always had. The shortcoming of the project was that leap motion sensor did not always work very smoothly. Many users could not easily grab the ball since the leap motion could not detect their hand positions very well. The tip for using this project is that you always have to worry about the Z position since this is based on a 3D space. Also, always be patient!

 

 

 

 

 

//Processing Code Tap 1 

import de.voidplus.leapmotion.*;    //import from library
import processing.serial.*;

LeapMotion leap;    //declaring the object

String myString = null;
Serial myPort;

int NUM_OF_VALUES = 3;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;  /** this array stores values from Arduino **/

int worldSize = 400; 

//the center of the palm 
float handX;  
float handY;
float handZ;

//handGrab
float hGrab;

float sphX;   
float sphY;
float sphZ;
float sphRad;
color sphColor;

//velocity of the sphere
float velX;    
float velY;
float velZ;

int previousSensorValues;

void setup () {
  size (600, 600, P3D);  //3D space
  background (255);

  setupSerial();

  leap = new LeapMotion(this);    //initializing the object

  sphX = 0;                       //x, y,z of the center of the circle
  sphY = 0;
  sphZ = 0;
  sphRad = 30;                    //radius of the sphere
  sphColor = color(255, 0, 0);    //color of the sphere
  sphereDetail(30);               //detail of the sphere

  //velocity at random with the range of -3 and 3
  velX=random (-3, 3);
  velY=random (-3, 3);
  velZ=random (-3, 3);
}

// update!
// check the value!
// and then display

void draw () {
  background (255);

  updateSerial();
  printArray(sensorValues);


  leapMotionUpdate();    //call methods of the object

  //center the hand
  float posX = map(handX, 0, width, -200, 200);
  float posY = map(handY, 0, height, -200, 200);
  float posZ = map(handZ, 80, 20, -150, 150);
 
  //println (hGrab);  //shows the value of 1.0 when I grab the circle

  float distance = dist(sphX, sphY, sphZ, posX, posY, posZ); //distance between sphere and hand
  //println(distance);
  if (distance < sphRad) {
    sphColor = color(0, 255, 0);   // if distance is smaller than radius then the circle turns green
    if (hGrab > 0.8) {  //open hand = 0, closed hand = 1
      sphColor = color(255, 255, 0);  //if you grab the button, then its gonna turn yellow
      sphX = posX;  //position of sphere = position of hand
      sphY = posY;
      sphZ = posZ;
    }
  } else {
    sphColor = color(0, 0, 255);  //color it blue
  }

  //sphere increases velocity on X, Y, and Z
  sphX= sphX+velX;
  sphY= sphY+velY;
  sphZ= sphZ+velZ;

  //X - cannot move outside of side walls

  float area = sphRad + 1;
  if (sphX > (worldSize/2)-area) {
    sphX = worldSize/2-area;
    velX = -velX;
  } else if (sphX<(-worldSize/2)+area) {
    sphX = -worldSize/2+area;
    velX = velX*-1;
  }
  //Y - cannot move outside of the ceiling and floor
  if (sphY > (worldSize/2)-area) {
    sphY = worldSize/2-area;
    velY = -velY;
  } else if (sphY<(-worldSize/2)+area) {
    sphY = -worldSize/2+area;
    velY=velY*-1;
  }
  //Z - cannot move outside the back and front
  if (sphZ > (worldSize/2)-area) {
    sphZ = worldSize/2-area;
    velZ = -velZ;
  } else if (sphZ<(-worldSize/2)+area) {
    sphZ = -worldSize/2+area;
    velZ = velZ*-1;
  }

  velX *= 0.99;
  velY *= 0.99;
  velZ *= 0.99;

  //translate the world to the middle of the space
  translate(width/2, height/2);   

  // to rotate on x-axis and y-axis
  float rotAngleY = map(sensorValues[0], 261, 762, -PI/4, PI/4);
  float rotAngleX = map(sensorValues[1], 259, 763, PI/4, -PI/4);
  //float rotAngleY = map(mouseX, 0, width, -PI/4, PI/4);
  //float rotAngleX = map(mouseY, 0, height, PI/4, -PI/4);

  if (sensorValues[1]>=763) {
    rotAngleX = map (sensorValues[1], 259, 763, 0, 0);
  }

  if ((sensorValues[0]>506)||(sensorValues[0]<499)) {
    rotateY(rotAngleY);
  }
  if ((sensorValues[1]>511)||(sensorValues[1]<499)) {
    rotateX(rotAngleX);
  }

  //world
  noFill();
  noStroke();
  drawBox(0, 0, 0, 400, 400, 400);    

  //side walls
  fill(50);
  noStroke();
  drawBox(200, 0, 0, 2, 400, 400);
  drawBox(-200, 0, 0, 2, 400, 400);

  //background
  fill(155);
  noStroke();
  drawBox(0, 0, -200, 400, 400, 2);  

  //ceiling and floor
  fill(220);
  drawBox(0, 200, 0, 400, 2, 400);
  drawBox(0, -200, 0, 400, 2, 400);

  //sphere that follows the position of hand
  noStroke();    
  fill(0);
  drawSphere (posX, posY, posZ, 10);

  //draw the sphere
  stroke(sphColor);
  noFill();
  drawSphere (sphX, sphY, sphZ, 30);
}

//function to draw multiple spheres
void drawSphere (float x, float y, float z, int size) {
  pushMatrix ();
  translate (x, y, z);
  sphere(size);
  popMatrix();
}

//function to draw multiple boxes
void drawBox(float x, float y, float z, float w, float h, float d) {
  pushMatrix();
  translate(x, y, z); // this way you can change the position of the box
  box(w, h, d); // you can only change the size of the box (w,h,d)
  popMatrix();
}

void keyPressed() {
  velX=random (-15, 15);
  velY=random (-15, 15);
  velZ=random (-15, 15);
}

void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 2 ], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = 'n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}

void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = 'n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
  if (sensorValues[2] == 1) {  
    velX=random (-15, 15);
    velY=random (-15, 15);
    velZ=random (-15, 15);
    sensorValues[2] = previousSensorValues;
  }
  if (previousSensorValues == 1) {
    sensorValues[2] = 0;
  }
}


//Processing Code Tab 2
// ======================================================
// Table of Contents:
// ├─ 1. Callbacks
// ├─ 2. Hand
// ├─ 3. Arms
// ├─ 4. Fingers
// ├─ 5. Bones
// ├─ 6. Tools
// └─ 7. Devices
// ======================================================

// ======================================================
// 1. Callbacks

void leapOnInit() {
  // println("Leap Motion Init");
}
void leapOnConnect() {
  // println("Leap Motion Connect");
}
void leapOnFrame() {
  // println("Leap Motion Frame");
}
void leapOnDisconnect() {
  // println("Leap Motion Disconnect");
}
void leapOnExit() {
  // println("Leap Motion Exit");
}


void leapMotionUpdate () {
  // ...

  int fps = leap.getFrameRate();
  for (Hand hand : leap.getHands ()) {


    // ==================================================
    // 2. Hand

    int     handId             = hand.getId();
    PVector handPosition       = hand.getPosition();
    PVector handStabilized     = hand.getStabilizedPosition();
    PVector handDirection      = hand.getDirection();
    PVector handDynamics       = hand.getDynamics();
    float   handRoll           = hand.getRoll();
    float   handPitch          = hand.getPitch();
    float   handYaw            = hand.getYaw();
    boolean handIsLeft         = hand.isLeft();
    boolean handIsRight        = hand.isRight();
    float   handGrab           = hand.getGrabStrength();
    float   handPinch          = hand.getPinchStrength();
    float   handTime           = hand.getTimeVisible();
    PVector spherePosition     = hand.getSpherePosition();
    float   sphereRadius       = hand.getSphereRadius();

    handX = handPosition.x;
    handY = handPosition.y;
    handZ = handPosition.z;
    hGrab = handGrab;  //renaming it because there is another handGrab and they would interfere with each other

    // --------------------------------------------------
    // Drawing
    //hand.draw(); //draws out the hand


    // ==================================================
    // 3. Arm

    if (hand.hasArm()) {
      Arm     arm              = hand.getArm();
      float   armWidth         = arm.getWidth();
      PVector armWristPos      = arm.getWristPosition();
      PVector armElbowPos      = arm.getElbowPosition();
    }

    // ==================================================
    // 4. Finger

    Finger  fingerThumb        = hand.getThumb();
    // or                        hand.getFinger("thumb");
    // or                        hand.getFinger(0);

    Finger  fingerIndex        = hand.getIndexFinger();
    // or                        hand.getFinger("index");
    // or                        hand.getFinger(1);

    Finger  fingerMiddle       = hand.getMiddleFinger();
    // or                        hand.getFinger("middle");
    // or                        hand.getFinger(2);

    Finger  fingerRing         = hand.getRingFinger();
    // or                        hand.getFinger("ring");
    // or                        hand.getFinger(3);

    Finger  fingerPink         = hand.getPinkyFinger();
    // or                        hand.getFinger("pinky");
    // or                        hand.getFinger(4);

    for (Finger finger : hand.getFingers()) {
      // or              hand.getOutstretchedFingers();
      // or              hand.getOutstretchedFingersByAngle();

      int     fingerId         = finger.getId();
      PVector fingerPosition   = finger.getPosition();
      PVector fingerStabilized = finger.getStabilizedPosition();
      PVector fingerVelocity   = finger.getVelocity();
      PVector fingerDirection  = finger.getDirection();
      float   fingerTime       = finger.getTimeVisible();

      // ------------------------------------------------
      // Drawing

      // Drawing:
      // finger.draw();  // Executes drawBones() and drawJoints()
      // finger.drawBones();
      // finger.drawJoints();

      // ------------------------------------------------
      // Selection

      switch(finger.getType()) {
      case 0:
        // System.out.println("thumb");
        break;
      case 1:
        // System.out.println("index");
        break;
      case 2:
        // System.out.println("middle");
        break;
      case 3:
        // System.out.println("ring");
        break;
      case 4:
        // System.out.println("pinky");
        break;
      }


      // ================================================
      // 5. Bones
      // --------
      // https://developer.leapmotion.com/documentation/java/devguide/Leap_Overview.html#Layer_1

      Bone    boneDistal       = finger.getDistalBone();
      // or                      finger.get("distal");
      // or                      finger.getBone(0);

      Bone    boneIntermediate = finger.getIntermediateBone();
      // or                      finger.get("intermediate");
      // or                      finger.getBone(1);

      Bone    boneProximal     = finger.getProximalBone();
      // or                      finger.get("proximal");
      // or                      finger.getBone(2);

      Bone    boneMetacarpal   = finger.getMetacarpalBone();
      // or                      finger.get("metacarpal");
      // or                      finger.getBone(3);

      // ------------------------------------------------
      // Touch emulation

      int     touchZone        = finger.getTouchZone();
      float   touchDistance    = finger.getTouchDistance();

      switch(touchZone) {
      case -1: // None
        break;
      case 0: // Hovering
        // println("Hovering (#" + fingerId + "): " + touchDistance);
        break;
      case 1: // Touching
        // println("Touching (#" + fingerId + ")");
        break;
      }
    }


    // ==================================================
    // 6. Tools

    for (Tool tool : hand.getTools()) {
      int     toolId           = tool.getId();
      PVector toolPosition     = tool.getPosition();
      PVector toolStabilized   = tool.getStabilizedPosition();
      PVector toolVelocity     = tool.getVelocity();
      PVector toolDirection    = tool.getDirection();
      float   toolTime         = tool.getTimeVisible();

      // ------------------------------------------------
      // Drawing:
      // tool.draw();

      // ------------------------------------------------
      // Touch emulation

      int     touchZone        = tool.getTouchZone();
      float   touchDistance    = tool.getTouchDistance();

      switch(touchZone) {
      case -1: // None
        break;
      case 0: // Hovering
        // println("Hovering (#" + toolId + "): " + touchDistance);
        break;
      case 1: // Touching
        // println("Touching (#" + toolId + ")");
        break;
      }
    }
  }


  // ====================================================
  // 7. Devices

  for (Device device : leap.getDevices()) {
    float deviceHorizontalViewAngle = device.getHorizontalViewAngle();
    float deviceVericalViewAngle = device.getVerticalViewAngle();
    float deviceRange = device.getRange();
  }
}



//Arduino Code

/*
  Thumb Joystick demo v1.0
  by:http://www.seeedstudio.com
  connect the module to A0&A1 for using;
*/
int buttonPin = 0;

void setup() {
  Serial.begin(9600);
}

void loop() {

  int pot1 = analogRead(A0);
  int pot2 = analogRead(A1);

  if (pot2 == 1023) {
    buttonPin = 1;
  } else {
    buttonPin = 0;
  }

  Serial.print(pot1);
  Serial.print(",");
  Serial.print(pot2);
  Serial.print(",");
  Serial.print(buttonPin);
  Serial.println();

  delay(100);

}

Interaction Lab: Lab 12-Media Controller (Daniel)

Lab Date: May 5th 2017

Instructor: Professor Daniel

Lab Partner: Andrew Huang

Aim of Today’s Lab: Control an image with sound using an Arduino controller in Processing.

Materials Used: Potentiometer, Bread Board, Jump Cables

20170505_132128

For the lab we decided to move the picture of the rocket up and down so that we could make it look like the rocket is being shoot. At the end we added a rocket shooting sound file as well.

Step 1 – Creating the circuit

The circuit for this exercise was pretty simple since we used one potentiometer. We connected the middle leg to Analog 0 and the other two to 5V and ground each.

Step 2- Serial Communication

This step was the most challenging for us since we could not get our Arduino value to Processing for a while. The value on Processing fluctuated a lot while we were receiving correct values on our Arduino serial monitor. With Dan’s help and by looking at Dan’s presentation, we found that println should be under the while loop.

Step 3- Image Upload

This part was not too challenging. We found an image of a rocket being launched. Then, we added it to Processing  by dragging and dropping it. After that, we coded Processing so that we could change the y coordinate of the image.

Step 4- Adding the sound

This part was also very simple. We added an example code for sound to our existing code. Then, we added a downloaded rocket launch sound.

Demo:

Conclusion and Reflection: I learned better how to control images and audios on Processing through this lab. Also, it was a good review of Arduino and Processing communication.

Sources:

https://www.extremetech.com/wp-content/uploads/2017/02/pslv-640×353.jpg

//Processing

/**
 * Load and Display 
 * 
 * Images can be loaded and displayed to the screen at their actual size
 * or any other size. 
 */
import processing.serial.*;
import processing.sound.*;

SoundFile file;

PImage img;  // Declare variable "a" of type PImage


Serial myPort;
int valueFromArduino;

void setup() {
  size(640, 360);
  // The image file must be in the data folder of the current sketch 
  // to load successfully
   printArray(Serial.list());
  // this prints out the list of all available serial ports on your computer.
  
  myPort = new Serial(this, Serial.list()[ 2 ], 9600);
  img = loadImage("pslv-640x353.jpg");  // Load the image into the program 
  file = new SoundFile(this, "Rocket Launch Sound Effect.mp3");
  file.play();
  file.loop();
}

void draw() {
  // Displays the image at its actual size at point (0,0)
   while ( myPort.available() > 0) {
    valueFromArduino = myPort.read();
    println(valueFromArduino);
  }
  background(0);
  //image(img, 0, 0);
  // Displays the image at point (0, height/2) at half of its size
  image(img, 0, valueFromArduino, img.width, img.height);
}

=====================================================================================

//Arduino

// the setup routine runs once when you press reset:
void setup() {
  // initialize serial communication at 9600 bits per second:
  Serial.begin(9600);
}

// the loop routine runs over and over again forever:
void loop() {
  // read the input on analog pin 0:
  int sensorValue = analogRead(A0);
  // print out the value you read:
  Serial.write(sensorValue);
  delay(10);        // delay in between reads for stability
}

Interaction Lab: Lab 11- Drawing Machines (Daniel)

Lab Date: April 28, 2017

Instructor: Professor Daniel

Lab Partner: Andrew Huang, (Amy Mao, Kathy Wang)

Aim of Today’s Lab: Create a drawing machine using stepper motors

Materials Needed: Stepper Motor, Green Legs, Pins, Red Circulating Support, Arduino Kit, a Pen, Stepper Motor Holder

20170428_132459

Step 1- Creating the half of the machine

We started with creating the half of the machine. First, we built the circuit and the physical component of the machine. Creating the physical component of the machine was easy. We just connected the legs with pins and the motor. The motor was connected to the holder at the end as well.

20170428_134603

We followed the circuit diagram provided on the IMA Website.

bipolarStepperMotor-300x254

Creating a circuit was a bit challenging since we did not follow the right colors for the Jump Cables. When the circuit became complicated, it was hard to keep track since the colors were mixed up. However, we figured the circuit out at the end:

20170428_134557

Step 2- Potentiometer

A Potentiometer was added to the circuit to control the speed of the motor. The Middle leg was connected to A0, and the others were connected to ground and 5V. Also, the code for speed control was added from the library.

Step 3- Connection

The last step was connecting two parts together. We just connected the legs using a pin and put a pen in the middle:

Reflection:

It was a fun and a simple lab. The art piece that we created was truly beautiful. I learned that using the right color jump cable is very important.

 

Interaction Lab: Final Project Proposal (Daniel)

Final Project Due Date: May 19, 2017

Partner: Nathalia Lin

Instructor: Professor Daniel

Before taking Interaction Lab, I had never thought deeply about interaction. In fact, I’d thought that technology would only cause the lack of interaction among people. However, throughout the semester, I have learned that technology can actually increase the amount of interaction among people. For instance, I had many interactions with other students, professors and IMA fellows when I was preparing for and presenting my projects. These interactions not only helped me academically, but also let me build stronger relationships with other people. If there weren’t the projects, I would not have been able to have these priceless interactions.

After I realized that I had amazing interactions with others through the projects and started preparing for my final project, I started to think about what interaction really is. It was an important question that came into my mind since the final project for “Interaction Lab” is supposed to be “Interactive”. To build something Interactive, I had to know what interaction really means. Then, I started with searching the definition of the word. On the dictionary the definition of the word “Interaction” is written as “reciprocal action of influence”. To me, the keyword in that definition was “Reciprocal”. My experiences of interactions through my projects were “reciprocal” in that I learned many things from the users’ inputs and the users also enjoyed trying my projects. Interaction is something that happens both ways. I believe that if something happens only one way, it would not be an interaction. Therefore, my definition of interaction is an effect that happens both ways, not one way.

For the final project, I wanted to create something that is highly interactive with its users. I first thought about creating an interactive animation, an animation that reacts to users input (from the sensor). However, my partner suggested about creating a game, and that was a compelling idea to me since I thought game has more potential of interaction. Through a game, besides an interaction between the user and the game, competitive interactions among users were also possible. So, our idea for the final project is creating a 3D Tetris game with Arduino, Processing and Leap Motion. Arduino would be used for the reset button (also possibly joystick to change dimensions), processing would be used to design 3D objects, and leap motion would be used to move around the blocks with the users’ own hands.

There is a 3D Tetris created with leap motion: https://apps.leapmotion.com/apps/fragmental-3d/osx. This is the project where we got our idea from. As it seems, the project is very well built with excellent graphics. The game’s quality is high in that it has different difficulty settings, scoring power, speed and so on. Sound effects are also amazing. The game also has different background themes. In short the game is nearly perfect (even the details of it). But, I think it would allow more interaction among users if the game had online multiplayer option. Since the amount of time given to us is not too much, I believe it would be difficult for our group to create as perfect as this. My goal is to create as much interaction as possible through the project, although it might lack some details compared to the model project.

Interaction Lab: Lab 10-3D Modeling (Daniel)

Lab Date: April 21, 2017

Instructor: Professor Dan

Aim of Today’s Lab: Create a 3D Model (wearable device, game controller, etc) using Tinkercad.

My choice was to create a game controller that uses an accelerometer.

-Materials needed: Accelerometer, Tinkercad, Caliper

I started with the basic design of the controller. I created a box in the middle and then created two handles by putting two torus ,and attached them to the parallel sides of the box. I made sure that the size of the handles were big enough to be held by measuring my own fingers and comparing them to the size of the torus that were sticking out from the box.

My next concern was creating a space for the accelerometer. First, I made a box shaped hole in the middle of the controller. I made the hole to be 25 mm for width and length since the length and width of the accelerometer were both 20 mm. I made the depth to be slightly deeper than the height of the acclerometer (since this data was not provided on the data sheet, I had to measure it using caliper) so that it would not stick out from the controller.

The last step I took was to create spaces for the screws to hold the accelerometer. I made holes using two cylinders with the radius of 1.1 mm. I picked the size after looking at the data sheet of accelerometer and figuring out the radius of holes for the screws. This activity was a bit challenging since I had a hard time aligning everything perfectly. Also, it was difficult for me to get the space between the holes right ( I measured using a ruler).

Conclusion: I realized how fun 3D designing can be. I also noticed how difficult it could be since I had to pay attention to minor details. I learned how to use Tinkercad and understood 3D designing better.

Bodacious Leelo

 

Interaction Lab: Lab 9_ Making My Own Stamp and AutoDesk Field Trip (Daniel)

Class Date: April 11, 2017

Instructor: Professor Daniel

Aim of the Lab: Design your stamp to be laser cut using Adobe Illustrator.

Materials to be prepared: Adobe Illustrator

I started approaching the project by thinking about the design of the stamp. I decided to put my last name in Chinese character in the middle and my first name with English on top of the stamp. I came up with the following design:

20170411_191806

Then, I watched the provided tutorial video to learn how to use Adobe illustrator.

Link: https://vimeo.com/212333844

Following the tutorial I created the basic design first using a square and a circle. Then, I typed my last name in the middle. After that, I used a ‘Type on a path tool’ to put my first name following the circle.

Picture Link: stamp

Then I had to adjust the stamp to the real scale. When I tried adjusting the first time, the stroke size was too big.

Picture Link: stamp.2

Finally, I adjusted the stroke size and reflected the design vertically since the stamp would be vertically flipped, following the tutorial. Here is the completed design.

Picture Link: stamp.3

Conclusion and Reflection: It was exciting for me that I started learning how to laser cut. The tutorial provided was very detailed and easy to follow, so this project was not too challenging. I felt that I would have to play around with AI a lot more to be comfortable with it.

An Auto desk field trip was a great experience in that I got to know about the company better. I learned about their products, goal, and strategy. Also, it was impressive to see their 3D printed creations.

20170414_140206 20170414_140227 20170414_140255 20170414_140304 20170414_140737

 

 

Interaction Lab Midterm Project: Draw Your Life

Project Presentation Date: March 30, 2017

Instructor: Professor Daniel

Project Partner: Nathalia Lin

Project Title: Draw Your Life

  • A painting tool that lets users paint in a color from real life

Description

“Draw Your Life” is a painting tool that can detect color from real life and lets its users paint in that color. The color sensor detects color from many different objects, and users can paint on their computer screens with the detected color. Users can also adjust the size of the stroke by pressing arrow buttons (left and right) on the keyboard. In addition, users can erase what they painted by pressing ‘e’ button on the keyboard and clicking and dragging the mouse.

Demo of the Project (Video)

Below is the completed version of the project:

 

Below is the picture of the completed physical component of the project (Arduino circuit covered with box):

20170331_135510

 

Conceptual Development

Ideation:

The motivation for the project for me was when my friend told me that she would be happy if a photo editing tool could get a color that you like from the physical life (hair color specifically) and let you edit your photo with that color. So I thought it would be a cool idea to start working on a color-sensing paint tool.

Below is the brainstorm sketch that I made in the beginning (it says “Use Color Sensor?/ RGB/ Processing/ Then Paint? (What to be added?)”):

20170401_175229

Research:

When we first started our project, we did not even know that a color sensor existed. So, I went and asked about a sensor that can detect color to the IMA fellows and found out that there are. The first sensor that I checked out was “TCS3200 Color Sensor”.  After following the tutorials provided on “How To Echatronics” I realized that the RGB values that I would receive were not realistic (I received negative numbers). The problem continued even after several IMA fellows and a professor helped us. Therefore, we decided to use a different color sensor, Grove 12C Color sensor.

The link for both TCS3200 Color Sensor and Grove 12C Color sensor are provided below:

http://wiki.seeed.cc/Grove-I2C_Color_Sensor/

Arduino Color Sensing Tutorial – TCS230 TCS3200 Color Sensor

The fist aim for the project was detecting the right RGB Values, and the 12C color sensor allowed us to get reasonable RGB values, so we decided to keep using it.

Technical Development

As I mentioned earlier, our group started with getting the right RGB Values. First we built the Arduino Circuit by connecting VCC to 5V, GND to GND, SDA to SDA, and SCL to SCL (which are A4 and A5 pins).

20170323_171505

Then we used the sample code provided from the lecture about multiple serial communication to send values from Arduino to Processing. And the result was successful (getting reasonable values for different colors on Processing):

The next step was creating a painting tool inside processing. Here is a brief brainstorming about what functions were to be added to the drawing tool:

20170401_181029

We started with an ellipse that would follow the mouse by setting it up as mouse x and mouse y. We also set up a function to make sure that the sensor is not always reading the value. We set up a void mouse click function to read the value only when the mouse is clicked.

After that we decided to make the painting done by strokes instead of ellipses. We set an if statement to not make the stroke negative (if stroke is less than 0, it equals 1).  Then, we set up our mouse pressed for left to draw and mouse clicked for right to read sensor value.

Since we decided to set up our background as white, we could make our erase as another stroke that would paint white. We set up an eraser tool by painting white when ‘e’ key is pressed and the left mouse button is pressed. But then, the problem emerged- the tool would draw and erase at the same time. At this point, I kept working on processing to fix this problem, and my partner decided to work on the physical component of the project. I could solve this problem by making a new function ( I named it void erase). My partner covered the sensor and the circuit with some card boards, making the project look much neater.

Lessons Learned

I believe the user test was very successful. Most of the users thought the idea of the project was brilliant. The shortcoming of the project would be that the sensor’s being inaccurate. Often times the sensor read the color as a different color (e.g. skin is read as red). To avoid this type of failure, users should try to attach the sensor as close to the object as possible. Also, the user should use a non shiny surface to be detected.

 

 

Arduino 

#include <Wire.h>
//#include <math.h>
#include <GroveColorSensor.h>

void setup()
{
	Serial.begin(9600);
	Wire.begin();
}

void loop()
{
	int red, green, blue;
	GroveColorSensor colorSensor;
	colorSensor.ledStatus = 1;							// When turn on the color sensor LED, ledStatus = 1; When turn off the color sensor LED, ledStatus = 0.
	while(1)
	{
		colorSensor.readRGB(&red, &green, &blue);		//Read RGB values to variables.
		delay(300);
      Serial.print (red);
        Serial.print(",");
        Serial.print(green);
        Serial.print(",");
        Serial.print (blue);
        Serial.println ();
		colorSensor.clearInterrupt();
	}
}


Processing

int erase = 0;
import processing.serial.*;
int red, green, blue;
String myString = null;
Serial myPort;
int myMouseButton;

int NUM_OF_VALUES = 3;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;      /** this array stores values from Arduino **/
int x= 2;

void setup() {
  size(500, 500);
  background(255);

  setupSerial();
}


void draw() {
  updateSerial();
  erase ();
  printArray(sensorValues);

  // use the values like this!
  // sensorValues[0] 

  //red=sensorValues[0];
  //green=sensorValues[1];
  //blue=sensorValues[2];
  if (mousePressed && (mouseButton == LEFT)&&erase==0) {
    stroke(red, green, blue);
    if (x<0) {
      x=1;
    } //stroke weight should not be negative

    strokeWeight (x);
    line(pmouseX, pmouseY, mouseX, mouseY); //draw if left mouse button is clicked
  }
}


void erase() {
  if (keyPressed) { 
    if (key=='e'&&mousePressed) { // eraser if 'e' is pressed
      erase =1;
      stroke (255);
      line (pmouseX, pmouseY, mouseX, mouseY);
    } else {
      erase = 0;
    }
    if (x<0) {
      x=1;
    } //stroke weight should not be negative
    strokeWeight (x);
  }
}

void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[0], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = 'n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}

void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = 'n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}
//read values only when the mouse's right button is clicked
void mouseClicked() {
  if ( myMouseButton == 1) {
    red=sensorValues[0];
    green=sensorValues[1];
    blue=sensorValues[2];
  }
}
//draw when LEFT Button of Mouse is clicked
void mousePressed () {
  if (mouseButton == LEFT) {
    myMouseButton=0;
  } else {
    myMouseButton=1;
  }
}
void keyPressed() {
  if (key == CODED) {
    if (keyCode == RIGHT) {    //Make Stroke thicker when right arrow is pressed
      x=x+3;
    } else if (keyCode == LEFT) {     //Make Stroke thinner when left arrow is pressed
      x=x-3;
    }
  }
}

Interaction Lab: Lab 6_ Serial Communication by Jung Hwan

Lab Date: March 17, 2017

Instructor: Professor Daniel

Lab Partner: Andrew Huang

Aim of Today’s Lab: Make Arduino and Processing talk to each other using serial communication.

 

Exercise 1- Ring the Buzzer by clicking the mouse (Processing to Arduino)

Materials Used: Buzzer, Bread Board, Arduino, Jump Wires

We built the circuit first, connecting the buzzer to digital pin 13 and ground of the Arduino. For the code, we set up the buzzer to ring at the tone of 1200 when the mouse is clicked (running the processing code). We did not struggle that much for this exercise since we had the example code (for LED) from the last class. Below is the result:

 

Exercise 2- Changing the color of the Square on Processing by pressing the Button (Arduino to Processing)

Materials Used: Jump Wires, Arduino, 10k Resistor, Button, Bread Board

This exercise was a bit more challenging for us, since we could not figure out the code for a while. With the help of some IMA fellows we could complete the project. It turned that our delay was too low and the circuit had a minor problem (ground jump wire was connected on a wrong spot). As we approached our first exercise, we built the circuits first by connecting the button to  ground and pin 13 through the 10k resistor; another leg was connected to 5V. And then we connected the Arduino and ran the code.

Conclusion and Reflection: I learned how to connect Arduino and Processing to communicate with each other. I completed 2 exercises; sending data from Processing to Arduino (by clicking) and the other way around (by pressing the button). I recognized that serial communication between these two languages could be more challenging than the earlier exercises that we’ve done since we have to work with both of the languages. I felt that we should always double check the circuit when using Arduino and double check the port when we establish communication between different languages.

Buzzer Code: 

Processing:

import processing.serial.*;
Serial myPort;
int val;

void setup(){
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[2], 9600);
}

void draw(){
  if (mousePressed){
    myPort.write('H');
//when the mouse is pressed, send the high value 
  } else {
    myPort.write('L');
  }
}

Arduino:

int val;

void setup() {
  Serial.begin(9600);
  pinMode(13, OUTPUT);
  

}

void loop() {
  while (Serial.available()){
    val = Serial.read();
  }
  if (val =='H'){
    tone(13, 1200, 20);
//when the value is high, ring the buzzer
  } else if (val == 'L'){
  }
  delay(10);
}

_________________________________

Button Code:

Processing:

import processing.serial.*;
Serial myPort;
int val, value;

void setup(){
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[2], 9600);
}

void draw(){
  if ( myPort.available() > 0) {
  value = myPort.read();
  println(value);
  }
  rect(0, 1, 50, 50);
  if (value == 1){
    fill(0, 0, 0);
//if the button is pressed (value is 1) change the square color
  } else {
    fill(255, 255, 255);
//if the button is unpressed turn back the square color
  }
}

Arduino:

int val;

void setup() {
  Serial.begin(9600);
  pinMode(13, INPUT);
  }

void loop() {
  int read = digitalRead(13);
  if (read == 1){
    Serial.write(1);
  //if the button is pressed send the value 1 
} 
else {
    Serial.write(0);
  }
  delay(100);
}

Interaction Lab: Stupid Pet Trick Documentation by Jung Hwan (Daniel)

Project Presentation Date: Mar. 10, 2017

Instructor: Professor Daniel

Aim of the Project: Make the Arduino do something interactive and stupid. My idea was to create a sweaty hand detector with a moisture sensor. I wanted to detect how sweaty the hand is when it touches the sensor and turn the servo to put up the “Sweaty Hand!!” sign and buzz the buzzer when the sensor value is high (which means the hand is sweaty).  Below is the simple sketch that I drew:

20170309_140154

Materials Used: glove, arduino, servo, bread board, buzzer, jumper cables, moisture sensor, card boards, electrical tape

Stage 1- Building the Circuit

I started building the project with the circuit. I followed the diagrams provided in our arduino toolkit. I connected the positive side of the buzzer to pin 8 and negative side to the ground. I also connected the servo to ground, 5V and pin 10. In addition, I connected the moisture sensor to ground, analog pin and 5V. A bread board had to be used since there was not enough space on the aruduino itself.  Below is a picture of the circuit:

20170308_162355

Stage 2- The physical component

I cut some cardboard to put under the servo to turn it without being disturbed. I cut some other smaller pieces to attach it to the wing of the servo to put the “Sweaty Hand!!” sign on it. I also attached the moisture sensor on the glove using electrical tape (I used regular tape in the beginning, but Professor Anthonius notified me later that it would work better with electrical tape).

20170308_162434                      20170308_175435

Stage 3- Coding

This is the stage that I struggled the most. I tried coding servo and buzzer output with moisture sensor input. I looked at the codes that we learned from our lab (temperature alarm) session and the arduino library (for servo).  With the help of Professor Anthonius I could code for the servo turn output and buzzer ring output according to the moisture sensor input.

Completed Project:

Note: I changed the tape from normal to electrical at the end- below is the picture of the final project with electrical tape:

20170308_182152

Conclusion and Reflection: I had so much fun building something that I designed for the first time. The most difficult part for me was coding. I felt that I would have to study coding more. I’ll have to remember what professor Daniel told me; just Copy and Pasting” would not work.

 

#include <Servo.h>

int sensorPin = A0; // select the input pin for the potentiometer
int sensorValue = 0; // variable to store the value coming from the sensor
int pos = 0; 
Servo myservo;
void setup() {
  Serial.begin(9600);
  pinMode(10, OUTPUT); // output for the servo
  myservo.attach(10);
  pinMode(8, OUTPUT); // output for buzzer
}
void loop() {
  // read the value from the sensor:
  sensorValue = analogRead(sensorPin);
  Serial.println(sensorValue);
  delay(10);
  if (sensorValue > 150) {
    myservo.write(95);
     //if the sensor is greater than 150, turn the servo to 95 degrees
    tone(8,1047,20);
    //if the sensor value is greater than 150, ring the buzzer
  }
  else {
    myservo.write(5);
  //if the sensor value is less than 150,return the servo to 5 degrees
  }
}

Interaction Lab: Lab 4-Sensors

Lab Date: March 3, 2017

Instructor: Professor Daniel

Documented by: Jung Hwan Park

Aim of Today’s Lab: Connect a sensor to the Arduino using it as input for some kind of output.

I decided to work with a moisture sensor to light up an LED light.

Step 1: Input

Materials: Moisture sensor, Jump Wires, Arduino, Wet sponge

I connected the moisture sensor to the Arduino with Jump wires, then uploaded the code. With some help from professor Daniel, I could check the humidity was changing when the sensor was in contact with a wet sponge.

 

Step 2: Output

Materials: 220R resistor, LED light, Jump wires, Arduino, Bread board, Wet Sponge, Moisture Sensor

I wanted to light up an LED light with the humidity input. With the help of Professor Antonius, I connected the LED light to a bread board, eventually connecting it to the Arduino going through the resistor. By creating  conditionals (light up when it is humid) I could successfully light the LED light up.

Conclusion and Reflection: It was generally a fun lab. I learned how to work with a moisture sensor. I practiced how to create conditionals as well. I also gained basic understandings of input and output.

// Test code for Grove - Moisture Sensor 
int sensorPin = A0; // select the input pin for the potentiometer
int sensorValue = 0; // variable to store the value coming from the sensor7

void setup() {
    // declare the ledPin as an OUTPUT:
    Serial.begin(9600);
}
void loop() {
    // read the value from the sensor:
    sensorValue = analogRead(sensorPin);
    Serial.print("sensor = " );
    Serial.println(sensorValue);
    delay(1000);
}


//code for the added LED light 
int sensorPin = A0; // select the input pin for the potentiometer
int sensorValue = 0; // variable to store the value coming from the sensor7

void setup() {
    // declare the ledPin as an OUTPUT:
    Serial.begin(9600);
    pinMode(13,OUTPUT);
}
void loop() {
    // read the value from the sensor:
    sensorValue = analogRead(sensorPin);
    Serial.println(sensorValue);
    delay(10);
    //if the sensor is greater than 500 then turn on the LED
    if (sensorValue>500){
      digitalWrite(13,HIGH);
    }else {
      digitalWrite(13,LOW);
    }
}