Documentation of the Final Project

Author: Wanchen Zhao (Leslie)

Instructor: Antonius

Partner: Jiaqiao Xiang (Frank)

Project: An User Experience Device for Marketing Purpose of the New Thor Movie

Techniques: Processing (include drawing with Processing); Leap Motion; Serial communication; RGB

Materials: cardboard box, solderless breadboard, Arduino, servo, wires, Leap Motion

Inspiration and Outline of the Project

We wanted to do a follow-up from our midterm project, by keeping the patterns we drew back then. After deciding to create a user experience for the new Thor movie, in which Thor meets Dr. Strange, we decided that we would each come up with an idea for a game based on our own patterns, which we will control with Leap Motion. I came up with the idea of making my pattern into a jigsaw puzzle, while Frank decided to let users explore his pattern by moving hand across the Leap Motion. These are the two rounds of our user experience; only by finishing mine can the user go into Frank’s. To incorporate Arduino with Processing, we decided to make a box that would open at the very end of the process, in which we place a movie coupon that can be rewarded to the users for finishing the game.

The Making of My Puzzle

To make the puzzle, I took out the part of the pattern from my midterm code, and modified it to make it a still pattern. The code is the first one in code source. Afterwards I simply ran the code, screenshot the pattern, and used online application to cut the pattern into 9 pieces by setting the mode to “3*3”. This is my pattern, or the effect when the jigsaw puzzle is finished:

Screen Shot 2017-05-19 at 1.22.57 PM

To make the puzzle, I would have to import the library of Leap Motion and create booleans to move my puzzle pieces when Leap Motion senses hand gestures, and to make the next piece appear when the previous piece is in the right position. Of course it would be very hard to move the pieces into exactly the right position, so with Jiwon’s help I set an integer called “off” at the beginning, which means offsetting a given range of inaccuracies when completing the puzzle. I learned about the function of “handPressed = !handPressed”, in which “handPressed” is a boolean that was set as false in the setup; with the “!”, the boolean can now change freely between being true and false when user’s fingers open and close. This is shown as the second code in the code source.

To truly make the puzzle pieces work, I would have to create another tab in my code called “puzzles”. In this tab I created a class called “Puzzles”, in which I defined all the right positions of the pieces, before loading them into the first tab. This is the third code in the code source.

After these two steps, the code would work on its own as a puzzle when the laptop is connected to Leap Motion. I didn’t like the little ellipse that shows as a signal of hand, which would merely change colors when the user’s fingers open and close. I want to make it more vivid. So I replaced the ellipse with the hand emojis:

hand_whiteok hand_white

I downloaded a picture of Dr.Strange and edited the puzzle’s instruction on it, placed the picture at the beginning of the puzzle, and used booleans and “mousePressed” to make the puzzle appear. The instruction page is the picture below. With the same method I also added a picture when the puzzle is finished. The way I told the computer to work is to have the picture appear when the last piece is in the right position. The important thing is to keep the images in a correct order, because the code itself runs in an order.

startImg

In addition, for Frank’s pattern, I used my trigonometry knowledge to figure out the coordinates of the star in his pattern for him. This is the draft:

IMG_8846

Putting the Codes Together

We did this after our individual codes were done, and this is the most difficult part of the entire project. The first thing was that both of us had a lot of booleans and counters, so it was really confusing and messed up when we first put the code together. Therefore we had to first change the names of our booleans and counters in our individual codes before the merge.

The biggest problem when we merged our code was that when mine was done, or even before mine far from complete, Frank’s complete pattern would simply appear directly on the screen. For a whole morning couldn’t figure out why this was happening. Finally it occurred to me: both games require Leap Motion, so with our code not properly merged, they were actually running at the same time. For the times when Frank’s pattern appeared long before my puzzle was finished, the hand movement when doing my puzzle was already enough to complete his pattern. To avoid this, Luis instructed us to use a boolean that runs through the entire 600-line code, and this tells the computer to only run Frank’s code after my puzzle is complete.

I discovered that two puzzle pieces, the 7th and the 9th, were the hardest to control with Leap Motion. Therefore I decided to put them on the screen at the very beginning, to both provide users with a little sneak peak and to make the game easier. So in setup, I let them appear at the same time as the 1st piece does, but in their right positions.

We also added the videos, which Frank edited with iMovie from the movie’s trailers, to the beginning and end of the entire game. Following the rule that the code runs in an order, we put the beginning video before the first introduction picture. The problem was the video wouldn’t disappear after it finished. Luis introduced to us the functions of “duration()” and “time()”, which we used to measure the length of the video, and afterwards used the “if{}” function to let the video disappear when the time passed since running the code is longer than the length of the video. Each part of Frank’s pattern is defined with booleans named from “show1” to “show11”, which were all set as false at the very beginning. The condition of running the ending video is for all these 11 “show”s to be true.

Serial Communication

Our Serial communication part was much simpler than Processing, since all we used was a servo to open the box. This is the circuit:

IMG_8845

The Arduino code is the fourth one in code source. The principle is very simple: basically when the ending video starts playing, Processing writes “1”; when Arduino reads “1”, the servo turns 180 degrees. Otherwise Processing would always write “0” and Arduino doesn’t react. The relevant code in Processing is incorporated into the “if{}” where the ending video plays.

Physical Device

I was in charge of making the box. I color printed both of our patterns, logos of Dr.Strange and Thor, cut them out and glued them onto the cardboard box. I also used a black marker to make sure that even the bits where the papers left out were also black. I put the Arduino and the breadboard at the bottom, pulled out the servo and covered these devices with a cardboard of appropriate size. I found online a logo of the movie and a standard-looking movie coupon pattern, and merged them together with photo editing app. Then I printed it out. On top of this cardboard, I placed the movie coupon.

CCCCCopoun

IMG_8869

We thought that everything was almost done; all we had to do is to control the box lids with the servo. But it turned out that it was very hard. After countless failures, we finally got inspiration from Antonius to let the servo lock and unlock the box by placing it at the top and attaching a piece of long, thin cardboard to lock the box on the servo’s wing. This was a really great idea and the making was so much simpler than using the servo to open the box!

To this point our project is all finished. The complete process and all the hard work is shown in this video:

The final code is the last one in source code.

I really enjoyed presenting it on the IMA final show. People “wow”ing over our project really gave me a sense of achievement. When the founder of Hackidea Bank, which I believe is a company of relative industry, even gave me her card after seeing our project, how I wish I had majored in IMA!

To Be Improved

The first thing is that we would add another transition between our two games, to give tutorial to controlling Frank’s pattern. During the IMA show, we found out that this is the least user-friendly part of our project, and we would’ve improved it in this way if we had had more abundant time.

There’s one more thing I would improve regarding my jigsaw puzzle. For the virtual hand I used only white hand’s emojis, but our users will be of different races, and I certainly don’t want to be offensive. So I would also enable the users to choose hand colors, since there are indeed emojis of different skin colors.

Afterwords

I learned a lot from a semester in Interaction Lab. Without it, I could have never learned such cool things like using Processing or Serial communication. The most amazing thing about IMA major is that it’s a combo of many traditional majors, including computer science, art and probably engineering. It involves creativity, patience, professional knowledge and lots of hard work. It made me really think about how the technologies that facilitate our daily lives work; this is something that I’ve never paid attention to before. In a world of emerging technologies, even though not everybody is an expert in this, I still find it necessary to at least know something about it.

I’m a business major, and I’ve realized how Interaction Lab is related to business through our final project, which was created for marketing purpose. Even though marketing is still quite different from my finance track, I still find significant interconnection between IMA and what I do. A field that seemed so irrelevant before suddenly becomes much closer because of Interaction Lab.

The biggest gain from Interaction Lab that I want to talk about is far from how fun it is to explore a new major, or how many obstacles I’ve overcome; it’s that it can really complete my personality. As a finance major, I demand high efficiency, quick mind and quick decisions; investment opportunities may be gone in a few seconds. But IMA is so different; it requires revising one’s work, coming up with new ideas along the way, and perfecting everything over and over again. Even though achieving this has been more challenging for me, I do believe that a person’s characteristics shouldn’t be limited by what he/she does; it should be complete for the person to fit into all kinds of environments.

Last but certainly not the least, I met cool professors and fellows, all of which are able, helpful and pleasant people. I’m getting jealous of IMA students for having such a inclusive and friendly community!

I’m infinitely grateful for all the knowledge, help and support from my professor Antonius and all the fellows. I’ve enjoyed all the time I spent this semester on the 8th floor, and I’m looking forward to an opportunity of being back for more!

//#1 My jigsaw puzzle pattern's code:

PFont myFont;
int counter;
int step = 15;

void setup() {

  background(0);
  smooth();
  size(600, 600);

  myFont = createFont("AppleMyungjo", 40);
  textFont(myFont);
  textAlign(CENTER, CENTER);
}

void draw() {
  pushStyle();
  textSize(40);
  counter+=30;

  pushMatrix(); 
  fill(255,100,20);
  translate(width/2, height/2);
  text("β", 0, 180);
  text("γ", 30, 160);
  text("δ", 60, 180);
  text("η", 90, 160);
  text("θ", 120, 150);
  text("ξ", 150, 120);
  text("β", 170, 80);
  text("γ", 190, 40);
  text("δ", 190, -40);
  text("η", 175, -80);
  text("θ", 155, -110);
  text("ξ", 130, -140);
  text("β", 110, -160);
  text("γ", 90, -175);
  text("δ", 60, -175);
  text("η", -40, -180);
  text("θ", -70, -160);
  text("ξ", -100, -150);
  text("β", -130, -130);
  text("γ", -150, -100);
  text("δ", -170, -60);
  text("η", -170, 40);
  text("θ", -160, 80);
  text("ξ", -140, 110);
  text("β", -120, 130);
  text("γ", -90, 145);
  text("δ", -70, 170);
  text("η", -40, 175);
  popMatrix();  

  noFill();
  ellipse(300, 300, 450, 450);
  ellipse(300, 300, 425, 425);
  ellipse(300, 300, 300, 300);
  ellipse(300, 300, 121.132, 121.132);
  ellipse(300, 300, 150, 150);
  fill(0, 0, 0);
  ellipse(300, 118.75, 62.5, 62.5);
  ellipse(118.75, 300, 62.5, 62.5);
  ellipse(300, 481.25, 62.5, 62.5);
  ellipse(481.25, 300, 62.5, 62.5);
  noFill();
  ellipse(300, 118.75, 60, 60);
  ellipse(118.75, 300, 60, 60);
  ellipse(300, 481.25, 60, 60);
  ellipse(481.25, 300, 60, 60);
  noFill();
  rect(193.935, 193.935, 212.132, 212.132);
  quad(150, 300, 300, 150, 450, 300, 300, 450);
  fill(255, 100, 20);

  textFont(myFont);
  textAlign(CENTER, CENTER);
  textSize(60);

  text("β", 310, 118.75);
  text("α", 128.75, 290);
  text("γ", 315, 471.25);
  text("λ", 495, 300);

  pushMatrix();
  translate(width/2, height/2);
  rotate(radians(counter));
  noFill();
  stroke(255, 100, 20);
  rect(0, 10, 50, 50);

  popMatrix();
}


//#2 My puzzle - first code:

import de.voidplus.leapmotion.*;

LeapMotion leap;

Puzzle[] puzzles = new Puzzle[9];

boolean handPressed=false;
boolean mouseChange = false;
boolean puzzleChange = false;
boolean end = false;

PImage handImg;
PImage okHandImg;
PImage startImg;
PImage endImg;

int imgW;
int imgH;

int off = 50;
int counter3 = 0;
int handX, handY;

void setup() {
  size(600, 600);

  handImg = loadImage("hand_white.png");
  okHandImg = loadImage("ok hand_white.png");
  startImg = loadImage("startImg.jpg");
  endImg = loadImage("endImg.jpg");

  imgW = width/3;
  imgH = height/3;

  puzzles[0] = new Puzzle(0, loadImage("img1.png"), imgW / 2, imgH /2);
  puzzles[1] = new Puzzle(1, loadImage("img2.png"), 3 * imgW / 2, imgH / 2);
  puzzles[2] = new Puzzle(2, loadImage("img3.png"), 5 * imgW / 2, imgH / 2);
  puzzles[3] = new Puzzle(3, loadImage("img4.png"), imgW / 2, 3 * imgH / 2);
  puzzles[4] = new Puzzle(4, loadImage("img5.png"), 3 * imgW / 2, 3 * imgH / 2);
  puzzles[5] = new Puzzle(5, loadImage("img6.png"), 5 * imgW / 2, 3 * imgH / 2);
  puzzles[6] = new Puzzle(6, loadImage("img7.png"), imgW / 2, 5 * imgH / 2);
  puzzles[7] = new Puzzle(7, loadImage("img8.png"), 3 * imgW / 2, 5 * imgH / 2);
  puzzles[8] = new Puzzle(8, loadImage("img9.png"), 5 * imgW / 2, 5 * imgH / 2);

  imageMode(CENTER);

  puzzles[0].showImg = true;

  leap = new LeapMotion(this);
}


void leapOnInit() {
}
void leapOnConnect() {
}
void leapOnFrame() {
}
void leapOnDisconnect() {
}
void leapOnExit() {
}

void draw() {
  background(255);

  if (puzzleChange == true) {
    puzzles[0].showImg = true;
    puzzles[6].showImg = true;
    puzzles[8].showImg = true;
    puzzles[6].x = puzzles[6].rightX;
    puzzles[6].y = puzzles[6].rightY;
    puzzles[8].x = puzzles[8].rightX;
    puzzles[8].y = puzzles[8].rightY;
  }
  if (mousePressed == true) {
    puzzleChange = true;
    mouseChange = true;
  };
  if (handPressed == false && puzzles[8].x == puzzles[8].rightX && puzzles[8].y == puzzles[8].rightY ){
    end = true;
  }
  if (handPressed == true) {
    puzzles[counter3].x = handX;
    puzzles[counter3].y = handY;
  }

  for (int i=0; i<puzzles.length; i++) {
    puzzles[i].display();
  }

  if (handPressed == false && puzzles[counter3].x >= puzzles[counter3].rightX - off && puzzles[counter3].x <= puzzles[counter3].rightX + off && puzzles[counter3].y >= puzzles[counter3].rightY - off && puzzles[counter3].y <= puzzles[counter3].rightY + off) {
    puzzles[counter3].x = puzzles[counter3].rightX;
    puzzles[counter3].y = puzzles[counter3].rightY;

    if (counter3 < puzzles.length - 1) {
      counter3++;
    }
    puzzles[counter3].showImg = true;
  }

  for (Hand hand : leap.getHands ()) {

    PVector handPosition       = hand.getPosition();
    float   handPinch          = hand.getPinchStrength();

    handX = int(handPosition.x);
    handY = int(handPosition.y);

    if (handPinch > .5) { 
      handPressed=true;
      image(okHandImg, handX, handY, width/10, height/10);
    } else {
      image(handImg, handX, handY, width/10, height/10);
      handPressed=false;
    }


    for (Finger finger : hand.getFingers()) {

      switch(finger.getType()) {
      case 0:
        break;
      case 1:
        break;
      case 2:
        break;
      case 3:
        break;
      case 4:
        break;
      }

      int     touchZone        = finger.getTouchZone();

      switch(touchZone) {
      case -1: 
        break;
      case 0: 
        break;
      case 1: 
        break;
      }
    }


    for (Tool tool : hand.getTools()) {

      int     touchZone        = tool.getTouchZone();


      switch(touchZone) {
      case -1:
        break;
      case 0: 
        break;
      case 1: 
        break;
      }
    }
  }
  if (end == true){
    image(endImg, width/2,height/2);
  }

  if (mouseChange == false) {
    image(startImg, width / 2, height / 2);
  }
  }

void handPressed() {
  if (handX >= puzzles[counter3].x && handX <= puzzles[counter3].x + width / 3 && handY >= puzzles[counter3].y && handY <= puzzles[counter3].y + height / 3) {
    
    handPressed = !handPressed;
  }
}


//#3 My puzzle - second code (the class "Puzzles"):

class Puzzle{
  
  int id;
  PImage img;
  boolean showImg = false;
  int x;
  int y;
  int rightX;
  int rightY;
   
  Puzzle(int _id, PImage _img, int _rightX, int _rightY){
    x = width / 2;
    y = height / 2;
    id = _id;
    img = _img;
    rightX = _rightX;
    rightY = _rightY;
  }
  
  void display(){
   if(showImg == true){
    image(img, x, y, imgW, imgH); 
   }
  }  
}

//#4 Arduino code:

#include <Servo.h>
Servo servo;
int reading;

void setup() {
  Serial.begin(9600);
  servo.attach(9);

}

void loop() {
while (Serial.available()) {
  reading = Serial.read();
} 
  if (reading == 1){
    servo.write(180);
  }else{
    servo.write(0);
  }

}

//#5 Final project's complete code:

import de.voidplus.leapmotion.*;
import processing.video.*;
import processing.serial.*;

Serial myPort;

Movie mv_opening;
Movie mv_ending;

PFont myFont;
int val;      
int preval;
int counter, counter1, counter2;
float curvesin, curveSin;
float curveaddition, curveAddition;
int timepause;
int timepause2;
boolean instruction2 = false;
boolean backgroundloop = false;


int x, y;

PImage MagicPatternImg;
PImage FrankInstruction;
boolean show1 = false;
boolean show2 = false;
boolean show3 = false;
boolean show4 = false;
boolean show5 = false;
boolean show6 = false;
boolean show7 = false;
boolean show8 = false;
boolean show9 = false;
boolean show10 = false;
boolean show11 = false;

color col_01 = color(180, 150);
color col_02 = color(180, 150);
color col_03 = color(180, 150);
color col_04 = color(180, 150);
color col_05 = color(180, 150);

LeapMotion leap;
int handX, handY, handZ;

Puzzle[] puzzles = new Puzzle[9];

boolean handPressed=false;
boolean mouseChange = false;
boolean puzzleChange = false;
boolean end = false;
boolean videoEnd = false;
boolean playStar = false;
boolean openVideo = true;

PImage handImg;
PImage okHandImg;
PImage startImg;
PImage endImg;

int imgW;
int imgH;

int off = 50;
int counter3 = 0;

void setup() {
  printArray(Serial.list());
  String portName = Serial.list()[1];
  myPort = new Serial(this, portName, 9600);

  background(0);
  smooth();// Draws all geometry with smooth (anti-aliased) edges.
  size(600, 600);
  counter = 0;
  counter1 = 0;
  counter2 = 40;
  curvesin = 0; 
  curveSin=0;
  curveaddition = 0.1;
  curveAddition=0.1;
  timepause=0;
  leap = new LeapMotion(this); 
  myFont = createFont("AppleM yungjo", 50);

  //leslie
  handImg = loadImage("hand_white.png");
  okHandImg = loadImage("ok hand_white.png");
  startImg = loadImage("startImg.jpg");
  endImg = loadImage("endImg.jpg");
  FrankInstruction = loadImage("FrankInstruction.jpg");

  imgW = width/3;
  imgH = height/3;

  puzzles[0] = new Puzzle(0, loadImage("img1.png"), imgW / 2, imgH /2);
  puzzles[1] = new Puzzle(1, loadImage("img2.png"), 3 * imgW / 2, imgH / 2);
  puzzles[2] = new Puzzle(2, loadImage("img3.png"), 5 * imgW / 2, imgH / 2);
  puzzles[3] = new Puzzle(3, loadImage("img4.png"), imgW / 2, 3 * imgH / 2);
  puzzles[4] = new Puzzle(4, loadImage("img5.png"), 3 * imgW / 2, 3 * imgH / 2);
  puzzles[5] = new Puzzle(5, loadImage("img6.png"), 5 * imgW / 2, 3 * imgH / 2);
  puzzles[6] = new Puzzle(6, loadImage("img7.png"), imgW / 2, 5 * imgH / 2);
  puzzles[7] = new Puzzle(7, loadImage("img8.png"), 3 * imgW / 2, 5 * imgH / 2);
  puzzles[8] = new Puzzle(8, loadImage("img9.png"), 5 * imgW / 2, 5 * imgH / 2);
  puzzles[0].showImg = true;

  textFont(myFont);
  textAlign(CENTER, CENTER);
  imageMode(CENTER);
  MagicPatternImg = loadImage("magic pattern.png");
  mv_opening = new Movie(this, "Open Video.mp4");
  mv_ending = new Movie(this, "End Video.mp4");
  //mv_opening.loop();
}

void leapOnInit() {
}
void leapOnConnect() {
}
void leapOnFrame() {
}
void leapOnDisconnect() {
}
void leapOnExit() {
}

void draw() {


  if (videoEnd == true) {
    image(startImg, width/2, height/2);
  }

  if (playStar == false) {
    background(255);
    if (puzzleChange == true) {
  
      puzzles[0].showImg = true;
      puzzles[6].showImg = true;
      puzzles[8].showImg = true;
      puzzles[6].x = puzzles[6].rightX;
      puzzles[6].y = puzzles[6].rightY;
      puzzles[8].x = puzzles[8].rightX;
      puzzles[8].y = puzzles[8].rightY;
    }
    if (mousePressed == true) {
      puzzleChange = true;
      mouseChange = true;
    };
    if (handPressed == false && puzzles[7].x == puzzles[7].rightX && puzzles[7].y == puzzles[7].rightY ) {
      end = true;
      //print("/////////");
    }
    if (handPressed == true) {
      puzzles[counter3].x = handX;
      puzzles[counter3].y = handY;
    }

    for (int i=0; i<puzzles.length; i++) {
      puzzles[i].display();
    }

    if (handPressed == false && puzzles[counter3].x >= puzzles[counter3].rightX - off && puzzles[counter3].x <= puzzles[counter3].rightX + off && puzzles[counter3].y >= puzzles[counter3].rightY - off && puzzles[counter3].y <= puzzles[counter3].rightY + off) {
      puzzles[counter3].x = puzzles[counter3].rightX;
      puzzles[counter3].y = puzzles[counter3].rightY;

      if (counter3 < puzzles.length - 1) {
        counter3++;
      }
      puzzles[counter3].showImg = true;
    }

    for (Hand hand : leap.getHands ()) {

      PVector handPosition       = hand.getPosition();
      float   handPinch          = hand.getPinchStrength();

      handX = int(handPosition.x);
      handY = int(handPosition.y);

      if (handPinch > .5) { 
        handPressed=true;
        image(okHandImg, handX, handY, width/10, height/10);
      } else {
        image(handImg, handX, handY, width/10, height/10);
        handPressed=false;
      }


      for (Finger finger : hand.getFingers()) {

        switch(finger.getType()) {
        case 0:
          break;
        case 1:
          break;
        case 2:
          break;
        case 3:
          break;
        case 4:
          break;
        }

        int     touchZone        = finger.getTouchZone();

        switch(touchZone) {
        case -1: 
          break;
        case 0: 
          break;
        case 1: 
          break;
        }
      }

      for (Tool tool : hand.getTools()) {

        int     touchZone        = tool.getTouchZone();

        switch(touchZone) {
        case -1:
          break;
        case 0: 
          break;
        case 1: 
          break;
        }
      }
    }

    if (end == true) {
      image(endImg, width/2, height/2);
      if (mousePressed==true) {
        background(0);
        playStar = true;
        show10 = true;
      }
    }

    if (mouseChange == false) {
      image(startImg, width / 2, height / 2);
    }

    if (openVideo == true) {
      background(0);
      mv_opening.play();
      if (mv_opening.time() < 24.98) {

        image(mv_opening, width/2, height/2);
      } else {
        openVideo = false;
      }
    }
  } else { 

/////////////////////////////////////////////////////lesie code end
/////////////////////////////////////////////////////////////////lesie code end

    leapMotionStuff();
    checkposition();
    //background(0);
    smooth();
    if (show10) {
      if (instruction2 == false){
      
      image(FrankInstruction, width/2, height/2);
      }
      
      if (mousePressed == true) {
        
        if(backgroundloop == false){
        
        background(0);
        backgroundloop = true;
      }
      
        show11= true; 
        instruction2 = true;
      } 
    }

    if (show11) {
      pattern1();  
      //show10b=false;
      println("show11");
    }
  }
  if (show1&&show2&&show3&&show4) {
    //
    timepause  ++;
  }
  if (show1&&show2&&show3&&show4 && timepause > 200) {
    //println(timepause);
    //mode 2
    background(0);
    pushMatrix();
    translate(x, y);
    rotate(frameCount* 0.02);
    image(MagicPatternImg, 0, 0, width, height);
    popMatrix();


    fill(col_01);
    ellipse(300, 100, 18, 18);

    fill(col_02);
    ellipse(430, 500, 18, 18);

    fill(col_03);
    ellipse(90, 252, 18, 18);

    fill(col_04);
    ellipse(510, 252, 18, 18);

    fill(col_05);
    ellipse(170, 500, 18, 18);

    print ("100,100 is", dist(x, y, 100, 100));
    println ("500,500 is", dist(x, y, 500, 500));
    println ("500,100 is", dist(x, y, 100, 500));
    println ("100,500 is", dist(x, y, 500, 100));

    if (dist(x, y, 300, 100)<100) {
      col_01 = color(0, 100, 255);
      show5 = true;
    }
    if (dist(x, y, 430, 500)<100 && show5 == true ) {
      col_02 = color(0, 100, 255);
      show6 = true;
    }
    if ( show6 == true && show5 == true ) {
      stroke(0, 100, 255);
      line(300, 100, 430, 500);
    }
    if (dist(x, y, 90, 252)<100 && show6 == true) {
      col_03 = color(0, 100, 255);
      show7 = true;
    }
    if ( show6 == true && show7 == true ) {
      stroke(0, 100, 255);
      line(430, 500, 90, 252);
    }
    if (dist(x, y, 510, 252)<100 && show7 == true) {
      col_04 = color(0, 100, 255);
      show8 = true;
    }
    if ( show8 == true && show7 == true ) {
      stroke(0, 100, 255);
      line(90, 252, 510, 252);
    }
    if (dist(x, y, 170, 500)<100 && show7 == true) {
      col_05 = color(0, 100, 255);
      show9 = true;
    } 
    if ( show8 == true && show9 == true ) {
      stroke(0, 100, 255);
      line(510, 252, 170, 500);
      line(170, 500, 300, 100);
      timepause2 ++;
    }
    if (show1&&show2&&show3&&show4&&show5&&show6&&show7&&show8&&show9 && timepause2 > 50) {
      image(mv_ending, 300, 300, width, height);
      mv_ending.play();
      myPort.write(1);
    } else {
      myPort.write(0);
    }
  }
}



void handPressed() {
  if (handX >= puzzles[counter3].x && handX <= puzzles[counter3].x + width / 3 && handY >= puzzles[counter3].y && handY <= puzzles[counter3].y + height / 3) {

    handPressed = !handPressed;
  }
}



void leapMotionStuff() {
  int fps = leap.getFrameRate();

  for (Hand hand : leap.getHands ()) {
    //background(255);
    //hand.draw();
    PVector handPosition = hand.getPosition();
    x = int(handPosition.x);
    y = int(handPosition.y);
    handZ = int(handPosition.z);
  }
}

void checkposition() {

  //println(x);
  //println(y);
  println("x,y is", x, y);
  if (x <= 200 && y <=250 ) {
    show1=true;
    // println("1 is true");
  } else if (x >= 350 && y <=200) {
    show2=true;
    // println("2 is true");
  } else if (x <= 200 && y >=350) {
    show3=true;
    //println("3 is true");
  } else if (x >= 350 && y >=350) {
    show4=true;
    //println("4 is true");
  }
}


void pattern1() {

  pushStyle();
  colorMode(RGB);
  counter+=30; // increase counter by 30 every time, 
  noFill(); // make sure the pattern doesn't fill in 

  stroke(255, 100, 20);

  pushMatrix();
  translate(width/2, height/2);
  rotate(radians(counter2));
  counter2+=140; // make sure it has different rotation speed with others
  stroke(255, 100, 20);
  pushMatrix();//push another matrix
  scale(1.05);// increase the scale by 1.05
  translate(100, 20);//translate (100,20) to (0,0)
  rotate(radians(110));//translate staring at (0,0) to 110 degrees
  arc(0, 0, 28, 20, PI, PI+HALF_PI);
  arc(28, -20, 28, 20, HALF_PI, HALF_PI+HALF_PI);
  arc(28, 0, 28, 20, PI+HALF_PI, PI+HALF_PI+HALF_PI);
  arc(0, -20, 28, 20, 0, HALF_PI);
  popMatrix();
  popMatrix();

  // the larger flower circle inside
  // pushMatrix inside pushMatrix
  pushMatrix();
  translate(width/2, height/2);
  rotate(radians(counter+10));
  stroke(255, 100, 20);
  pushMatrix();//push another matrix
  scale(1.2);
  translate(100, 20);//translate (100,20) to (0,0)
  rotate(radians(110));//translate staring at (0,0) to 110 degrees
  arc(0, 0, 28, 20, PI, PI+HALF_PI);
  arc(28, -20, 28, 20, HALF_PI, HALF_PI+HALF_PI);
  arc(28, 0, 28, 20, PI+HALF_PI, PI+HALF_PI+HALF_PI);
  arc(0, -20, 28, 20, 0, HALF_PI);
  popMatrix();
  popMatrix();

  // the blue rotating circle inside
  pushMatrix(); //
  translate(width/2, height/2);
  rotate(radians(frameCount));
  stroke(0, 100, 255);
  point(0, 70);
  popMatrix();

  // the flower circle outside
  // pushMatrix inside pushMatrix
  pushMatrix();
  translate(width/2, height/2);
  rotate(radians(counter+10));
  stroke(255, 100, 20);
  pushMatrix();// push another matrix
  scale(1.92); // increase the scale by 1.92
  translate(100, 20);//translate (100,20) to (0,0)
  rotate(radians(110));//translate staring at (0,0) to 110 degrees
  arc(0, 0, 28, 20, PI, PI+HALF_PI);
  arc(28, -20, 28, 20, HALF_PI, HALF_PI+HALF_PI);
  arc(28, 0, 28, 20, PI+HALF_PI, PI+HALF_PI+HALF_PI);
  arc(0, -20, 28, 20, 0, HALF_PI);
  popMatrix();
  popMatrix();

  if (show1 == true) {
    blueDynamicFlower();
  }

  popStyle();
  // this one is to make the blue dynamic flower stop generating and make it to form a blue circle
  // so that the blue dynamic flower will not be having so many “petals”

  if (show2 == true) {
    spinning_rectangle();
  }

  if (show3 == true) {
    the_circle();
  }
  if (show4 == true) {
    Dodecagram();
  }
}


void blueDynamicFlower() {

  // the blue dynamic flower part
  pushMatrix(); 
  translate(width/2, height/2);
  rotate(radians(frameCount));
  noFill();
  stroke(0, 100, 255);
  point(0, 75+(sin(curvesin)*20));
  //point(x,y); float: x/y-coordinate of the point
  //if it is just point(0,75); then it will just be a point rotate in a circle, which I used above
  //add sin(curvesin) in the y-coordinate, 
  //thus it will give people the illusion of the sin curve 
  curvesin += curveaddition;

  popMatrix();
}

void spinning_rectangle() {
  pushMatrix(); 
  translate(width/2, height/2);
  counter1 += 8;
  rotate(radians(counter1));
  noFill();
  stroke(255, 100, 20);
  rect(0, 10, 20, 20);
  popMatrix();
  pushMatrix(); 
  translate(width/2, height/2);// translate the triangle to the center of the picture
  noStroke();
  rotate(radians(counter*0.75));// make the triangle rotate 
  noFill();
  stroke(255, 100, 20);
  triangle(30, 10, 20, 30, 40, 30);
  popMatrix();
}

void the_circle() {
  pushMatrix(); 
  //pushMatrix() and popMatrix() are used in conjunction with the other 
  //transformation functions and may be embedded to control the scope 
  //of the transformations.

  fill(255, 255, 255, 20);
  // (255,255,255) shows the white color, 20 shows the transparency.

  translate(width/2, height/2);
  //translate it in the middle of the screen so it can rotate well

  rotate(radians(counter+40));
  //rotate in the radius of counter+40, while counter = counter + 30.
  // if counter exceeds 360, for example, 390, it equals 30)

  text("γ", 30, 152); 
  //30, 54 are x,y coordinates
  //text imported from "AppleM yungjo"

  rotate(radians(counter*0.75));
  popMatrix();
}

void Dodecagram() {
  noFill();
  stroke(255, 100, 20);
  triangle(127, 200, 473, 200, 300, 500);
  triangle(127, 400, 473, 400, 300, 100);
  triangle(200, 132, 200, 472, 500, 308);
  triangle(100, 300, 400, 473, 400, 124);


  fill(0, 2); 
  // 0 is the rgb-color variable
  // 2 is the alpha-opacity of the fill
  noStroke(); // Disables drawing the stroke (outline). 
  rect(0, 0, width, height);
}

void movieEvent(Movie m) {
  m.read();
}

Week 12 – Audio and Image Controller

Author: Wanchen Zhao (Leslie)

Instructor: Antonius

Partner: Yufan Wang (Esme)

Lab section: 1:15 ~ 2:30 pm

Materials: Arduino, solderless breadboard, wires, potentiometer

WechatIMG5

This is the last week of Interaction Lab recitation, and we’re making a image and audio controller. We thought about using a potentiometer to control a comic consisting of four scenes and a piece of music to go with the mood of the comic, and how much the handle is turned determines which scene is shown and the music’s volume. These are the four scenes of this comic:

1

2

3

4

picture source: MomComic.com

The analog read of a potentiometer is within the range from 0 to 1023. We divided this range into four ranges when we wrote the Arduino code, and for each range the output would be a number from 0 to 3. The Arduino code is down below.

At first we had some problem when testing the Arduino code; the “Serial.println()” was referred to as the wrong object, so the output numbers were merely the real potentiometer’s numbers, which was somewhere between 0 to 1023.

Then we connected the Arduino with Processing, and imported the pictures and the audio. Here we realized a problem. Processing wouldn’t recognize the “0~1023” in our Arduino code; it can only recognize analog between 0 and 255. The range of 0 to 1023 in Arduino’s code have to be mapped into 0 to 255 for Processing to recognize. The second code is the Processing code.

And this is the final result of our controller:

 

Arduino code:

int potPin = 0;
int ledPin = 13;
int val = 0;

void setup() {
  Serial.begin(9600);
  pinMode(ledPin, INPUT);
}

void loop() {
  potPin = analogRead(A0);
  // Serial.println(potPin);

  if (potPin < 256) {
    Serial.write(0);
  } else if (potPin < 512) {
    Serial.write(1);
  } else if (potPin < 768) {
    Serial.write(2);
  } else if (potPin < 1024) {
    Serial.write(3);
  }
  delay(300);
}



Processing code:

PImage photo1;
PImage photo2;
PImage photo3;
PImage photo4;
import processing.sound.*;
import processing.serial.*;
Serial myPort;
int valueFromArduino;
int val;
SoundFile soundfile;
void setup() {
  String portName = Serial.list()[2];
  myPort = new Serial(this, portName, 9600);
  printArray(Serial.list());
  size(350, 310);
  photo1 = loadImage("1.jpg");
  photo2 = loadImage("2.jpg");
  photo3 = loadImage("3.jpg");
  photo4 = loadImage("4.jpg");
  soundfile = new SoundFile(this, "Redbone.mp3");
  //soundfile.play();
  soundfile.loop();
}

void draw() {
  if (myPort.available()>0) {
    val = myPort.read();
  
  if (val == 0) {
    image(photo1, 0, 0);
  }
  if (val == 1) {
    image(photo2, 0, 0);
  }
  if (val == 2){
    image(photo3, 0, 0);
  }
  if (val == 3){
    image(photo4, 0, 0);
  }
  
  soundfile.amp(map(val, 0, 3, 1.0, 0.0));
  }
}

void keyPressed() {
  if (soundfile.isPlaying() > 0) {
    soundfile.stop();
  } else {
    soundfile.play();
  }
}

Lab 11: The Drawing Machine

Author: Wanchen Zhao (Leslie)

Instructor: Antonius

Partner: Yufan Wang (Esme)

Lab section: 1:15~2:30 pm

Materials: Arduino, solderless breadboard, stepper motor, SH75440NE ic chip, 5V power supply, potentiometer, wires, laser cut mechanisms, blue pen

This week’s recitation we’re making an automatic drawing machine. This device’s principle is that it’s powered by a stepper motor, which controls a rotating wheel connected to laser cut mechanisms, which in turn has a pen on the other side. We first set up the whole device based on the instruction:

IMG_8724

Next we uploaded the code from online library into Esme’s Arduino, powered the Arduino and now we had enabled the machine to draw automatically. The pen was moving like crazy at first, but we adjusted the arms of the mechanisms before it was able to limit the drawing range on one A4 paper. This is the machine drawing on paper:

The next step was to combine ours with that of the group sitting next to us, to let two arms hold one pen:

IMG_8729

And this is the newly set up device drawing on paper:

Next up we inserted potentiometers into our devices, which allow us to control the speed of the arms through adjusting the switch on the potentiometers:

And finally, these are their works, which we have displayed on the lab’s wall!

IMG_8737

IMG_8738

IMG_8743

 

 

Final Project Blog Post

Author: Wanchen Zhao (Leslie)

Instructor: Antonius

My Definition of Interaction

From my understanding, “interaction” in interactive media is the communication between human beings or the environment and a certain range of devices. These devices and gadgets are produced from various technologies and tools, including Processing, Arduino, 3D printing, laser cutting, etc. Such devices contain sensors that sense different variables, and are supposed to respond accordingly when these variables change or are in certain ranges. It’s people or the environment that can make these variables be in different conditions that the devices can sense. The “interaction” refers to the process of devices responding to the variables accordingly.

In addition, there should always be an adequate reason that a user would desire to use the device. The reason can be to facilitate everyday life for either everybody or a community (for example, people with certain disabilities), to provide entertainment, or to provide a unique user experience.

Final Project Proposal

For our final project, we’re thinking about a user experience designed for marketing purpose of the coming Dr. Strange sequel, where Dr. Strange meets Thor. It will be based on the magic gloves we created for our midterm project, but this time we will lose the gloves, and combine our midterm patterns with a device newly introduced to us called Leap Motion. We’re thinking about turning the pattern I did into a puzzle, because it’s not symmetric in any ways, and all the shapes making up the pattern is in one layer. Regarding Frank’s pattern, since that one has different layers and is largely symmetric, turning it into a puzzle would be over complicated; plus, a puzzle of a highly symmetric pattern would generate many visually correct solutions, yet only one solution could be inserted into Processing, so it wouldn’t make sense. Therefore, we’ll use Frank’s pattern as more of a user experience than a game, and the user would have to use the Leap Motion complete that pattern, hence to make all layers of the pattern appear, before the physical door opens and a virtual invitation to the sequel’s premiere appears.

Critique of Established Form of Interaction

The most important form of established interaction for our final project is Leap Motion. It requires attaching a small USB, called Leap Motion Controller, to a computer with the corresponding app. By using 3 infrared LEDs and 2 monochromatic IR cameras, the devices senses a roughly hemispherical area, and the furthest distance is around 1 meter. It recognizes human hands, both palms and all the fingers, as input, and enables hands to move objects around on the screen without touching the controller.

From my personal perspective, the pro of Leap Motion is that it facilitates, or even enables the creation of relative user experience; the con is that most users would be first-timers, so they usually wouldn’t be familiar with how large a range the controller can sense, so it might not be too accurate.

The way Leap Motion relates to our final project is that we would like to move puzzle pieces around, and complete the latter pattern on either a screen or a projector using this device. We want to create a user experience that resembles Kinect.

Week 10: Experimenting with Tinkercad

Author: Wanchen Zhao (Leslie)

Instructor: Antonius

Lab section: 1:15~2:30 pm

Today we’re experimenting with Tinkercad. I chose to experiment with a Thumb Joystick, and create a game controller on Tinkercad. This is how the Joystick loos like:

Screen Shot 2017-04-21 at 4.08.29 PM

I thought it was possible to make a hole of the exact shape of the Joystick inside the game controller, and I did so. First I made the shell of the controller with one box and 2 round roofs standing up. The width and length of the box is the same as that of the Joystick’s bottom, while the height equals to the 12.1 mm, which is the height of the Joystick’s electric components combined.

Screen Shot 2017-04-21 at 1.53.24 PM

Next I created the bottom of the Joystick, which is 1.6 mm tall (1.6 mm = 7.3 mm + 31 mm – 36.7 mm). The tiny semicircles at the four sides’ diameter is 4.2 mm. The two on the shorter sides of the box are in the middle, while the four on the longer sides are each 10 mm towards the sides. Three are solid shapes while three others are set to “holes”.

Screen Shot 2017-04-21 at 2.03.36 PM

Next I created the cylinder to fit in the top of the electric components of the Joystick. This part is 5.7 mm tall (5.7 mm = 7.3 mm – 1.6 mm), and needs to be lifted 1.6 mm above the plane. The diameter equals to the width of the box, which is 20 mm. Finally it’s combined with the bottom.

Screen Shot 2017-04-21 at 2.07.09 PM

Then I realized that before I transfer these parts into a hole and insert them into the shell of the controller, I would also need to create a space inside the controller to fit in the wires and other small particles on the left side of the cylinder. Creating a box of the same height as the cylinder would do it.

Finally I transferred these parts into a “hole”, and inserted it into the shell of the controller. In order to make the hole visible, I set the controller’s shell to “transparent”.

Screen Shot 2017-04-21 at 2.14.00 PM

Screen Shot 2017-04-21 at 2.19.21 PM

I find Tinkercad to be a relatively easy-to-handle tool among the tools we’ve come across so far in the course. Since we’re thinking about combing 3D printing into our final project, this lab is very useful.

DIY Rubber Stamp on Laser Cutter

Author: Wanchen Zhao (Leslie)

Instructor: Antonius

Materials: Adobe Illustrator

This week, since we’re learning about digital fabrication and laser cutting, the task is to recreate a stamp pattern that can be applied to laser cutting, following a tutorial on Vimeo. First of all, this is the pattern of a clover I drew:

IMG_8605

After some basic setups, the first step is to create the two concentric circles in the pattern. The button on the top left menu of creating patterns can be changed from rectangles to ellipses, and when drawing the ellipses, hold the “shift” key to draw circles. The stroke need to be changed to 20, and the “transform” tab on the top right corner enables adjusting the circles I’ve drawn to be roughly of the same sizes as the circles in the png file.

IMG_8607

The next thing to do is to create the texts going around inside the two circles. Following the tutorial, I drew one more concentric circle, this one smaller and inside the texts. I then changed the “Texts” function to “Texts on path”, and used the inner circle as path to line up my texts. This path also enabled me to change the location of my texts along the circle.

After writing down “NYUSH IMA” (I decided to add “IMA” at this point), I found it difficult to add my name as well on the same path. The method I figured was to create one more slightly larger circle, use it as the path for “LESLIE”, but this time inside the path.

IMG_8608

The added texts aren’t very clear. It’s my fault, I should’ve taken the picture after making the original drawing’s layer invisible. The texts are on a different layer from the two outside circles, called “texts”.

Next I drew the clover, this time on another layer “clover”. The way certainly wasn’t shown in the tutorial but it wasn’t hard. Simply draw a relatively long and narrow heart pattern with the Curve function, make the fill black, then copy it and paste three times. Last I rotated them by 90, 180 and 270 degrees respectively, and moved everything to the middle with all the heart tips pointing towards the circles’ center. Then I drew an arc with a heavy stroke (around 7 pt) as the stem.

Before it’s done, it’s also necessary to change all the texts into patterns by using the “Create outline” function. Else the laser cutter won’t recognize the texts.

IMG_8610

It seems to be done, but it’s not. If it’s put this way, the laser cutter will cut out the patterns, which is exactly the opposite of what I would want. Therefore, following the tutorial, I started a new layer which I called “edge”, where I drew a circle slightly bigger than the pattern and put it on the back of the pattern. Then I clicked on the first icon to the right in “pathfinder”.

Finally, I created a circle with no fill, the same size as the outside circle I just created on “edge”; this circle has to have the skinniest stroke of 0.1 pt, otherwise the cutter won’t cut.

It’s also important to keep in mind that the stamp would be a reflection of the pattern when it’s actually cut out. Therefore, to get what I want, I would have to select everything, and transform them into the vertical reflection of the way they were now.

IMG_8612

Before “reflect”

IMG_8613

After “reflect”

Now I’m done, and I’m ready to laser cut my personalized stamp!

 

Interaction Lab Midterm Project: Magic Glove

Author: Wanchen Zhao (Leslie)
Instructor: Antonius
Project: Magic Glove
Partner: Frank
Materials: solderless breadboard, Arduino, wires, 10k resistors, flex sensors, FSR sensors, glove, Processing

Inspiration & Description
We got our inspiration from the Marvel movie Dr.Strange, where Dr. Stephen Strange releases different patterns combining basic geometric patterns and mysterious texts made up of Greek letters. We decided to recreate this effect using Arduino and Processing. Since we can’t do Dr.Strange’s tempting magic, we hide the trigger to the magic into a glove, that can generate likewise patterns that we drew with Processing code from scratch with a simple movement.

The Process
Drawing with Processing

The visual effect is the most time-consuming and most challenging part of our project. We each drew one pattern with Processing. I started off with the outline structure, or the basic patterns of ellipses and rectangles. To better the effect, I proceeded to adding Greek letters into spaces between the outer circles. The Greek letters come from the internet, all I had to do was copying and pasting them into the “text()” code. But a very time-consuming process was to find a font that supports Greek letters. Luis introduced a website called dafont.com to me, where I could download free fonts, put them into my code’s folder and run it. The function installed in Processing is “PFont myFont” in set up, “myFont = createFont(“AppleMyungjo,50))”, where “AppleMyungjo” is the font and 50 is the size. Not a lot of fonts support Greek letters, and at first for quite a few times I ended up with the “α”s and “β”s unrecognized. I realized it would be so much easier to use English letters, but since we were inspired by Dr.Strange, it was best to make the replication as close as possible. And in order to make the letters go around the circle, I also used the “rotate(radians(counter))” function.
This is how my final animated effect looks like:


The change of color is achieved by a special function in “stroke()” or “fill()”, called “map()”. By inserting “frameCount()” into the parenthesis, Processing enables the pattern to change colors and brightness gradually, giving the pattern a glowing effect.

Serial Communication

Whenever the flex sensor is pressed, the screen generates a pattern. We’re going to hide the two flex sensors in two gloves, so that they each controls different patterns. This is how the flex sensors and resistors are connected to the breadboard:
IMG_7778

WechatIMG226
Before enabling Arduino to talk to Processing, this is to show that the flex sensor is able to send signals when pressed:


Below is the effect of connecting the Arduino with Processing. The way the code works is to print “1” when one flex sensor senses the pressure, and print “2” when another senses the pressure. When both are pressed or neither are pressed, it prints out “0”. Afterwards all we had to do is to use Serial communication, and assign “1” and “2” to the two patterns.

A Sensor Change

However, turns out at the end of the day, our biggest obstacle was not drawing the patterns with Processing; it was the sensor. The flex sensors refused to work anymore right at the night before the presentation, so we had to switch sensors at the last minute. I thought about my Stupid Pet Trick project, where I used FSR to replicate a smart pillow that can sense a person’s head on it. I recalled that FSR is much more sensitive than flex sensors, so we borrowed the last two left in the equipment room. Since I was familiar with FSR because of my Stupid Pet Trick, I did the physical installation and the Arduino code, without any staff help. Fortunately, the Processing code requires almost no change besides the set up.

Other Challenges

Actually, both of the FSR sensors didn’t come in handy at first as well. One had broken wires and need to be re-soldered; the other was the one I used for my Stupid Pet Trick, which had no wires. For the latter we tried using clips along with M/M wires, but the wires fell off easily and the wires still weren’t long enough to be attached to the gloves. Our final solution was to re-solder both. Jiwon helped us with the soldering. It took a while but it was worth it, because our wires never fell off from the breadboard again afterwards.

IMG_7788

IMG_7787

When we ran it afterwards, there was something very weird with my code: no matter how I rewrote the code, there was always an extra set of Greek letters at the bottom right corner of my circle; what’s more, it was always green, refusing to change color and brightness along with the rest of the pattern. When everything else was done, during the debugging process, Jiwon helped us check it and it turns out the problem came from a repetition of the “textAlign”.

After attaching the FSRs to the gloves, we made a edited video, sound effect and subtitles attached:

The Final Versions of the Codes

 

At the source code section, both the Processing code and the Arduino code can be found.

Gains from the Project

Taking Interaction Lab is the first time I encounter computer programming, and combining it with physical interaction. Even though during the first half of the semester we practiced every Friday during lab, it’s not constant enough for me to truly deepen my knowledge and understanding of Processing and Arduino. For midterm project, since we’re drawing the Dr.Strange patterns completely with Processing, I can say I have truly obtained the basic Processing drawing and animation skills we have discussed so far in class. What’s more, with digging into the Processing tutorials and the kind help from our fellows, I have also explored more functions that have polished my drawing, such as RGB, PFont, and text(). I don’t major in IMA, and before taking Interaction Lab I have always thought I can never be able do anything cool on computer. It’s fascinating to discover that I actually have potential in this field. What’s more, I also familiarized myself with Arduino’s sensor installation and Serial communication,enabling myself to transfer signals between Arduino and Processing.

Second, as a student of finance major, most of the academics I deal with have rules and fixed answers, so it’s refreshing and inspiring to encounter IMA, where creativity is just as important as rules and knowledges. I used to think coding was fixed and follows strict patterns, but Interaction Lab has made me realize that how a person writes code depends a lot on his or her own way of thinking.

The skills I mentioned above, learned and comprehended from Interaction Lab, play an important role in this world with emerging technology and innovation. Even though I didn’t choose this path, it’s always benefitting to have some knowledge about it.

This midterm project has also taught me that how close I am to perfection depends on how much effort I’m willing to put into what I do. We spent a lot of time on the 8th floor during the days before the presentation, staying there everyday until the lab closed. Even though it was tiring and time-consuming spending much time on the tiniest details, I realize that the more careful we are, the more likely our project won’t have problems during presentation. And sometimes if not paid attention to, details can have a huge impact on the overall effect. A good example would be the extra, single-color line of Greek letters that originally existed in my pattern.

Looking back, I think I cherish this process a lot. Not only did I put what I learned in the first half of the semester into practice, I also learned so much more about Processing in accordance to the project. I’m proud of myself for going through this. The midterm project has truly been a valuable experience.

//processing
import processing.serial.*;
Serial myPort; // Create object from Serial class
int val; // Data received from the serial port

int preval;

int counter, counter1, counter2;
float curvesin, curveSin;
float curveaddition, curveAddition;
boolean drawMagic=false;
int x, y;
PFont myFont;
// boolean makes drawmagic function possible to be controlled by key

int step = 15;
void setup() {

printArray(Serial.list()); // In this case, I can find the port number
String portName = Serial.list()[1];
myPort = new Serial(this, portName, 9600);

background(0);
smooth();// Draws all geometry with smooth (anti-aliased) edges.
size(600, 600);
counter = 0;
counter1 = 0;
counter2 = 40;
curvesin = 0;
curveSin=0;
curveaddition = 0.1;
curveAddition=0.1;//

// import font "AppleM yungjo" into the system
myFont = createFont("AppleM yungjo", 50);
textFont(myFont);
textAlign(CENTER, CENTER);
}

void draw() {

if ( myPort.available() > 0) { // If data is available,
val = myPort.read(); // read it and store it in val
println(val);

if (preval!=val) {
background(0);
}

if (val==1) {
pattern1();
}
if (val==2) {
pattern2();
}
}
preval=val;
}

void pattern1() {

pushStyle();
colorMode(RGB);
counter+=30; // increase 30 everytime,

noFill();
stroke(255, 100, 20);

//pushMatrix() and popMatrix() are used in conjuction with the other
//transformation functions and may be embedded to control the scope
//of the transformations.
// the γ cirlcle
pushMatrix();
fill(255, 255, 255, 20);
translate(width/2, height/2);
rotate(radians(counter+40));
text("γ", 30, 152); // x,y coordinates
rotate(radians(counter*0.75));
popMatrix();

// the stars consisted of trianangles
noFill();
stroke(255, 100, 20);
triangle(127, 200, 473, 200, 300, 500);
triangle(127, 400, 473, 400, 300, 100);
triangle(200, 132, 200, 472, 500, 308);
triangle(100, 300, 400, 473, 400, 124);

// call xy position to check the xy coordinates
//xyposition();
fill(0, 2);
// 0-rgb-color variable or hex value
//alpha-opacity of the fill
noStroke(); // Disables drawing the stroke (outline).
rect(0, 0, width, height);

// the servral circles
ellipse(width/2, height/2, 100, 100);
ellipse(width/2, height/2, 200, 200);
ellipse(width/2, height/2, 300, 300);
ellipse(width/2, height/2, 400, 400);

// the outside flower circle
// pushMatrix inside pushMatrix
pushMatrix();
translate(width/2, height/2);
rotate(radians(counter+10));
stroke(255, 100, 20);
pushMatrix();//push another matrix
scale(1.92); // increase the scale by 1.92
translate(100, 20);//translate (100,20) to (0,0)
rotate(radians(110));//tranlate staring at (0,0) to 100 degrees
arc(0, 0, 28, 20, PI, PI+HALF_PI);
arc(28, -20, 28, 20, HALF_PI, HALF_PI+HALF_PI);
arc(28, 0, 28, 20, PI+HALF_PI, PI+HALF_PI+HALF_PI);
arc(0, -20, 28, 20, 0, HALF_PI);
popMatrix();
popMatrix();

// the inside smallerflower circle
// pushMatrix inside pushMatrix
pushMatrix();
translate(width/2, height/2);
rotate(radians(counter2));
counter2+=140;
stroke(255, 100, 20);
pushMatrix();//push another matrix
scale(1.05);
translate(100, 20);//translate (100,20) to (0,0)
rotate(radians(110));//tranlate staring at (0,0) to 100 degrees
arc(0, 0, 28, 20, PI, PI+HALF_PI);
arc(28, -20, 28, 20, HALF_PI, HALF_PI+HALF_PI);
arc(28, 0, 28, 20, PI+HALF_PI, PI+HALF_PI+HALF_PI);
arc(0, -20, 28, 20, 0, HALF_PI);
popMatrix();
popMatrix();

// the inside larger flower circle
// pushMatrix inside pushMatrix
pushMatrix();
translate(width/2, height/2);
rotate(radians(counter+10));
stroke(255, 100, 20);
pushMatrix();//push another matrix
scale(1.2);
translate(100, 20);//translate (100,20) to (0,0)
rotate(radians(110));//tranlate staring at (0,0) to 100 degrees
arc(0, 0, 28, 20, PI, PI+HALF_PI);
arc(28, -20, 28, 20, HALF_PI, HALF_PI+HALF_PI);
arc(28, 0, 28, 20, PI+HALF_PI, PI+HALF_PI+HALF_PI);
arc(0, -20, 28, 20, 0, HALF_PI);
popMatrix();
popMatrix();

// blue roatating circle inside
pushMatrix(); //
translate(width/2, height/2);
rotate(radians(frameCount));
stroke(0, 100, 255);
point(0, 70);
popMatrix();

//spinning rectangle

pushMatrix();
translate(width/2, height/2);

counter1 += 8;

rotate(radians(counter1));

//rotate in the counter randians, counter increases 30 everytime
// if counter exceeds 360, for example, 390, it euqals 30)
noFill();
stroke(255, 100, 20);
rect(0, 10, 20, 20);
popMatrix();

pushMatrix();
//pushMatrix() and popMatrix() are used in conjuction with the other
//transformation functions and may be embedded to control the scope
//of the transformations.
translate(width/2, height/2);// translate the triangle to the center of the picture
noStroke();
rotate(radians(counter*0.75));// make the triangle rotate
noFill();
stroke(255, 100, 20);
triangle(30, 10, 20, 30, 40, 30);
popMatrix();

// the flower part
pushMatrix(); //
translate(width/2, height/2);
rotate(radians(frameCount));
noFill();
stroke(0, 100, 255);
point(0, 75+(sin(curvesin)*20));
// point(x,y); float: x/y-coordinate of the point
//if it is just point(0,75); then it will just be a point rotate in a circle
//add sin(curvesin) in the y-coordinate,
//thus it will give people the illusion of the sin curve
curvesin += curveaddition;

popMatrix();

popStyle();
if (mousePressed) {
curveaddition = 0;
}
}

//xyposition is created to show the xy coordiantes of each points at the top left screen
// when don't need it, just comment out xypositon(), so that it will disaapear
//thanks for chirstian
void xyposition () {
x = mouseX;
y = mouseY;
fill(0);
//the place for rectangle
rect(0, 0, 100, 100);
fill(240);
text(x, 45, 20);
text(y, 45, 70);
}

void pattern2() {
//counter = 0;
textSize(40);//the Greek letters going around the circle
//different letters occupy different locations
colorMode(HSB);
fill(map(frameCount % 255, 0, 255, 90, 255), 255, map(frameCount % 255, 0, 255, 90, 255));
pushMatrix();
translate(width/2, height/2);
rotate(radians(counter));
text("β", 0, 170);
text("γ", 30, 160);
text("δ", 60, 160);
text("η", 90, 140);
text("θ", 120, 130);
text("ξ", 140, 120);
popMatrix();

//pushMatrix();//pushMatrix() and popMatrix() are used in conjunction with the other
////transformation functions and may be embedded to control the scope of the transformations.
//translate(width/2, height/2);
//rotate(radians(counter));//rotate in the counter radians, counter increases 30 every time
////if counter exceeds 360, for example, 390, it equals 30
//text("β", 0, 200);
//popMatrix();

//pushMatrix();
//translate(width/2, height/2);
//rotate(radians(counter));
//text("γ", 30, 190);
//popMatrix();

//pushMatrix();
//translate(width/2, height/2);//translate the patterns to the center of the canvas
//rotate(radians(counter));
//text("δ", 60, 180);
//popMatrix();

//pushMatrix();
//translate(width/2, height/2);
//rotate(radians(counter));
//text("η", 90, 170);
//popMatrix();

//pushMatrix();
//translate(width/2, height/2);
//rotate(radians(counter));
//text("θ", 120, 160);
//popMatrix();

//pushMatrix();
//translate(width/2, height/2);
//rotate(radians(counter));
//text("ξ", 150, 150);
//popMatrix();

noFill();//the general shape of the patterns
ellipse(300, 300, 450, 450);
ellipse(300, 300, 425, 425);
ellipse(300, 300, 300, 300);
ellipse(300, 300, 121.132, 121.132);
ellipse(300, 300, 150, 150);
fill(0, 0, 0);
ellipse(300, 118.75, 62.5, 62.5);
ellipse(118.75, 300, 62.5, 62.5);
ellipse(300, 481.25, 62.5, 62.5);
ellipse(481.25, 300, 62.5, 62.5);
noFill();
ellipse(300, 118.75, 60, 60);
ellipse(118.75, 300, 60, 60);
ellipse(300, 481.25, 60, 60);
ellipse(481.25, 300, 60, 60);
noFill();
rect(193.935, 193.935, 212.132, 212.132);
quad(150, 300, 300, 150, 450, 300, 300, 450);
fill(255, 100, 20);

textFont(myFont);
textAlign(CENTER, CENTER);
textSize(70);

//stroke(map(frameCount % 255, 0, 255, 90, 255), 255, map(frameCount % 255, 0, 255, 90, 255));//the four letters occupying the four small circles
//textAlign(CENTER);
//textSize(70);

counter+=step; //the rotating, multiple rectangles in the middle
//println(counter);
pushMatrix();
translate(width/2, height/2);
rotate(radians(counter));
noFill();
stroke(255, 100, 20);
rect(0, 10, 50, 50);

popMatrix();

counter+=step;//increase 15 every time
//println(counter);
pushMatrix();
translate(width/2, height/2);
rotate(radians(counter));
noFill();
stroke(map(frameCount % 255, 0, 255, 90, 255), 255, map(frameCount % 255, 0, 255, 90, 255));
rect(5, 10, 50, 50);

popMatrix();

counter+=step;
//println(counter);
pushMatrix();
translate(width/2, height/2);
rotate(radians(counter));
noFill();
stroke(map(frameCount % 255, 0, 255, 90, 255), 255, map(frameCount % 255, 0, 255, 90, 255));
rect(10, 10, 50, 50);

popMatrix();

counter+=step;
//println(counter);
pushMatrix();
translate(width/2, height/2);
rotate(radians(counter));
noFill();
stroke(map(frameCount % 255, 0, 255, 90, 255), 255, map(frameCount % 255, 0, 255, 90, 255));
rect(15, 10, 50, 50);
curveSin+=curveAddition;
popMatrix();
if (mousePressed) {
curveAddition=0;
}
}

//arduino
void setup() {
Serial.begin(9600);
//1 // number "1" //striing '1'//char value

// Serial.write()
// sends one byte as a number

// Serial.print()
// sends multiple bytes as a char
}

void loop() {
int val1 = analogRead(A0);
int val2 = analogRead(A1);

// just to test!
// Serial.print(val1);
// Serial.print(",");
// Serial.print(val2);
// Serial.println();

if (val1 < 40) {
//Serial.println(1);
Serial.write(1);
} else if (val2 < 120) {
Serial.write(2);
} else {
Serial.write(0);
}

}

Week 6: Serial Communication

Author: Wanchen Zhao (Leslie)
Instructor: Antonius
Lab section: 1:15~2:30 pm
Partner: Esme

Materials: Arduino, solderless breadboard, switch, resistors (220 and 10k), wires, LEDs
Today’s task is to create interaction between Arduino and Processing. The interaction should be mutual, meaning that the Arduino would have to give order to processing, and vice versa. Thinking of the code of up and down buttons on the keyboard which we introduced in class the other day, I decided to connect these two keys with red and green LED lights. The Arduino wouldn’t directly understand “UP” and “DOWN”, so that’s where Processing came into place.
The Processing code would have to import “myPort”, which is a library with various sample codes available in our computers. The Processing then converts “up” and “down” keys on the keyboard into H and L, which stand for high and low, for the Arduino to understand.
This is the circuit of the two LEDs:
IMG_7676
The code on Processing 3.0:
import processing.serial.*;
Serial myPort;
int val;
void setup() {
printArray(Serial.list());
// this prints out the list of all
// available serial ports on your computer.
myPort = new Serial(this, Serial.list()[ 2 ], 9600);
// Change the Serial.list()[0] to your port
}
void draw() {
// to send a value to the Arduino
while (myPort.available() >0) {
val = myPort.read();
}
println(val);
if (keyCode == UP) {
myPort.write(‘H’);
} else if (keyCode == DOWN) {
myPort.write(‘L’);
}
}
And the code on Arduino, which recognizes letters H and L and converts them to the outputs– the 2 LED lights:
int val;
void setup() {
Serial.begin(9600);
pinMode(13, OUTPUT);
pinMode(12, OUTPUT);
}

void loop() {
// receive a value from processing
while (Serial.available()) {
val = Serial.read();
}

// test the value
if (val == ‘H’) {
digitalWrite(13, HIGH);
digitalWrite(12, LOW);
} else if (val == ‘L’) {
digitalWrite(13, LOW);
digitalWrite(12, HIGH);
}
}
And now when I press up and down keys on my keyboard, the red and green lights will respectively turn on, like traffic lights:


But this isn’t over yet, because it’s still necessary for the Arduino to make Processing do something. I thought of Processing sensing pressing a switch, and would reflect the press with a number 1 or 0. This would require a switch to be put in as an input, and ordering Processing to print out 1 or 0 when the switch is pressed and not pressed.
This is the modified Arduino code that enables such changes:
int val;
void setup() {
Serial.begin(9600);
pinMode(13, OUTPUT);
pinMode(12, OUTPUT);
pinMode(11, INPUT);
}

void loop() {
// receive a value from processing
while (Serial.available()) {
val = Serial.read();
}
if (digitalRead (11)==1) {
Serial.write(1);

}
else {
Serial. write (0);
}
// test the value
if (val == ‘H’) {
digitalWrite(13, HIGH);
digitalWrite(12, LOW);
} else if (val == ‘L’) {
digitalWrite(13, LOW);
digitalWrite(12, HIGH);
}
}
I originally wanted to let Processing print out stuff more complicated than 1 or 0 — “on” and “off”, after a failed attempt, I asked a fellow and was told that I would need to keep in mind to keep Processing’s printout as simple as possible, because Processing usually can only recognize single digits, and trying to print out something like “on””off” would be creating trouble for myself. Thus I decided to stick to numbers 1 and 0 in the end.
This is the final effect of Arduino giving Processing an order:


This lab helps me see the possible interaction between the two tools we’ve encountered so far, and makes me realize that Arduino and Processing both can do something that the other cannot, and thus can compensate for each other. It also helps me transfer my orders from Processing to Arduino, or the other way around, into something these devices can understand. I think it’s going to be very useful for the coming midterm project.

Week 5: Stupid Pet Trick

Arthor: Wanchen Zhao (Leslie)
Instructor: Antonius
Lab section: 1:15~2:30 pm

Stupid pet trick: Annoying Alarm Clock
Stupid pet trick materials: solderless breadboard, Arduino, FSR, wires, 100K resistor, buzzer, servo, bobby pin, elastic, doll.
I got the idea of this trick from times when people turn off their alarm clock, go back to sleep, and end up with a day’s schedule completely messed up. What if the alarm clock can sense whether your head is still on the pillow a few minutes after the alarm goes off, and won’t stop bothering you until you get up?
This trick would thus require three outputs: the alarm, represented by the buzzer; a pressure sensor, representing the pillow; and an arm operated by a servo, which is what makes the alarm clock annoying. After talking to Antonius about this idea, I learned about “millis” in processing, which allows me to measure time. I also got to know FSR, which senses the pressure; else, if I use a simple touch sensor which only senses pressure at a certain spot, the person could simply turn his head to other parts of the pillow where the sensor can’t sense and be soundly asleep. I borrowed the FSR from the lab, and connected it to my breadboard following online tutorial.
This is what the connection looks like. It already includes the servo, the buzzer and the FSR; the things missing are the doll representing the sleeping person, and the arm attached to the servo.
IMG_7605
The next step was writing the code. I want my project to work in this order:
1. The alarm clock goes off for 5 seconds;
2. It lets the person snooze for 5 more seconds;
3. The person doesn’t get up, and the arm starts slapping the person;
4. The person gets up, and the arm stops slapping.
Therefore, the code includes a lot of conditions, which would require the “if/ else” sentences. The FSR codes are provided in the online tutorial, so I inserted sentences I needed into my code. We have learned about the buzzer’s and servo’s codes before. However, I realized this had a lot of conditions and I made the process very hard for myself to code. Antonius told me that I would need to transfer certain sentences into orders for not the computer but for people. He helped me with my code; but I would need to be independent since this is my project, so I only used his code as reference, and wrote it again, this time with some changes I wanted, by myself.
This is the code:
int fsrPin = 0;
int fsrReading;

#include
Servo servo;

void setup() {
Serial.begin(9600);
pinMode(7, OUTPUT);
servo.attach(6);
// put your setup code here, to run once:
}

void loop() {
fsrReading = analogRead(fsrPin);
Serial.print(“FSR: “);
Serial.print(fsrReading); //ask what the FSR is reading.
Serial.print(” How long since Arduino was on: “);
Serial.println(millis()); //ask arduino how long it has been since it has been awake;

// if the FSR is high it means the person is still sleeping
if (fsrReading > 100) {

// if it’s between 5000 and 10000 make sound
if (millis() > 5000 and millis() 10000 and millis() 15000) { // if it’s above 15000 make no sound
//if it’s a above 15000 move servo
servo.write(60);
delay(100);
servo.write(0);
delay(100);
}

} else { // if the FSR is not being pressed then turn alarm off
noTone(7);
servo.write(0);
}
}
For the arm of the servo, since the doll I could find is small, I accordingly attached a bobby pin to the servo with a piece of elastic. And since the FSR is very sensitive, I used it to represent the pillow instead of plugging it into an actual pillow. If I had plugged it into an actual pillow, it would sense the pressure from the pillow even after the person gets up, and thus wouldn’t work properly.
This is the final effect of the alarm clock, shot right before the Friday show:


During the show, except a short period of time when the arm on the servo fell off, my alarm clock worked fine. I was especially proud that on the show, my project made a lot of people laugh.
My first IMA project is a great experience. Not only did I learn about several new Arduino functions, such as “millis” and FSR, I also managed to create a gadget that operates various sensors and works properly, based on a problem I observed from real life. I learned a lot and had a lot of fun during the process, and I think this is a great opportunity to lay the foundation for my bigger projects later in this course.

Week 4: Sensors

Author: Wanchen Zhao (Leslie)
Instructor: Antonius
Lab section: 1:15~2:30 pm
Materials: moisture sensor, wet sponge, solderless breadboard, wires, LED, resistor, buzzer, servo

Today we’re experimenting with sensors, in order to prepare for the stupid pet trick next week.
I chose the moisture sensor, which came in with an additional piece of wet sponge. And throughout the lab, I connected this moisture sensitivity with LED, servo and buzzer.IMG_7569
IMG_7571
IMG_7573
I started with the LED, and the plan was that it would light up if the sensor senses moisture. I was warned by Antonius that I would have to connect the LED with a 220R resistor or else it would burn down. The LED would act as an output, whose output number would have to be specified in the Arduino code. Even though the sensor is an analog, in this case it only has two modes: “HIGH” for the LED to light up, and “LOW” for otherwise.
With the help from Antonius, I familiarized myself with the “if, else” code.

void setup() {
Serial.begin(9600);
pinMode(7,OUTPUT);
// put your setup code here, to run once:

}

void loop() {
Serial.println(analogRead(2));
if(analogRead(2)>100){
digitalWrite(7,HIGH);

}else{
digitalWrite(7,LOW);
}
// put your main code here, to run repeatedly:

}

And here’s how it would work:

trim.2E9AFC94-BE4E-4A3E-9EC7-01F7401AEEE7

And I was also warned to use wires of appropriate colors, or else others would get confused.

Next I did the servo one on my own. With the code we had yesterday that would move the servo, I plugged it accordingly into the “if, else” code; the servo is still an output, but the code for it is different from LED.

#include <Servo.h>
Servo servo;
void setup() {
// put your setup code here, to run once:
Serial.begin(9600);
servo.attach(7);
}

void loop() {
Serial.println(analogRead(2));
if(analogRead(2)>100){
servo.write(60);
}else{
servo.write(0);
}
// put your main code here, to run repeatedly:
}

When the sensor senses the moisture (touches the wet sponge), the servo’s wing would rotate for 60 degrees; when the sensor doesn’t, the servo’s wing turns back to its starting position.

IMG_7572

The last one with the buzzer was a bit tricky, because the buzzer has a code different from those of LED and servo; what’s more, compared to the LED, the buzzer is truly more like an analog: the code needs to have an input of the noise’s frequency.

void setup() {
Serial.begin(9600);
pinMode(8,OUTPUT);
// put your setup code here, to run once:

}

void loop() {
Serial.println(analogRead(2));
if(analogRead(2)>100){
tone(8,1000);
}if(analogRead(2)<100){
noTone(8);
}
// put your main code here, to run repeatedly:

}

Here the input of the noise is 1000; that means it vibrates 1000 times every second.

IMG_7574

This lab session has given me more clue on how to work with my stupid pet trick in the coming week. Through practice, I now have a better understanding of how to connect sensor with output and make them work together. What’s more important is that I’ve learned how to control some basic outputs through coding, as well as the “if, else” scenario. I think this session may be a great inspiration for my stupid pet trick.