Final Project Mini Theater(Rudi)-Samantha&Silvia

Our final project is called Mini Theater. We first got the idea of creating a puppet that can mimic people’s action and movement from marionettes. But we feel that puppets like marionettes are usually not very easy to play with because we have to learn skills of how to control them properly. So we wonder if they could be controlled much easier by making them able to mimic what we do so that everyone can play with these “revised marionettes” easily. Also, we think that the normal toys which sing once we press their tummies are not very engaging in interaction. This is also why we’d like to create such a toy puppet. We were inspired by the pompom mirror that we can make a puppet that mirrors our movement. What’s more, we want the users to use this interactive installation to create their own little stories, which is why it got its name “Mini Theater”. Continue reading

Processing part:
import processing.serial.*;
import gab.opencv.*;
import processing.video.*;
import java.awt.Rectangle;

int NUM_OF_VALUES = 3;

Serial myPort;
String myString;

// This is the array of values you might want to send to Arduino.
int values[] = new int[NUM_OF_VALUES];

Capture video;
OpenCV opencv;
PImage src;
ArrayList<Contour> contours;
ArrayList<Contour> contours2;

// <1> Set the range of Hue values for our filter
int rangeLow = 27;
int rangeHigh = 32;
int rangeLow2 = 128;
int rangeHigh2 = 137;


import oscP5.*;
OscP5 oscP5;

PVector posePosition;
PVector poseOrientation;

boolean found;
float eyeLeftHeight;
float eyeRightHeight;
float mouthHeight;
float mouthWidth;
float nostrilHeight;
float leftEyebrowHeight;
float rightEyebrowHeight;

float poseScale;



void setup() {
  video = new Capture(this, 640, 480);
  video.start();

  opencv = new OpenCV(this, video.width, video.height);
  contours = new ArrayList<Contour>();

  size(1400, 480, P2D);


  frameRate(30);

  posePosition = new PVector();
  poseOrientation = new PVector();

  oscP5 = new OscP5(this, 8338);
  oscP5.plug(this, "mouthWidthReceived", "/gesture/mouth/width");
  oscP5.plug(this, "mouthHeightReceived", "/gesture/mouth/height");
  oscP5.plug(this, "eyebrowLeftReceived", "/gesture/eyebrow/left");
  oscP5.plug(this, "eyebrowRightReceived", "/gesture/eyebrow/right");
  oscP5.plug(this, "eyeLeftReceived", "/gesture/eye/left");
  oscP5.plug(this, "eyeRightReceived", "/gesture/eye/right");
  oscP5.plug(this, "jawReceived", "/gesture/jaw");
  oscP5.plug(this, "nostrilsReceived", "/gesture/nostrils");
  oscP5.plug(this, "found", "/found");
  oscP5.plug(this, "poseOrientation", "/pose/orientation");
  oscP5.plug(this, "posePosition", "/pose/position");
  oscP5.plug(this, "poseScale", "/pose/scale");


  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 7 ], 9600);
  // check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index of the port

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = 'n'  Linefeed in ASCII
  myString = null;
}


void draw() {

  // Read last captured frame
  if (video.available()) {
    video.read();
  }

  //First Color
  // <2> Load the new frame of our movie in to OpenCV
  opencv.loadImage(video);

  // Tell OpenCV to use color information
  opencv.useColor();
  src = opencv.getSnapshot();

  // <3> Tell OpenCV to work in HSV color space.
  opencv.useColor(HSB);

  // <4> Copy the Hue channel of our image into 
  //     the gray channel, which we process.
  opencv.setGray(opencv.getH().clone());

  // <5> Filter the image based on the range of 
  //     hue values that match the object we want to track.
  opencv.inRange(rangeLow, rangeHigh);

  // <6> Get the processed image for reference.
  //colorFilteredImage = opencv.getSnapshot();

  ///////////////////////////////////////////
  // We could process our image here!
  // See ImageFiltering.pde
  ///////////////////////////////////////////

  // <7> Find contours in our range image.
  //     Passing 'true' sorts them by descending area.
  contours = opencv.findContours(true, true);
  pushMatrix();
  scale(-1, 1);  
  translate(-640, 0);
  // <8> Display background images
  image(src, 0, 0);

  //image(colorFilteredImage, src.width, 0);

  // <9> Check to make sure we've found any contours
  if (contours.size() > 0) {
    // <9> Get the first contour, which will be the largest one
    Contour biggestContour = contours.get(0);

    // <10> Find the bounding box of the largest contour,
    //      and hence our object.
    Rectangle r = biggestContour.getBoundingBox();

    // <11> Draw the bounding box of our object
    noFill(); 
    strokeWeight(2); 
    stroke(255, 0, 0);
    rect(r.x, r.y, r.width, r.height);

    // <12> Draw a dot in the middle of the bounding box, on the object.
    noStroke(); 
    fill(255, 0, 0);
    ellipse(r.x + r.width/2, r.y + r.height/2, 30, 30);
    //println(r.x + r.width/2);
    //println(r.y + r.height/2);
    popMatrix();
  }



  //Second Color
  opencv.loadImage(video);

  // Tell OpenCV to use color information
  opencv.useColor();
  src = opencv.getSnapshot();

  // <3> Tell OpenCV to work in HSV color space.
  opencv.useColor(HSB);

  // <4> Copy the Hue channel of our image into 
  //     the gray channel, which we process.
  opencv.setGray(opencv.getH().clone());

  // <5> Filter the image based on the range of 
  //     hue values that match the object we want to track.
  opencv.inRange(rangeLow2, rangeHigh2);

  // <6> Get the processed image for reference.
  //colorFilteredImage = opencv.getSnapshot();

  ///////////////////////////////////////////
  // We could process our image here!
  // See ImageFiltering.pde
  ///////////////////////////////////////////

  // <7> Find contours in our range image.
  //     Passing 'true' sorts them by descending area.
  contours2 = opencv.findContours(true, true);
  pushMatrix();
  scale(-1, 1);  
  translate(-640, 0);
  // <8> Display background images
  //image(src, 0, 0);

  //image(colorFilteredImage, src.width, 0);

  // <9> Check to make sure we've found any contours
  if (contours2.size() > 0) {
    // <9> Get the first contour, which will be the largest one
    Contour biggestContour2 = contours2.get(0);

    // <10> Find the bounding box of the largest contour,
    //      and hence our object.
    Rectangle m = biggestContour2.getBoundingBox();

    // <11> Draw the bounding box of our object
    noFill(); 
    strokeWeight(2); 
    stroke(255, 0, 0);
    rect(m.x, m.y, m.width, m.height);

    // <12> Draw a dot in the middle of the bounding box, on the object.
    noStroke(); 
    fill(129, 42, 183);
    ellipse(m.x + m.width/2, m.y + m.height/2, 30, 30);
    //println(m.x + m.width/2);
    //println(m.y + m.height/2);
    popMatrix();
  }




  //frog
  pushMatrix();
  scale(-1, 1);  
  translate(-2040, 0);
  fill(137, 70, 183);
  rect(640, 0, 760, height);
  translate(width/2, 0);
  stroke(0);
  // if (found) {
  Contour biggestContour = contours.get(0);
  Rectangle r = biggestContour.getBoundingBox();
  Contour biggestContour2 = contours2.get(0);
  Rectangle m = biggestContour2.getBoundingBox();
  translate(posePosition.x, posePosition.y);
  //translate(width/2, height/2);
  scale(poseScale);
  //    ellipse(100,100,100,100);
  //rect(100,100,200,100);
  noFill();
  line(-11, eyeLeftHeight * -9 - 10, -35, -43);
  line(11, eyeRightHeight * -9 - 10, 35, -43);
  //arc(-20, eyeLeftHeight * -9 - 5, 35, 15, radians(200), radians(340));
  //arc(20, eyeRightHeight * -9 - 5, 35, 15, radians(200), radians(340));
  // ellipse(0,0, 3,3);
  fill(255);
  ellipse(-20, eyeLeftHeight * -9, 30, 10);
  ellipse(20, eyeRightHeight * -9, 30, 10);
  fill(0);
  ellipse(-20, eyeLeftHeight * -9, 10, 10);
  ellipse(20, eyeRightHeight * -9, 10, 10);
  fill(234, 38, 38);
  ellipse(0, 10, mouthWidth* 4.5, mouthHeight * 3);
  noFill();
  arc(0, nostrilHeight * -1 + 15, 100, 40, radians(25), radians(155));
  fill(245, 185, 204);
  ellipse(-50, eyeLeftHeight * -9 + 20, 16, 8);
  ellipse(50, eyeRightHeight * -9 + 20, 16, 8);
  noFill();
  arc(-20, eyeLeftHeight * -9 + 74, 45, 45, radians(180), radians(250));
  arc(20, eyeRightHeight * -9 + 74, 45, 45, radians(290), radians(360));
  float RightHandX = map(r.x, 0, 500, -100, 100);
  //println("yyyyyy: "+r.x);
  //line(-40, eyeLeftHeight * -9 + 60, RightHandX + r.width/2 - 500, r.y + r.height/2 - 330);
  line(-40, eyeLeftHeight * -9 + 60, RightHandX, r.y + r.height/2 - 330);
  line(RightHandX, r.y + r.height/2 - 330, RightHandX - 5, r.y + r.height/2 - 315);
  line(RightHandX, r.y + r.height/2 - 330, RightHandX + 5, r.y + r.height/2 - 317);
  line(RightHandX, r.y + r.height/2 - 330, RightHandX - 13, r.y + r.height/2 - 320);
  float LeftHandX = map(m.x, 0, 500, -100, 100);
  //line(40, eyeRightHeight * -9 + 60, LeftHandX + m.width/2 - 500, m.y + m.height/2 - 330);
  line(40, eyeRightHeight * -9 + 60, LeftHandX, m.y + m.height/2 - 330);
  line(LeftHandX, m.y + m.height/2 - 330, LeftHandX - 5, m.y + m.height/2 - 315);
  line(LeftHandX, m.y + m.height/2 - 330, LeftHandX + 5, m.y + m.height/2 - 317);
  line(LeftHandX, m.y + m.height/2 - 330, LeftHandX - 13, m.y + m.height/2 - 320);
  fill(0);
  rect(-2.5, eyeRightHeight * -9 + 57.5, 5, 5, 18, 18, 18, 18);
  triangle(0, eyeRightHeight * -9 + 60, -20, eyeRightHeight * -9 + 55, -20, eyeRightHeight * -9 + 65);
  triangle(0, eyeRightHeight * -9 + 60, 20, eyeRightHeight * -9 + 55, 20, eyeRightHeight * -9 + 65);
  fill(0);
  // }
  popMatrix();
  
  
   int z = round (map(mouthWidth, 10, 17, 0, 180 )); 
   println("mouth width is " + z);
   values[0] = z;

  //for (int i = 0; i < 100; i++) {
  // int n = millis();
  // float a = n % (m.y + m.height/2 - 330);
  // println("AAAAA: "+a);



  // float V = m.y + m.height/2 - 330;
  // println("VVVVV: "+V);

  // float W = n % (V - a);
  // println("WWWWW: "+W);
  // if (int(W) > 0) {
  //  values[0] = 550;
  //} else {
  //  values[0] = 300;
  //}

  // }


  //long currentMillis = millis();

  //if (currentMillis - previousMillis >= interval) {
    // save the last time you blinked the LED
    //previousMillis = currentMillis;
    float V = m.y + m.height/2 - 330;
    float d = map(V,130,-340,0,180);
    int e = round(d);
    println("LeftHand height: " + e);
    values[1] = e;


    float Q = r.y + r.height/2 - 330;
    float f = map(Q,130,-340,0,180);
    int g = floor(f);
    println("RightHand height: " + g);
    values[2] = g;
    //  println("VVVVV: "+V);
  //}




  sendSerialData();
  echoSerialData(1);
  //println(r.x, m.x);
}






public void mouthWidthReceived(float w) {
  //println("mouth Width: " + w);
  mouthWidth = w;
}

public void mouthHeightReceived(float h) {
  //println("mouth height: " + h);
  mouthHeight = h;
}

public void eyebrowLeftReceived(float h) {
  //println("eyebrow left: " + h);
  leftEyebrowHeight = h;
}

public void eyebrowRightReceived(float h) {
  //println("eyebrow right: " + h);
  rightEyebrowHeight = h;
}

public void eyeLeftReceived(float h) {
  //println("eye left: " + h);
  eyeLeftHeight = h;
}

public void eyeRightReceived(float h) {
  //println("eye right: " + h);
  eyeRightHeight = h;
}

public void jawReceived(float h) {
  //println("jaw: " + h);
}

public void nostrilsReceived(float h) {
  //println("nostrils: " + h);
  nostrilHeight = h;
}

public void found(int i) {
  //println("found: " + i); // 1 == found, 0 == not found
  found = i == 1;
}

public void posePosition(float x, float y) {
  //println("pose positiontX: " + x + " Y: " + y );
  posePosition.x = x;
  posePosition.y = y;
}

public void poseScale(float s) {
  //println("scale: " + s);
  poseScale = s;
}

public void poseOrientation(float x, float y, float z) {
  //println("pose orientationtX: " + x + " Y: " + y + " Z: " + z);
  poseOrientation.x = x;
  poseOrientation.y = y;
  poseOrientation.z = z;
}


void oscEvent(OscMessage theOscMessage) {
  if (theOscMessage.isPlugged()==false) {
    //println("UNPLUGGED: " + theOscMessage);
  }
}

void mousePressed() {
  /*
  for (int i = 0; i < 2; i++) {
   
   color c = get(mouseX, mouseY);
   println("r: " + red(c) + " g: " + green(c) + " b: " + blue(c));
   
   int hue = int(map(hue(c), 0, 255, 0, 180));
   println("hue to detect: " + hue);
   
   rangeLow = hue - 5;
   rangeHigh = hue + 5;
   
   
   }
   */




  color c = get(mouseX, mouseY);
  println("r: " + red(c) + " g: " + green(c) + " b: " + blue(c));

  int hue = int(map(hue(c), 0, 255, 0, 180));
  println("hue to detect: " + hue);

  rangeLow = hue - 5;
  rangeHigh = hue + 5;
}



void sendSerialData() {
  String data = "";
  for (int i=0; i<values.length; i++) {
    data += values[i];
    //if i is less than the index number of the last element in the values array
    if (i < values.length-1) {
      data += ","; // add splitter character "," between each values element
    } 
    //if it is the last element in the values array
    else {
      data += "n"; // add the end of data character "n"
    }
  }
  //write to Arduino
  myPort.write(data);
}


void echoSerialData(int frequency) {
  //write character 'e' at the given frequency
  //to request Arduino to send back the values array
  if (frameCount % frequency == 0) myPort.write('e');

  String incomingBytes = "";
  while (myPort.available() > 0) {
    //add on all the characters received from the Arduino to the incomingBytes string
    incomingBytes += char(myPort.read());
  }
  //print what Arduino sent back to Processing
  print( incomingBytes );
}




Arduino part:
#define NUM_OF_VALUES 3    /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
#include <Servo.h>


/** DO NOT REMOVE THESE **/
int tempValue = 0;
int valueIndex = 0;

/* This is the array of values storing the data from Processing. */
int values[NUM_OF_VALUES];

int righthand = 11;
int lefthand = 9;
int mouth = 13;


Servo servoR;
Servo servoL;
Servo servoM;



void setup() {
  Serial.begin(9600);
  servoR.attach(righthand);
  servoL.attach(lefthand);
  servoM.attach(mouth);

}

void loop() {
  getSerialData();

  // add your code here
  // use elements in the values array
  // values[0]
  // values[1]



  servoL.write(values[1]);
  servoR.write(values[2]);
  //servoM.write(values[0]);
  delay(50);
  //Serial.println(values[2]);
  
  

  
}


//recieve serial data from Processing
void getSerialData() {
  while (Serial.available()) {
    char c = Serial.read();
    //switch - case checks the value of the variable in the switch function
    //in this case, the char c, then runs one of the cases that fit the value of the variable
    //for more information, visit the reference page: https://www.arduino.cc/en/Reference/SwitchCase
    switch (c) {
      //if the char c from Processing is a number between 0 and 9
      case '0'...'9':
        //save the value of char c to tempValue
        //but simultaneously rearrange the existing values saved in tempValue
        //for the digits received through char c to remain coherent
        //if this does not make sense and would like to know more, send an email to me!
        tempValue = tempValue * 10 + c - '0';
        break;
      //if the char c from Processing is a comma
      //indicating that the following values of char c is for the next element in the values array
      case ',':
        values[valueIndex] = tempValue;
        //reset tempValue value
        tempValue = 0;
        //increment valuesIndex by 1
        valueIndex++;
        break;
      //if the char c from Processing is character 'n'
      //which signals that it is the end of data
      case 'n':
        //save the tempValue
        //this will b the last element in the values array
        values[valueIndex] = tempValue;
        //reset tempValue and valueIndex values
        //to clear out the values array for the next round of readings from Processing
        tempValue = 0;
        valueIndex = 0;
        break;
      //if the char c from Processing is character 'e'
      //it is signalling for the Arduino to send Processing the elements saved in the values array
      //this case is triggered and processed by the echoSerialData function in the Processing sketch
      case 'e': // to echo
        for (int i = 0; i < NUM_OF_VALUES; i++) {
          Serial.print(values[i]);
          if (i < NUM_OF_VALUES - 1) {
            Serial.print(',');
          }
          else {
            Serial.println();
          }
        }
        break;
    }
  }
}

Class 25 Assignment(Rudi)-Samantha

After reading the reading and going through the slides, I decide to choose the “clever Cheese installation” developed by Christian Möller. I think this is a very interesting installation and it is so surprising to find that it can detect how “sincere” the actresses were smiling, which is very magical. This means that computers are able to know how people feel and how true their feelings are through analyzing their facial expressions. I assume that maybe this could be the first step for computers to be as intelligent as humans to understand how people feel and become more like human because of this ability to read expressions. If they can do this, then they may have the capacity to do things that are even more amazing, like deducing what we might be thinking or why we show a certain kind of expression, being truly “intelligent”. And I think this installation is similar to the face recognition technology in real life and in my project. In real life, when we register or some applications and enter or exit a country, our faces have to be scanned to keep the data and to make sure we are qualified. In my project, I used OpenCV to detect some of the facial movements of users. I think these techniques are similar to that of the “clever Cheese installation”, but the latter is clearly much more advanced because it also has to evaluate the expressions.

        

see a video: http://christianmoeller.com/Cheese

 As what was mentioned in the reading, Christian Möller used “six flat panel monitors”, so I think they might also apply the face recognition technology. The computer tracked the actresses’ faces and the angles of their eyes, mouths etc. And I think it is very hard for the developers to enable the computer to make correct analyses of whether the smiles were sincere or not. In order to do this, they have to do a lot of research on what different expressions indicate and send all these data of findings to the computer. And there are millions of expressions as well as changes of expression. Even a slight change may bring a huge difference to the meaning of a certain expression. So in order to be accurate, the developers have to study and define countless situations and data. And here is an interesting link I found about the recognition of expressions: http://concept-script.com/lectures/11_EXPRESSION_OF_EMOTIONS.pdf

Interaction Lab Recitation Lab 11(Rudi)-Samantha

This week we are trying to use a physical controller to control media elements in processing. And I was very interested in the blur function of images in processing so I decided to use the potentiometer to control how blurry the image gets. This was my initial idea and I began to search for pictures in the links given in the recitation instructions, where I found a beautiful picture of a bird called blue jay. This inspired me to think of adding audio of the singing sounds of birds to processing. And because the visual effect of the picture is controlled by the potentiometer, I came up with the idea of using it to control the volume of the sound, too. Then the whole idea will be that the volume of the singing sound becomes high when the blue jay can be seen clearly and low when it is blurry.  Continue reading

Arduino code:
void setup() {
  Serial.begin(9600);
  
}


void loop() {
  int sensorValue = map(analogRead(A0), 0, 1023, 0, 70);
  Serial.write(sensorValue);
  delay(10);
}


Processing code:
PImage photo;
import processing.sound.*;
SoundFile file;

import processing.serial.*;

Serial myPort;
int valueFromArduino = 0;

float volumeValue;
  
void setup() {
  size(800, 800);
  
  printArray(Serial.list());
  // this prints out the list of all available serial ports on your computer.
  
  myPort = new Serial(this, Serial.list()[6], 9600);
  
  photo = loadImage("Blue-jay.jpg");
  file = new SoundFile(this,"BirdWhistle.wav");
  file.loop();
  //file.amp(-1 * valueFromArduino);
}

void draw() {
  image(photo, 0, 0);
  
  
   while ( myPort.available() > 0) {
    valueFromArduino = myPort.read();
    println(valueFromArduino);
  }
  
  filter(BLUR, valueFromArduino);
  
  float volume = map(valueFromArduino, 0, 70, 1, 0);
  file.amp(volume);

}

Interaction Lab Final Project Essay(Rudi)-Samantha

I think interaction describes the kind of relationship between the action of one thing and another thing’s/things’ response to it. That is to say, it is a mechanism where one thing responds to another, creating some effects on one or all things engaged in this kind of come-and-go communication. There are a lot of examples of interaction in real life and it happens almost all the time. The revolving doors of our school’s academic building are always interacting with us when we use them. We walk straight to the revolving door and get closer and closer. Once we are close to it enough(the distance between us and the door is smaller than a certain value), the still door starts to respond to our action of walking near to it and begins to revolve to let us get in and transfers us to the other side. So the door interacts with us in order to enable us to use it to go to the other side of it when we want to.  Continue reading

Interaction Lab Stamp(Rudi)-Samantha

In class 19, we were asked to use Adobe Illustrator to create the drawing of a stamp and below is the stamp I drew:

      

I’m thinking of applying similar techniques to draw the patterns of the box of the puppet cinema in which we put the scene and the puppet in the final project by using the laser cutting machine and maybe try using the 3D printer to print the tree trunk as one of the props.

Interaction Lab Recitation Lab 8(Rudi)-Samantha

This week we are trying to use stepper motors to create drawing machines and I find this very interesting because it is exciting to make the arms draw things automatically following the instructions sent from Processing. My partner was Quoey and we really had fun building it.

First, I built the circuit that controlled the stepper according to the schematic given. And this should be done very carefully because there were a lot of wires needed and we should always check the circuit thoroughly before connecting it to the computer in case of accidents like short circuits. And I didn’t succeed moving my motor properly at first because I mixed the positions of two wires. Continue reading

Arduino part:
int valueFromProcessing;
const int stepsPerRevolution = 200;
#include <Stepper.h>

// change this to the number of steps on your motor
#define STEPS 200

// create an instance of the stepper class, specifying
// the number of steps of the motor and the pins it's
// attached to
Stepper stepper(STEPS, 8, 9, 10, 11);

// the previous reading from the analog input
int previous = 0;

void setup() {
  // set the speed of the motor to 30 RPMs
  stepper.setSpeed(30);
  Serial.begin(9600);
}

void loop() {
    //int val = analogRead(0);
  //map(val, 0, 1023, 0, 200);

  if (valueFromProcessing == 'H') {
  // get the sensor value
  stepper.step(stepsPerRevolution);
  }else{
    stepper.step(-stepsPerRevolution);
  }

  // remember the previous value of the sensor
  //previous = val;
  
}

Processing part:
import processing.serial.*;

Serial myPort;



void setup() {
  size(500, 500);
  background(0);

  printArray(Serial.list());
  // this prints out the list of all available serial ports on your computer.
  
  myPort = new Serial(this, Serial.list()[ 6 ], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.
}


void draw() {
  // to send a value to the Arduino
  if (mousePressed) {
    myPort.write('H');
  } else {
    myPort.write('L');
  }
}

Interaction Lab Mid-term: My Own Project(Rudi)-Samantha

The project I made for mid-term is called “The Song of a Goldfish” and I first got the idea by combining several elements that I really loved, which are Chinese-style-drawn goldfish, the
“storing input” function in processing and the piano. The goldfish touching the bubbles and the “storing input” function gave me the inspiration to let the fish touch bubbles with trace. Then the piano gives me the idea to make the bubbles work in a way similar to the keys. All these made up my rough draft of the project. Considered that the goldfish and bubbles are involved, I came up with the idea of adding a seashell whose pearl can be a LED which changes color when the bubbles are touched. So this is how I got my storyboard. Continue reading

Arduino part:
#define NUM_OF_VALUES 4    /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/


/** DO NOT REMOVE THESE **/
int tempValue = 0;
int valueIndex = 0;

/* This is the array of values storing the data from Processing. */
int values[NUM_OF_VALUES];

int red = 9; //this sets the red led pin
int green = 10; //this sets the green led pin
int blue = 6; //this sets the blue led pin
int buzzer = 5;
float f;

void setup() {
  Serial.begin(9600);
  pinMode(red, OUTPUT);
  pinMode(green, OUTPUT);
  pinMode(blue, OUTPUT);
  pinMode(buzzer, OUTPUT);
}

void loop() {
  getSerialData();

  // add your code here
  // use elements in the values array
  // values[0]
  // values[1]
  analogWrite(red, values[0]);
  analogWrite(green, values[1]);
  analogWrite(blue, values[2]);
 // Serial.println(values[0]);
 // Serial.println(values[1]);
 // Serial.println(values[2]);
  
  if (values[3] > 0) {
    f = map(values[3], 0, 255, 400, 1000);
    tone(buzzer, f, 100);
    //  delay(100);
    noTone(8);
  }
}


//recieve serial data from Processing
void getSerialData() {
  if (Serial.available()) {
    char c = Serial.read();
    //switch - case checks the value of the variable in the switch function
    //in this case, the char c, then runs one of the cases that fit the value of the variable
    //for more information, visit the reference page: https://www.arduino.cc/en/Reference/SwitchCase
    switch (c) {
      //if the char c from Processing is a number between 0 and 9
      case '0'...'9':
        //save the value of char c to tempValue
        //but simultaneously rearrange the existing values saved in tempValue
        //for the digits received through char c to remain coherent
        //if this does not make sense and would like to know more, send an email to me!
        tempValue = tempValue * 10 + c - '0';
        break;
      //if the char c from Processing is a comma
      //indicating that the following values of char c is for the next element in the values array
      case ',':
        values[valueIndex] = tempValue;
        //reset tempValue value
        tempValue = 0;
        //increment valuesIndex by 1
        valueIndex++;
        break;
      //if the char c from Processing is character 'n'
      //which signals that it is the end of data
      case 'n':
        //save the tempValue
        //this will b the last element in the values array
        values[valueIndex] = tempValue;
        //reset tempValue and valueIndex values
        //to clear out the values array for the next round of readings from Processing
        tempValue = 0;
        valueIndex = 0;
        break;
      //if the char c from Processing is character 'e'
      //it is signalling for the Arduino to send Processing the elements saved in the values array
      //this case is triggered and processed by the echoSerialData function in the Processing sketch
      case 'e': // to echo
        for (int i = 0; i < NUM_OF_VALUES; i++) {
          Serial.print(values[i]);
          if (i < NUM_OF_VALUES - 1) {
            Serial.print(',');
          }
          else {
            Serial.println();
          }
        }
        break;
    }
  }
}

Processing part:
import processing.serial.*;

int NUM_OF_VALUES = 4;  /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
float red;
float green;
float blue;
float buzzer;

Serial myPort;
String myString;

// This is the array of values you might want to send to Arduino.
int values[] = new int[NUM_OF_VALUES];

int x = 125;
int y = 375;
int speed = 1;

int num = 40;
//float mx[] = new float[num];
//float my[] = new float[num];
float b1Y[] = new float[num];
float b2Y[] = new float[num];
float b3Y[] = new float[num];
float b4Y[] = new float[num];
float b5Y[] = new float[num];
float B2Y[] = new float[num];
float B3Y[] = new float[num];
float B4Y[] = new float[num];
float B5Y[] = new float[num];


void setup() {
  size(900, 500);
  noStroke();
  //stroke(217, 247, 252);
  //strokeWeight(2);
  //fill(217, 247, 252);
for (int i = 0; i < num; i++) {
b1Y[i] = -100;
}
for (int i = 0; i < num; i++) {
b2Y[i] = -100;
}
for (int i = 0; i < num; i++) {
b3Y[i] = -100;
}
for (int i = 0; i < num; i++) {
b4Y[i] = -100;
}
for (int i = 0; i < num; i++) {
b5Y[i] = -100;
}
for (int i = 0; i < num; i++) {
B2Y[i] = -100;
}
for (int i = 0; i < num; i++) {
B3Y[i] = -100;
}
for (int i = 0; i < num; i++) {
B4Y[i] = -100;
}
for (int i = 0; i < num; i++) {
B5Y[i] = -100;
}
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 6 ], 9600);
  // check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index of the port

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = 'n'  Linefeed in ASCII
  myString = null;

  //  frameRate(120);
}

void draw() {
  background(255);


  // Instead of writing out all the code about the ball is draw(), I 
  // simply call three functions. How do I know the names of these 
  // functions? I made them up!
  move1(); 
  bounce1();
  display1();

  move2(); 
  bounce2();
  display2();


fish();
  //triangle(mouseX - 25, mouseY - 10, mouseX - 35, mouseY - 23, mouseX - 50, mouseY - 27);

  
  //fill(245, 119, 150);
  //ellipse(mx[index], my[index], i - 4, i - 4);
  //ellipse(mx[index], my[index] + 15, i - 4, i - 4);
  //fill(255, 255, 255);
  //ellipse(mx[index], my[index], i - 10, i - 10);
  //ellipse(mx[index], my[index] + 15, i - 10, i- 10);
  //fill(0, 0, 0);
  //ellipse(mx[index], my[index], i - 17, i - 17);
  //ellipse(mx[index], my[index] + 15, i - 17, i - 17);
  //fill(245, 119, 150);
  //ellipse(mx[index] - 30, my[index] + 7.5, 49, 45);
  //quad(mx[index] - 50, my[index] + 7.5, mx[index] - 100, my[index] - 8, mx[index] - 110, my[index] - 25, mx[index] - 80, my[index] - 20);
  //quad(mx[index] - 50, my[index] + 7.5, mx[index] - 100, my[index] + 15.5, mx[index] - 110, my[index] + 32.5, mx[index] - 80, my[index] + 27.5);
  //arc(mx[index] - 50, my[index], 3 * i, 3 * i, radians(0), radians(200));
  //arc(mx[index] - 50, my[index], 3 * i, 3 * i, radians(160), radians(360));
  //}

  // sends the values to Arduino.
  sendSerialData();

  // This causes the communication to become slow and unstable.
  // You might want to comment this out when everything is ready.
  // The parameter 20 is the frequency of echoing. 
  // The higher this number, the slower the program will be
  // but the higher this number, the more stable it will be.
  echoSerialData(200);
}

// Where should functions be placed?
// You can define your functions anywhere in the code 
// outside of setup() and draw(). However, the convention 
// is to place your function definitions below draw().

// A function to move the ball
void move1() { 
  // Change the x location by speed
  x = x + speed;
}

// A function to bounce the ball
void bounce1() {
  // If we’ve reached an edge, reverse speed

  if ((x > height) || (x < 125)) {
    speed = speed * - 1;
  }
}

// A function to display the ball
void display1() {

  int BubbleTouched = 0;
 

  //BubbleB1
  if (dist(mouseX, mouseY, width/10, x) < 30) {
    //fill(188, 222, 40);
    //ellipse(width/10, x, 40, 40);
    BubbleTouched = 1;
    values[0] = 188;
    values[1] = 222;
    values[2] = 40;
    values[3] = 10;
    println("b1");
  } 
  //BubbleB2
  else if (dist(mouseX, mouseY, (width/10) * 3, x) < 30) {
    //fill(44, 176, 240);
    //ellipse((width/10) * 3, x, 40, 40);
    BubbleTouched = 2;
    values[0] = 44;
    values[1] = 176;
    values[2] = 240;
    values[3] = 30;
     println("b2");
  }
  //BubbleB3
  else if (dist(mouseX, mouseY, (width/10) * 5, x) < 30) {
    //fill(28, 201, 26);
    //ellipse((width/10) * 5, x, 40, 40);
    BubbleTouched = 3;
    values[0] = 28;
    values[1] = 201;
    values[2] = 26;
    values[3] = 50;
     println("b3");
  }
  //BubbleB4
  else if (dist(mouseX, mouseY, (width/10) * 7, x) < 30) {
    //fill(255, 0, 0);
    //ellipse((width/10) * 7, x, 40, 40);
    BubbleTouched = 4;
    values[0] = 255;
    values[1] = 0;
    values[2] = 0;
    values[3] = 70;
     println("b4");
  }
  //BubbleB5
  else if (dist(mouseX, mouseY, (width/10) * 9, x) < 30) {
    //fill(8, 206, 181);
    //ellipse((width/10) * 9, x, 40, 40);
    BubbleTouched = 5;
    values[0] = 8;
    values[1] = 206;
    values[2] = 181;
    values[3] = 90;
     println("b5");
  } else {
    fill(217, 247, 252);
    values[0] = 0;
    values[1] = 0;
    values[2] = 0;
    values[3] = 0;
  }
  
  if (BubbleTouched == 1) {
    fill(188, 222, 40, 50);
    trace1();
    fill(188, 222, 40);
    ellipse(width/10, x, 40, 40);
  }
  else {
    fill(217, 247, 252, 50);
    trace1();
    fill(217, 247, 252);
    ellipse(width/10, x, 40, 40);
  }
  if (BubbleTouched == 2) {
    fill(44, 176, 240, 50);
    trace2();
    fill(44, 176, 240);
    ellipse(width/10 * 3, x, 40, 40);
  }
  else {
    fill(217, 247, 252, 50);
    trace2();
    fill(217, 247, 252);
    ellipse(width/10 * 3, x, 40, 40);
  }
  if (BubbleTouched == 3) {
    fill(28, 201, 26, 50);
    trace3();
    fill(28, 201, 26);
    ellipse(width/10 * 5, x, 40, 40);
  }
  else {
    fill(217, 247, 252, 50);
    trace3();
    fill(217, 247, 252);
    ellipse(width/10 * 5, x, 40, 40);
  }
  if (BubbleTouched == 4) {
    fill(255, 0, 0, 50);
    trace4();
    fill(255, 0, 0);
    ellipse(width/10 * 7, x, 40, 40);
  }
  else {
    fill(217, 247, 252, 50);
    trace4();
    fill(217, 247, 252);
    ellipse(width/10 * 7, x, 40, 40);
  }
  if (BubbleTouched == 5) {
    fill(8, 206, 181, 50);
    trace5();
    fill(8, 206, 181);
    ellipse(width/10, x * 9, 40, 40);
  }
  else {
    fill(217, 247, 252, 50);
    trace5();
    fill(217, 247, 252);
    ellipse(width/10 * 9, x, 40, 40);
  }
  
  //fill(217, 247, 252);
  //ellipse((width/10) * 9, x, 40, 40);
  
}


void move2() { 
  // Change the x location by speed
  y = y - speed;
}

// A function to bounce the ball
void bounce2() {
  // If we’ve reached an edge, reverse speed
  if ((y > 375) || (y < 0)) {
    speed = speed * - 1;
  }
}

// A function to display the ball
void display2() {
  
  int Bubbletouched = 0;

  fill(217, 247, 252);
  ellipse(width/20, y, 40, 40);
  //ellipse((width/10) * 2, y, 40, 40);
  //ellipse((width/10) * 4, y, 40, 40);
  //ellipse((width/10) * 6, y, 40, 40);
  //ellipse((width/10) * 8, y, 40, 40);
  ellipse((width/10) * 10 - width/20, y, 40, 40);
  
  //BubbleA1
  if (dist(mouseX, mouseY, width/20, y) < 30) {
    fill(237, 126, 7);
    ellipse(width/20, y, 40, 40);
    values[0] = 237;
    values[1] = 126;
    values[2] = 7;
    values[3] = 1;
  }
  //BubbleA2
  else if (dist(mouseX, mouseY, (width/10) * 2, y) < 30) {
    //fill(227, 21, 69);
    //ellipse((width/10) * 2, y, 40, 40);
    Bubbletouched = 1;
    values[0] = 227;
    values[1] = 21;
    values[2] = 69;
    values[3] = 20;
  }
  //BubbleA3
  else if (dist(mouseX, mouseY, (width/10) * 4, y) < 30) {
    //fill(255, 234, 0);
    //ellipse((width/10) * 4, y, 40, 40);
    Bubbletouched = 2;
    values[0] = 255;
    values[1] = 234;
    values[2] = 0;
    values[3] = 40;
  }
  //BubbleA4
  else if (dist(mouseX, mouseY, (width/10) * 6, y) < 30) {
    //fill(144, 6, 214);
    //ellipse((width/10) * 6, y, 40, 40);
    Bubbletouched = 3;
    values[0] = 144;
    values[1] = 6;
    values[2] = 214;
    values[3] = 60;
  }
  //BubbleA5
  else if (dist(mouseX, mouseY, (width/10) * 8, y) < 30) {
    //fill(250, 179, 86);
    //ellipse((width/10) * 8, y, 40, 40);
    Bubbletouched = 4;
    values[0] = 250;
    values[1] = 179;
    values[2] = 86;
    values[3] = 80;
  }
  //BubbleA6
  else if (dist(mouseX, mouseY, (width/10) * 10 - width/20, y) < 30) {
    fill(208, 176, 242);
    ellipse((width/10) * 10 - width/20, y, 40, 40);
    values[0] = 208;
    values[1] = 176;
    values[2] = 242;
    values[3] = 100;
  }
  /*
   else {
    fill(217, 247, 252);
    values[0] = 0;
    values[1] = 0;
    values[2] = 0;
    values[3] = 0;
  }
  */
  
  if (Bubbletouched == 1) {
    fill(227, 21, 69, 50);
    Trace2();
    fill(227, 21, 69);
    ellipse((width/10) * 2, y, 40, 40);
  }
  else {
    fill(217, 247, 252, 50);
    Trace2();
    fill(217, 247, 252);
    ellipse((width/10) * 2, y, 40, 40);
  }
  if (Bubbletouched == 2) {
    fill(255, 234, 0, 50);
    Trace3();
    fill(255, 234, 0);
    ellipse((width/10) * 4, y, 40, 40);
  }
  else {
    fill(217, 247, 252, 50);
    Trace3();
    fill(217, 247, 252);
    ellipse((width/10) * 4, y, 40, 40);
  }
  if (Bubbletouched == 3) {
    fill(144, 6, 214, 50);
    Trace4();
    fill(144, 6, 214);
    ellipse((width/10) * 6, y, 40, 40);
  }
  else {
    fill(217, 247, 252, 50);
    Trace4();
    fill(217, 247, 252);
    ellipse((width/10) * 6, y, 40, 40);
  }
  if (Bubbletouched == 4) {
    fill(250, 179, 86, 50);
    Trace5();
    fill(250, 179, 86);
    ellipse((width/10) * 8, y, 40, 40);
  }
  else {
    fill(217, 247, 252, 50);
    Trace5();
    fill(217, 247, 252);
    ellipse((width/10) * 8, y, 40, 40);
  }
  
}

void sendSerialData() {
  String data = "";
  for (int i=0; i<values.length; i++) {
    data += values[i];
    //if i is less than the index number of the last element in the values array
    if (i < values.length-1) {
      data += ","; // add splitter character "," between each values element
    } 
    //if it is the last element in the values array
    else {
      data += "n"; // add the end of data character "n"
    }
  }
  //write to Arduino
  myPort.write(data);
}


void echoSerialData(int frequency) {
  //write character 'e' at the given frequency
  //to request Arduino to send back the values array
  if (frameCount % frequency == 0) myPort.write('e');

  String incomingBytes = "";
  while (myPort.available() > 0) {
    //add on all the characters received from the Arduino to the incomingBytes string
    incomingBytes += char(myPort.read());
  }
  //print what Arduino sent back to Processing
  print( incomingBytes );
}

void fish() {
  fill(245, 119, 150);
  ellipse(mouseX, mouseY, - 4, - 4);
  ellipse(mouseX, mouseY + 15, - 4, - 4);
  fill(255, 255, 255);
  ellipse(mouseX, mouseY, - 10, - 10);
  ellipse(mouseX, mouseY + 15, - 10, - 10);
  fill(0, 0, 0);
  ellipse(mouseX, mouseY, - 17, - 17);
  ellipse(mouseX, mouseY + 15, - 17, - 17);
  fill(245, 119, 150);
  ellipse(mouseX - 30, mouseY + 7.5, 49, 45);
  quad(mouseX - 50, mouseY + 7.5, mouseX - 100, mouseY - 8, mouseX - 110, mouseY - 25, mouseX - 80, mouseY - 20);
  quad(mouseX - 50, mouseY + 7.5, mouseX - 100, mouseY + 15.5, mouseX - 110, mouseY + 32.5, mouseX - 80, mouseY + 27.5);
  quad(mouseX - 50, mouseY + 7.5, mouseX - 105, mouseY - 5, mouseX - 140, mouseY + 7.5, mouseX - 105, mouseY + 12.5);
}


void trace1(){
int slowFrameCount = frameCount/10; 
  int which = slowFrameCount % num;
  b1Y[which] = x;
  //stroke(0);
  for (int i = 0; i < num; i++) {
    //which+1 is the smallest (the oldest in the array)
    int index = (which+1 + i) % num;
    ellipse(width/10, b1Y[index], i, i);
  }
}
void trace2(){
int slowFrameCount = frameCount/10; 
  int which = slowFrameCount % num;
  b2Y[which] = x;
  //stroke(0);
  for (int i = 0; i < num; i++) {
    //which+1 is the smallest (the oldest in the array)
    int index = (which+1 + i) % num;
    ellipse(width/10 * 3, b2Y[index], i, i);
  }
}
void trace3(){
int slowFrameCount = frameCount/10; 
  int which = slowFrameCount % num;
  b3Y[which] = x;
  //stroke(0);
  for (int i = 0; i < num; i++) {
    //which+1 is the smallest (the oldest in the array)
    int index = (which+1 + i) % num;
    ellipse(width/10 * 5, b3Y[index], i, i);
  }
}
void trace4(){
int slowFrameCount = frameCount/10; 
  int which = slowFrameCount % num;
  b4Y[which] = x;
  //stroke(0);
  for (int i = 0; i < num; i++) {
    //which+1 is the smallest (the oldest in the array)
    int index = (which+1 + i) % num;
    ellipse(width/10 * 7, b4Y[index], i, i);
  }
}
void trace5(){
int slowFrameCount = frameCount/10; 
  int which = slowFrameCount % num;
  b5Y[which] = x;
  //stroke(0);
  for (int i = 0; i < num; i++) {
    //which+1 is the smallest (the oldest in the array)
    int index = (which+1 + i) % num;
    ellipse(width/10 * 9, b5Y[index], i, i);
  }
}

void Trace2(){
int slowFrameCount = frameCount/10; 
  int which = slowFrameCount % num;
  B2Y[which] = y;
  //stroke(0);
  for (int i = 0; i < num; i++) {
    //which+1 is the smallest (the oldest in the array)
    int index = (which+1 + i) % num;
    ellipse(width/10 * 2, B2Y[index], i, i);
  }
}
void Trace3(){
int slowFrameCount = frameCount/10; 
  int which = slowFrameCount % num;
  B3Y[which] = y;
  //stroke(0);
  for (int i = 0; i < num; i++) {
    //which+1 is the smallest (the oldest in the array)
    int index = (which+1 + i) % num;
    ellipse(width/10 * 4, B3Y[index], i, i);
  }
}
void Trace4(){
int slowFrameCount = frameCount/10; 
  int which = slowFrameCount % num;
  B4Y[which] = y;
  //stroke(0);
  for (int i = 0; i < num; i++) {
    //which+1 is the smallest (the oldest in the array)
    int index = (which+1 + i) % num;
    ellipse(width/10 * 6, B4Y[index], i, i);
  }
}
void Trace5(){
int slowFrameCount = frameCount/10; 
  int which = slowFrameCount % num;
  B5Y[which] = y;
  //stroke(0);
  for (int i = 0; i < num; i++) {
    //which+1 is the smallest (the oldest in the array)
    int index = (which+1 + i) % num;
    ellipse(width/10 * 8, B5Y[index], i, i);
  }
}

Interaction Lab Mid-term: Project I tested(Rudi)-Samantha

One project that impressed me most is the “dandelion”. I think it is a really awesome project because it is very beautiful aesthetically, using the fluffy white ball as the dandelion and the picture of the flower in the background of the starry night sky looks so poetically romantic. It was so amazing to see the “petals” of the dandelion on the computer screen flew in the air and slowly disappeared in the air with the “wind” when I blew the furry ball. The process is so natural as if it was exactly what happened in real life. And I felt very relaxed blowing the “dandelion” as if my stress and tiredness all went away like the fading “petals”, showing that the project has a wonderful function of healing the mind. The interaction between the user and the project is very pleasant and highly sensitive.

But at first, I was confused what I should do with the furry ball so I think it would be better to add some instructions to tell the users to blow the ball and see what will happen on the screen such as putting a little card on the stick of the furball that reads “Blow Me!”, similar to that of the story “Alice in Wonderland”, where the character drinks from bottles on which there is a tip that reads “Drink Me!”. And also, I think it would a good idea if the dandelion and the night sky background could be dynamic (for example: the dandelion would sway a bit with the gentle breeze and the stars would shine[dark and bright]) to make the experience even better. And maybe the direction in which the “petals” fly can be adjusted to follow the direction the users blow the furball.

And I’m very curious how to use the earphones to detect signals as we blow the furball.