Museum of Mediocre Artefacts: Nick Sanchez’s Documentation

Long ago, internationally infamous inventor Jingles Fakhr sought to make his name known… After a long and toilsome inventing career, creating useless and inoperable oddities, he finally made his breakthrough discovery… the Perpetual Light Machine!

For our project, we sought to make an exhibit whereby users would be drawn down a long and scary dark hallway. At the end, a contraption of some sort would sit idly, willing unsuspecting guests to draw nearer and observe it. Once they did, we behind the scenes would do something to scare them. This was the premise.

Much effort and time was spent on ideation, and there were many ideas that were either dropped entirely or subtly embedded into the final concept. This was tedious, and occupied much of our time. Nevertheless, we persisted, and eventually decided on the loose idea centered around this fictional inventor named “Jingles Fakhr”. The story was that Dr. Jingles was one of the many inventors during the 1800s, who like his contemporaries Edison and Tesla, sought to experiment with electricity and light. Though many of his inventions didn’t work (some of which we would show as exhibition to provide context during the show), his one successful invention was the “Perpetual Light Machine”. The conflict of our story arises when we share that this invention has dubious origins, causing many who view it to feel uneasy, hallucinate, and in some cases, go crazy. It is for these “reasons” that we keep this artefact hidden behind a curtain, and discourage all but the most brave guests to venture in and observe it. After they do, we would go about staging our fright.

We had picked the corner of the IMA floor where the lockers stood as the space where we would stage this experience. To create this stage, we angled the lockers so that they narrowed as you walked towards the end of the hallway. The idea was to place the “Perpetual Light Machine” towards the end of the hallway, such that people would get considerably claustrophobic as they neared it. Once these audience members walk towards it and observe it, we inauspiciously place a costumed mannequin behind them. Once the mannequin is in position, a similarly costumed actor would would jump out at them, causing them to recoil and turn around. At this point, they would suddenly see the mannequin that wasn’t behind them before, and become even more terrified.

This was the plan. The challenge became planning for it. We coordinated with IMA staff to order several key props and settings off of Taobao. These things included curtains to cover the entire stage, a mannequin, a head, and some masks and hoods to costume the mannequin and actor, a head (for the dress form).

Initially, we planned to fabricate some broken electronics to represent the two intial oddities before the final Perpetual Motion Machine. To be honest, I made an automaton that could have turned with Arduino, but never implemented the appropriate circuitry to actually animate it. Nevertheless, this “automaton” was creepy, and clearly dysfuncional, which was the point. In addition, we never really got around to creating a phonograph-like pair of headphones. Consequently, we only had the dysfunctional automaton to show as the pretext to the Perpetual Light Machine.

The Perpetual Light Machine prop was a borrowed student project from Sun Jingyi. It was a 3mm acrylic translucent pyramid, which would glow based on the Arduino-LED setup underneath it.

Setting it up was not to difficult, but we improvised as we went along, making this entire process a little more time consuming. Nevertheless, the end result was rewarding and entirely worth it.

fright4 fright3 fright2 fright1

Augmented Reality Storytelling by Tyler Rhorick

Reading Response

http://ima.nyu.sh/documentation/2017/02/15/mixed-reality-story-and-response-by-tyler-rhorick/

Blippar Intervention

http://ima.nyu.sh/documentation/2017/02/22/i-am-limitless-animation-ar-tyler-rhorick/

Live Broadcast AR

For the Live Broadcast AR assignment, I was part of the team that tried to augment the IMA equipment room to tell the story of a student who was murdered by a cat for turning in equipment late.

Personal Reflection– Overall, the process of converting the space to tell our story with the green screen went pretty well. I would say that our biggest challenge in creating accurate scale, perspective, and lighting. As for the scale and perspective, we were able to achieve a believable enough positioning of the “victim” student after moving the camera angle and Diana multiple times, but the lighting was one thing we could never remedy. I think this means that for the future we should pay better attention to lighting conditions to give our final image a better overall effect. I think we could have figured it out if given more time, however, so I am not too sad walking away from this assignment.

Your Photogrammetry

Photogrammetry remained to be one of the most difficult assignments of the semester for me, for reasons I still cannot understand. What I was trying to do in this project was create a 3D model of the meowspace to further our story we created in the Live Broadcast, but this proved to be more difficult than anticipated because of the following challenges:

    • The real MeowSpace couldn’t be used– Because the meowspace was under modification when this project was assigned, my original plan failed. I was lucky to find a 3D model of meowspace in the lab that I ended up using, but this did cause some slight panic in the beginning.
  • Creating an accurate scan- The biggest problem with creating a photogrammetry model persisted to be difficulty in capturing images that could successfully be used by the program. I think I had a difficult time because the image I was trying to scan was pretty uniform in texture and lighting was hard to control against the surface of the structure.
  • Software- Another big problem that I had was in using the software. Even if the pictures were bad I could never figure out why the program never showed me a model after following the steps in the tech template. I had shown this problem to Christian, but we still couldn’t figure out what was happening.

Here is a folder containing all of the countless cat pictures I took trying to do this assignment.

Your Game Design Document

Here is our Game Design powerpoint.

Your Core Mechanic Documentation

Here is our Core Mechanic powerpoint.

Your MR interview with Storyboard and Scan

Because there was a misunderstanding when the groups were making their way into the green screen room, Matuez and me got split from our larger group where we had made a storyboard to play off the idea of the Sims. Because of this, we had to make a new model and storyline on the spot. To make the figure, we chose Adobe Fuse because it was quick and simple. We decided to make Vladimir Putin wearing makeup because of the recent ban on such imagery in Russia. As for the interview, it was decided that I would interview Matuez acting like Vladimir about such topics like Russia , the Ukraine, and his makeup.

Here you can see the video and Matuez’s perspective of the experience.

Immersive Sound

For the immersive sound project I decided to use Unity because my Max MSP trial gave out. To do this in unity I watched several tutorials on youtube. Basically the lesson of these tutorials was basically that you need to turn on 3D sound by changing the spacial blend. Using this technique I created a simple player and audio track of the NYU Shanghai alma matter. This player walks around the scene and the sounds get fainter as the player walks away. Here is a screen shot to see what I changed in the audio part to make it 3D.

Screen Shot 2017-05-24 at 9.49.18 PM

Final Project

Project Title: Shanghai Storylines

Partner: Mateuz

Elevator Brief: Shanghai Storylines is an Augmented Reality history experience that communicates the history of Shanghai’s Pudong area. Using their phone, the user can explore the Pudong area of Shanghai and learn more about the history that has often been untold of the area. The project imagines what Shanghai would have looked like from the NYU Shanghai academic building throughout history.

Extended Description: Shanghai Storylines is made technically possible using Unity and Vuforia. To start the experience the user walks up to an old Shanghai style window. Upon scanning the Vuforia marker, the user is introduced to the experience. The first view the user is made privy to is our imaged view of Shanghai from the NYU Shanghai Academic Building in the early 1900’s. The landscape was made in Unity.

Technology: Unity, Smart Phone, Vuforia

Development: Before we could start the project, we had to do a great deal of research concerning the history of Shanghai. To start this research, I met with Anna Greenspan, as a professor that has focused on the urbanization of China. She shared with me very interested texts talking about the historic foliage of Shanghai, which was research we used in the final project when we selected to make all of the trees broadleaf evergreen models, in accordance with Shanghai’s historic ecosystem. After this research was done, we needed a better idea of what was actually built in Pudong. Though we had the idea that it was just going to be fishing villages based on widespread “knowledge,” we still decided to research to make sure that was the correct narrative.

The first lead we got on the prior history of Shanghai came from finding a map on google images of the old Shanghai area. After researching more about this map, it showed that this map was one of very few of the area at the time and is widely considered one of the most reliable models of the area at the time.  Here is that map below:

Shanghai 1945

This map proved to be monumental in moving forward because it gave us the information that Pudong was not always called Pudong, but rather was formerly called Pootung. This information helped us find much more information about the area because this is what scholars have always referred to the area as. In searching for Pootung, we came across one book by Paul French called Old Shanghai. With a very detailed description of the Pootung area, we decided to dedicate our project to his research, which included a very colorful history of old Shanghai that included a foreign cemetery for those that died at sea, foreign occupation of a land controlled by the Chinese government, and animal warehouses that doubled in the nightly trade of prostitutes.

After we finished all of the research which took up most of our time in the first weeks to make sure that we were telling a compelling narrative, we began working on the technical side of the project. In hindsight, we should have started this part of the project way sooner because Matuez and me both had no experience in Unity nor 3D modeling.

Because of this we decided to split up the Unity work. I decided to work on getting the core mechanism of Vuforia working, while Matuez worked figuring out how to get the landscapes started. When it came to getting Vuforia working, we first decided that we wanted to markerless markers, but this proved to be more difficult than we anticipated, so we went back to using a marker. I also worked on getting the core mechanism of buttons and text boxes working so that we could communicate the story of Shanghai. While I was doing this Matuez was learning how to make terrains in Unity. He sent me a working model with the terrains started and then I watched the same tutorials to finish up the models. To modify what he gave me I decided to shape the terrain actually like that of Pudong. He had given me a square terrain, but I decided to be truer to the history we should try and get the right shape to the terrain. To do this I made a plane layer of maps from new and old and sculpted the landscape. Here is how that process looked.

Screen Shot 2017-05-16 at 1.03.41 AM

 

I also decided to add water to the scene, which I also attached a script to make the water move. In addition to this I flushed out some of the areas of the experience to give it a better sense of history like the docks, graveyard, and factory part. Here are some screen grabs of the finalized look of some of these areas.

Screen Shot 2017-05-16 at 3.46.47 AM

Screen Shot 2017-05-15 at 5.39.47 AM Screen Shot 2017-05-15 at 5.39.40 AM Screen Shot 2017-05-15 at 5.40.29 AM

All of the assets were made from other elements of the free asset store. For instance, I made docks out of extremely large and distorted pallets from a warehouse collection on unity.

In the end, our mechanisms definitely worked and I think we gave a great history of the region with the time we had. For the future, I would like to expand the historical content of the project and work to make the buttons and menus feel more integrated in the experience.

Here is a video of the mechanisms working:

Nicholas Sanchez (and Abiral) Scaring the Computer

The idea of instilling fear in a person seems intuitive, as there are many ways to accomplish this task, be it telling them a horrifying story or even scaring them via surprise. But to instill fear into a computer is a different idea altogether, not only because they are different beings that humanity has yet to fully comprehend, but also because the parameters by which the computer “feels” fear are not necessarily easy to define. By anthropomorphizing the computer, and using techniques with Max MSP, we managed to turn the computer into a gear in our metaphorical demonstration.

Our idea was simple. We would “scare” the computer using psychological horror. How? well a popular and particularly disturbing sub genre of horror is body horror, named so for its grotesque mutilation of human bodies. Why does seeing the human body in contorted states upset us? what about it is so horrifying? That is not for me to say, but the principle that seeing that imagery frightens people is a well grounded idea. We decided to apply this idea to the computer, and stage its fright by an analogous means.

To do this, we staged an interrogation were we asked the computer for a set of passwords. The computer would, of course, deny us access, so we would use several methods to “coerce” the computer. After each refusal, we would use some psychological method to scare the computer. The computer would respond and gradually emulate the experience of becoming increasingly terrified.

We crafted a script of this interrogation, and then set about creating the animations for the computer. To represent the computer, we took an image of “HAL 9000” from “2001: A Space Odyssey”, and edited in photoshop so that it could be easily manipulated (turned on and off) in afterEffects. After editing these layers, we imported them into afterEffects to animate the computer getting “scared” and create each response to the interrogation.

To make the robot voice, we used http://ttsreader.com/ to generate some text that the computer would speak during the interrogation. We took these sound files, edited them a bit, and then also exported them to afterEffects.

In afterEffects, we animated the computer such that the eye lit up when it was speaking. By using regular expressions, we mapped the audio levels to keyframes, and then mapped the brightness of the eye to these keynotes. In doing this, we managed to make the eye brightest when the voice was speaking, and grow dimmer in between words or when the audio had little sound. After we had made several clips of each reactionary dialogue as well as the end, we exported them from after effects for use in MAX MSP.

Editing the computer's "eye"

Editing the computer’s “eye”

MAX MSP is a new software for us, and its implementation was challenging. While it has many advantages, such as its ability to render video and such, it unfortunately lacks much of the logic that more traditional languages use. In addition, this software is visual, and not coded line by line, which presented another challenge. Nevertheless, after hours of toil and experimentation, we managed to create a MAX MSP build that played each of the videos on command.

The next step was to integrate Arduino into the sketch. Initially, we sought to use a pressure sensor to activate each of these subsequent videos. According to our script, after each of our “motivational” actions, a pressure sensor should activate the the respective video. Unfortunately, despite our best efforts with multiple force and touch sensors, we couldn’t create a consistent response from these sensors. Ultimately, we opted to use a pushbutton to activate the computer’s responses.

the vibration sensor

the vibration sensor

Abiral experimenting with he pressure sensor

Abiral experimenting with he pressure sensor

installing the push button onto the base of the box. This would activate each video.

installing the push button onto the base of the box. This would activate each video.

 

After finally having all the technical aspects completed, we just had to get our performance in order. Using various miscellaneous discarded computer parts and some stage presence, we pulled together a performance that conveyed the computer experiencing horror. Enjoy.

 

Final Project

Due Date : May 18

Instructor : Daniel

For the Final Project, I was assigned to create an interactive system of your choice using Processing and Arduino. I wanted to make an entertaining game that reflects life in Shanghai. As I like shooting game, I decided to make a first-person view shooting arcade game engaging the physical part as game controlling. Thinking about the target to shoot, I could think of smog which is a serious issue in the city, and set balloons with the masks hung on it as the target.

The most challenging and time consuming part was to figure out how to track the gun and how to implement first person view. First I considered using angle sensor and distance sensor as options, but player would not feel comfortable and feel like playing the arcade game with the fixed gun with a lot of limitations controlling it. Getting some advice from the professor and other classmates, I decided to use color tracking with web cam; color of the gun would be tracked and that tracked position control the view of the screen. I went to the the Computer Vision workshop held by professor Moon learned how the color tracking works. I wanted to apply it to track the position of the color, but there were a lot of problems find the exact position of the object I want to track because of the lighting and different angle affects the color the web cam recognizes. So I did a lot of research to do efficient object tracking, tried different kinds of libraries of color and blob tracking, and ended up using the codes from the multiple color tracking that reflect both the hsb and rgb features in the openCV library.

For displaying the game screen as a first person perspective, I considered different options such as using 3D background, using camera() function in the processing, etc. But as I was not familiar with these options, I just decided to control the location of the image displayed. This means Processing is just reprinting the image at the different location as the gun moves.

Then I started to make game screen, which has three kinds of background – day, night, and twilight, and added random colored balloons with masks popping up in random timing. Then I set the timer and score function, and displayed it at the top of the game screen. Followings are the screenshots of the game screen.

s3s1

s2

For the Arduino part, push button became a trigger of the gun, so that if I push the button, I can fire the gun. Serial communication was used to send the input from the Arduino to Processing. For the physical part of the gun, I used Tinkercad to design the 3D model of the gun.tkcad

I made a hole in the trigger part to put the breadboard in it, but when I actually printed it out, the hole was smaller than I expected. As a result, I could not put the breadboard with the push button inside the trigger part, so I just attached it outside as the picture below. Also, in order to keep the physical part as simple as I can, I used INPUT_PULLUP in Arduino code and did no use resistor instead of just INPUT and using resistor.

g

<Arduino Code for Push Button>c

 

 

Unless the player is wearing purple shirt, tracking the gun worked very well as the following video.

Then I created different kinds of screen such as initial screen, setting screen, counting screen, and result screen, etc to make the game more organized. Before each round of the game starts, the user will be asked to set the color of the gun in the setting screen by clicking the gun on the web cam. Also, in case the gun is not properly tracked, user can always go back to the setting screen during the game by pressing the ‘s’ key on the keyboard.

Here is the demo of the game.

smogdemo

Doing the project, I could explore a lot of different things from openCV to 3D modeling. This project was a lot of coding, and I could figure out how to efficiently dealing with the codes spending time on debugging. However, there is also limitation that I could not improve loading speed of the game. It seemed like there are so many things happening in processing as many images are loaded and relocated every milli seconds. Moreover, it took at least 7 seconds after the game started for serial communication to work so that I can shoot. I should attempt using different functions in Processing to load images instead of PImage for speed up, and care more about the delicacy of the physical part.

 

import processing.serial.*;
import gab.opencv.*;
import processing.video.*;
import java.awt.Rectangle;

Serial myPort;
int val;

int cameraX, cameraY;
int screen = 0; //init 0, setting 1, game 2, counting 3, result 4, smog 10
int scene = 1; //night 1, day 2, twilight 3

PImage chSmog;

void setup() {
  size(1400, 800, P3D);
  setupSerial();
  color_setup();
  bg_setup();
  balloons_setup();
  scoring_setup();
  countingSetup();
  chSmog = loadImage("chsmog.png");
}

void draw() {
  //updateSerial();
  if (screen == 1 || screen == 2 ){
    colortrack();
  }
  if (screen == 0){
    initScreen();
  }else if (screen == 1){
    settingScreen();
  }else if (screen == 2){
    gameScreen(scene);
  }else if (screen == 3){
    countingScreen();
  }else if (screen == 4){
    resultScreen();
  }else if (screen == 10){
    smogScreen();
  }
}

void play(){
  screen = 2;
}

void smog_attack(){
  screen = 10;
}

void count_masks(){
  screen = 3;
}

void showResult(){
  screen = 4;
}

void restart(){
  level = 1;
  scene = 1;
  needs = 4;
  score = 0;
  seconds = 45;
  lastAddTime = 0;
  balloons.clear();
  screen = 1;
}

void mousePressed() {
  if (screen == 0){
    screen = 1;
  }else if (screen == 1){
    color c = get(mouseX, mouseY);
    println("r: " + red(c) + " g: " + green(c) + " b: " + blue(c));     
    int hue = int(map(hue(c), 0, 255, 0, 180));      
    colors[0] = c;
    hues[0] = hue;      
    println("color index " + (0) + ", value: " + hue);
    startMillis = millis()/1000;
    ready = startMillis + 19;
    sx = 0;
    play();
  }
}

void keyPressed() {
  if (screen == 0){
    if (key == 'r'){screen = 1;}
  }
  if (key == 's') {
    screen = 1; 
  }
}

//Balloons
PImage redb, greenb, pinkb;
PImage shoot;
ArrayList<int[]> balloons = new ArrayList<int[]>();
int interval = round(random(2000,4000));
float lastAddTime=0; 
boolean shooting = false;
int gx, gy;
float bSpeed = 1;

void balloons_setup(){
  redb = loadImage("redb.png");
  greenb = loadImage("greenb.png");
  pinkb = loadImage("pinkb.png");
  redb.resize(100,250);
  greenb.resize(130,250);
  pinkb.resize(125,250);
  shoot = loadImage("fire1.png");
  shoot.resize(390,280);
}

void Balloons(){
  balloonAdder();
  balloonHandler();
}

void balloonHandler(){
  for (int i = 0; i < balloons.size(); i++){
    balloonDrawer(i);
    balloonFloating(i);
    balloonRemover(i);   
  }
}

void balloonAdder(){
  if (millis()-lastAddTime > interval){
    int x = int(random(1600)) - 900;
    int y = 200 + int(random(400));
    int rcolor = int(random(3));
    int[] randB = {x,y,rcolor};
    balloons.add(randB);
    lastAddTime = millis();    
  }
}


void balloonDrawer(int index){
  int[] balloon = balloons.get(index);
  PImage bcolor;
  if (balloon[2] == 0){
    bcolor = redb;
  } else if (balloon[2] == 1){
    bcolor = greenb;
  } else{
    bcolor = pinkb;
  }
  image(bcolor, balloon[0], balloon[1]);
}

void balloonFloating(int index){
  int[] balloon = balloons.get(index);
  balloon[1] -= bSpeed;  
}

void balloonRemover(int index){
  gx = width/2-cameraX-150;
  gy = height/2-cameraY;
  shooting = shoot();
  int[] balloon = balloons.get(index);
  if (balloon[1] < -800){
    balloons.remove(index);
    score();
  } else if (shooting
        && (gx > balloon[0]-50 && gx < balloon[0]+50) &&
          (gy > balloon[1]-50 && gy < balloon[1]+50)){
     balloons.remove(index);
     score();
  }
}

boolean shoot(){
  updateSerial();
  if (val == 0 || mousePressed){
    image(shoot,width/2-cameraX-220,height/2-cameraY-20);
    return true;
  }
  return false;
}

void setupSerial(){
  String portName = Serial.list()[1];
  myPort = new Serial(this, portName, 9600);
}

void updateSerial(){
  if (myPort.available() >0){
    val = myPort.read();
  }
  println(val);
}

//Draw Background
PImage night;
PImage gun;
PImage day;
PImage twilight;
PImage title;
PImage shanghai;
PImage smoke;
PImage start;
PImage next;
PImage restart;

void bg_setup(){
  title = loadImage("smog.png");
  shanghai = loadImage("shanghai.png");
  smoke = loadImage("smoke.png");
  start = loadImage("start.png");
  next = loadImage("continue.png");
  restart = loadImage("restart.png");
  night = loadImage("night.jpg");
  gun = loadImage("gun1.png");
  gun.resize(550,450);
  day = loadImage("Shanghai-skyline-panoramic-water.jpg");
  twilight = loadImage("twilight.jpeg");
}

void outline_night(){
  fill(0);
  noStroke();
  rect(0-cameraX,0-cameraY,1400,50);
  rect(0-cameraX,750-cameraY,1400,50);
  rect(0-cameraX,0-cameraY,100,800);
  rect(1300-cameraX,0-cameraY,100,800);
}

void outline_day(){
  fill(0);
  noStroke();
  rect(0-cameraX,0-cameraY,1400,50);
  rect(0-cameraX,750-cameraY,1400,50);
  rect(0-cameraX,0-cameraY,20,800);
  rect(1380-cameraX,0-cameraY,20,800);
}

void bg_night(){
  translate(cameraX,cameraY);
  night.resize(2800,1600);
  image(night,-1300,-750);  
}

void bg_day(){
  translate(cameraX,cameraY);
  day.resize(2800,1600);
  image(day,-1380,-750); 
}

void bg_twilight(){
  translate(cameraX,cameraY);
  twilight.resize(2800,1600);
  image(twilight,-1380,-750);
}

//Scoring
PImage clock; 
PImage mask;
PImage timesup;
PFont font;
int score = 0;
float seconds;  
float startTime = 45;
float ready;
float startMillis = 0;
int timerWidth = 200;
int level = 1;
int needs = 4;

void scoring_setup(){
  clock = loadImage("clock.png");
  mask = loadImage("mask.gif");
  clock.resize(40,40);
  mask.resize(40,40);
  timesup = loadImage("timesup.png");
  timesup.resize(450,220);
  
}

void score(){
  score++;
}

void printScore(){
  image(mask, 120-cameraX, 5-cameraY);
  font = loadFont("YuppyTC-Regular-25.vlw");
  textFont(font,32);
  textAlign(CENTER);
  fill(255);
  textSize(25);
  text(score, 185 - cameraX, 35 - cameraY);
  text("Level : "+level, 580 - cameraX, 35 - cameraY);
}

void timer(){
  seconds = startTime + startMillis - millis()/1000;
  
  if (seconds<0){
    smog_attack();
  }
}

void smogAttack(){
  screen = 10;
}

void drawTimer() {
  image(clock, 260-cameraX, 5-cameraY);
  noStroke();
  fill(189, 195, 199);
  rectMode(CORNER);
  rect(305-cameraX, 25 - cameraY, timerWidth, 10);
  if (seconds > 15 && seconds < startTime + startMillis -5) {
    fill(46, 204, 113);
  } else if (seconds > 5) {
    fill(230, 126, 34);
  } else if (seconds >0){
    fill(231, 76, 60);
  }
  if (seconds < startTime + startMillis -10){
    rectMode(CORNER);
    rect(305-cameraX, 25 - cameraY, timerWidth*(seconds/startTime), 10);
  }
  if (ready < seconds) {
    textAlign(CENTER);
    fill(255);
    textSize(50);
    text("Please save people in Shanghai from smog!nCollect the masks by shooting balloons", width/2-cameraX, height/2-cameraY);
  }
}

void levelup(){
  level++;
  scene = level % 3;
  needs += 2;
  bSpeed += 0.3;
  score = 0;
  seconds = 45;
  lastAddTime = 0;
  balloons.clear();
  screen = 1;
}

//Screens
PImage maskOn;
PImage crying;
PImage ss;
PImage smogattack;
PImage overbg;
int s = 0;
int f = 0;
int count = 0;
int count2 = 0;
int sx = 100;


void initScreen(){
  image(shanghai,0,height-600);
  smoke.resize(1100,400);
  image(smoke, 200,0);
  title.resize(650,320);
  image(title,width/2-325,height/2-260);
  start.resize(200,100);
  image(start,100,400);
}

void settingScreen(){
  if (video.available()) {
      video.read();
  }
  chSmog.resize(450,335);
  image(chSmog,width/2 + 100,100);
  textAlign(CENTER);
  fill(0);
  textSize(30);
  text("Click the gun to be tracked", width/2, height/2 + 230);
}


void gameScreen(int scene){
  if (scene == 1){
    bg_night();
    Balloons();
    outline_night();
  }else if(scene == 2){
    bg_day();
    Balloons();
    outline_day();
  }else {
    bg_twilight();
    Balloons();
    outline_day();
  }
  image(gun,width/2-cameraX-150,height/2-cameraY);
  printScore();
  timer();
  drawTimer();
  textFont(font,32);
  textAlign(CENTER);
  fill(255);
  textSize(15);
  text("Press 's' to reset the gun", 100-cameraX, height-30-cameraY);
  
}

void smogScreen(){
  smogattack = loadImage("smogattack.png");
  //smogattack.resize(1000, 600);
  if (scene == 1){
    overbg = loadImage("night.jpg");
    outline_night();
  }else if(scene == 2){
    overbg = loadImage("Shanghai-skyline-panoramic-water.jpg");
    outline_day();
  }else {
    overbg = loadImage("twilight.jpeg");
    outline_day();
  }
  image(overbg,0,0);
  image(smogattack, sx, 100);
  image(timesup, width/2-200, height/2-100);
  sx += 25;
  if (sx > 150){
    count_masks();
  }
}

void countingSetup(){
  maskOn = loadImage("gotmask.png");
  crying = loadImage("crying.png");
  ss = loadImage("smog_shanghai.jpg");
  ss.resize(1400,800);
  maskOn.resize(180,190);
  crying.resize(125, 250);
}

void countingScreen(){
  s = score;
  if (score >= needs) {
    s = needs;
  }else{
    f = needs - score;
  }
  image(ss,0,0);
  whoGotMasks();
  if (count > s){
  noMasks();}
  if (count2 > f){
    count = 0;
    count2 = 0;
    screen = 4;
    delay(1000);
  } 
}

void whoGotMasks(){
  if (count <= s){
    for (int i = 0; i < count; i++){
      if(i<=5) {image(maskOn, i*180+100, 100);}
      else {image(maskOn, (i-6)*180+100, 400);}
    }
  }
  count++;
  delay(500);
}

void noMasks(){
  if (count2 < f){
    for (int i = 0; i < count2+1; i++){
      image(crying, i*125+100, 400);
    }
  }
  count2++;
  delay(300);
}

void resultScreen(){
  fill(0);
  rect(0,0,1400,800);
  textFont(font,32);
  textAlign(CENTER);
  fill(255);
  textSize(50);
  text("Level : "+level+"nMasks Collected : "+score, 700, 100);
  if(score >= needs){
    text("Congratulations!nYou saved everyone in Shanghai! nLEVEL UP!",700,350);
    image(next, 575, 540);
    if (mousePressed){levelup();}    
  }else{
    text("More masks needed.. nGAME OVER",700,350);
    restart.resize(240,80);
    image(restart, 575, 540);
    if (mousePressed){restart();}
  }
}

//Color Tracking

Capture video;
OpenCV opencv;
PImage src;
ArrayList<Contour> contours;

// <1> Set the range of Hue values for our filter
//ArrayList<Integer> colors;
int maxColors = 4;
int[] hues;
int[] colors;
int rangeWidth = 10;

PImage[] outputs;

void color_setup(){
  video = new Capture(this, 640, 480);
  opencv = new OpenCV(this, 640, 480);
  contours = new ArrayList<Contour>();
  
  // Array for detection colors
  colors = new int[maxColors];
  hues = new int[maxColors];
  
  outputs = new PImage[maxColors];
  
  video.start();
}

void colortrack(){
  background(150);
  
  if (video.available()) {
    video.read();
  }

  // <2> Load the new frame of our movie in to OpenCV
  opencv.loadImage(video);
  
  // Tell OpenCV to use color information
  opencv.useColor();
  src = opencv.getSnapshot();
  
  // <3> Tell OpenCV to work in HSV color space.
  opencv.useColor(HSB);
  
  detectColors();
  
  // Show images
  image(src, 0, 0);
  for (int i=0; i<outputs.length; i++) {
    if (outputs[i] != null) {
      image(outputs[i], width-src.width/4, i*src.height/4, src.width/4, src.height/4);
      
      noStroke();
      fill(colors[i]);
      rect(src.width, i*src.height/4, 30, src.height/4);
    }
  }
  
  // Print text if new color expected
  textSize(20);
  stroke(255);
  fill(255);
  
  displayContoursBoundingBoxes();
}

void detectColors() {
    
  for (int i=0; i<hues.length; i++) {
    
    if (hues[i] <= 0) continue;
    
    opencv.loadImage(src);
    opencv.useColor(HSB);
    
    // <4> Copy the Hue channel of our image into 
    //     the gray channel, which we process.
    opencv.setGray(opencv.getH().clone());
    
    int hueToDetect = hues[i];
    //println("index " + i + " - hue to detect: " + hueToDetect);
    
    // <5> Filter the image based on the range of 
    //     hue values that match the object we want to track.
    opencv.inRange(hueToDetect-rangeWidth/2, hueToDetect+rangeWidth/2);
    
    //opencv.dilate();
    opencv.erode();
    
    // TO DO:
    // Add here some image filtering to detect blobs better
    
    // <6> Save the processed image for reference.
    outputs[i] = opencv.getSnapshot();
  }
  
  // <7> Find contours in our range image.
  //     Passing 'true' sorts them by descending area.
  if (outputs[0] != null) {
    
    opencv.loadImage(outputs[0]);
    contours = opencv.findContours(true,true);
  }
}

void displayContoursBoundingBoxes() {
  
  for (int i=0; i<contours.size(); i++) {
    
    Contour contour = contours.get(i);
    Rectangle r = contour.getBoundingBox();
    
    if (r.width < 40 || r.height < 40)
      continue;
    
    //stroke(255, 0, 0);
    //fill(255, 0, 0, 150);
    //strokeWeight(2);
    //rect(r.x, r.y, r.width, r.height);
    cameraX = r.x*1400/640;
    if (r.y > 310){
      cameraY = height-310*800/280;
    }else{
    cameraY = height-r.y*800/280;
  }
  }
}

Nicholas Sanchez Ghost-o-meter

The project for this week was to create some apparatus that gauged and recorded a user’s fear. While there are many ways to do this (Computer vision via face tracking, eye tracking, pixel differentiation, etc), I elected to work with Galvanic skin response instead. To do this, I decided to modify the project that we did in class, but amend it so that that the item was more interesting and interactive.

My idea was to create a “Gohst-o-meter”, or a ghost detection device. While this device would not actually detect ghosts, it would gauge the user’s GSR and use this data to “detect” ghosts. The theory of Galvanic Skin Response suggests that when a human enters a state of arousal (experiencing strong emotions, such as fear), their sweat glands will produce sweat, which is conductive. Thus, the human’s skin’s natural resistance goes down as the sweat makes the skin more conductive. Devices, like the Arduino’s analog input pins, are sensitive to this change in resistance, and thereby can gauge galvanic skin response, albeit to a certain degree. And while the theory behindGalvanic Skin Response is far from an exact science, like the means of gauging GSR, it can still be an indicator of a user’s heightened sense of fear.

The “Ghost-o-meter” would essentially be a box with a handle. Inside the box would be the Arduino, with two extruding extending into the handle. These probes would be places such that both constantly made contact with the user’s hand, thereby allowing the user’s GSR to be captured. The box would have some physical indication of the GSR, like an LED or buzzer, that would change flashing/beeping frequency based on the user’s GSR input. While it would not necessarily detect ghosts, it would detect how scared the user was. Say, for example, the user is in a well lit, comfortable space, and not experiencing any strong emotions. After calibrating to their current GSR, the “Ghost-o-meter” would buzz and blink at high intervals. Now, if they were to enter a “haunted” dark and scary space, perhaps they would be a little scared, and their GSR would rise, and the intervals at which the”Ghost-o-meter” buzzed and flashed would decrease. As the machine is now buzzing and flashing more, the user thus infers that there are ghosts around. Ideally, this heightened state of arousal would cause the “Ghost-o-meter” to buzz and blink even more rapidly, therefore indicating the strong presence of a “ghost”.

To build this project, I used 3mm MDF board, some 3 mm clear Acrylic, a spray-painted 3.2mm PVC pipe, miscellaneous electronics, and of course, Arduino.

ghost1

I designed a box in Adobe illustrator to house the electronics. I then iterated through until I had a box that could fir the components and the handle. I then laser cut this box such that 5 sides were MDF, and the top was clear acrylic, so that a user could see the inside electronics. To make the handle, I took some PVC from another project and put conductive copper tape on both sides I then soldered one wire to each of these strips of tape. These would serve as the probes. I took these wires through the pipe’s insides and and plugged one wire into 5 volts on the Arduino, and the other to the Analog 0 pin. I also added some LEDs and a buzzer for good measure, to act as the feedback mechanisms.

ghost2

Finally, to code this I had several issues. Mainly that each person has a different initial or resting GSR. So i took some code that collected average data over a few second and spliced it into my code. This new code allowed for the “Ghost-o-meter” to calibrate to the user’s resing GSR by averaging values collected over the first 5 seconds. This would be the lower threshold by which the Arduino would compare the GSR. Ultimately, this allowed me to set the lowest value to that of the user’s natural GSR, so that any escalation would cause the “Ghost-o-meter” to start buzzing or blinking rapidly.

ghost3

ghost4

Unfortunately, the first round of testing showed that the probes were a little to sensitive. So, I added a semi-conductive foam around these probes to decrease its sensitivity. This worked well, but the sensitivity was still to high. What this did change, however, was grip. adding the foam changed the handle from a GSR detector to a pressure sensor. This meant that if the user gripped the handlebar tightly, the “Ghost-o-meter” would act as if a “ghost” appeared by buzzing and flashing rapidly. By contrast, a slight grip would not elicit this response. While this addition took away from the “Ghost-o-meter”‘s GSR reading, I think it could still gauge fear because often when people are scared, they grip items tighter. Therefore, in some sense this “Ghost-o-meter” does record fear, just not in the way I originally intended.

Here are a few (staged) videos of the Ghost-o-meter in action!

Final Project – Oliver Ruston

Final Project – Game Controller Shoes

Instructor: Dan

For my final project I attempted to fabricate a shoe. I did this through 3D printing and sewing fabric. The purpose of 3D printing this shoe was so that equipment like the Arduino and various sensors could be accommodated. Had a normal shoe been used, then there may not have been space for these things. This idea was inspired by a 3D printed shoe which was on display at Autodesk. Although the aesthetics of the end result may not have been as good as Autodesk’s example, the desired function was achieved. The initial function of this shoe was planned to be either to change colors or to record steps taken. The first idea would have taken up too much space. The second idea was plausible, but needed more brainstorming as to what it would do in Processing.

The steps one would take in this shoe ended up controlling a walking game. In processing, the game is very simple, It theoretically goes on forever and just shoes a sprite walking forward in correlation to the steps taken in the shoe. All that is required for this is a background and character. The character stays in the same position while the background moves which gives the illusion of movement. In order to control this game with the shoe, buttons were placed within the sole. Since the sole cannot consist entirely of buttons, they were placed in strategic spots where the most pressure is applied by the foot. The buttons were placed in a cardboard sheet which was then sewn between two pieces of fabric in the shape of the 3D printed shoe. Since the 3D printed shoe had a space to put the Arduino, the only wire which would come out of the shoe was the usb connecting the Arduino to the computer. In the future, this is planned to be taken out and replaced with a wireless form.

The code in Arduino reads the buttons and sends their status to Processing through serial:

int val;
void setup() {
Serial.begin(9600);
pinMode(2, INPUT_PULLUP);
}void loop() {
while (Serial.available()) {
val = Serial.read();
}if(digitalRead(2) == HIGH){
Serial.write(1);
}else{
Serial.write(0);
}}

 

20170516_145803

20170516_145808

20170516_145816

20170523_140918

20170523_140922

Background b;
Night n;
Clouds c;
Mountain1 m1;
Mountain2 m2;
//Mountain3 m3;
Ground g;
Tower t;
Sprite s;
int inByte = 0;

import processing.serial.*;
Serial myPort;

void setup() {
  size(1000, 600);
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[3], 9600);
  b = new Background();
  n = new Night();
  c = new Clouds();
  m1 = new Mountain1();
  m2 = new Mountain2();
  //m3 = new Mountain3();
  g = new Ground();
  t = new Tower();
  s = new Sprite();
}

void draw() {
  while (myPort.available() > 0) {
  inByte = myPort.read();
  //println(inByte);
  if (inByte == 0){
    //s.move();
    c.move();
    m1.move();
    m2.move();
  }
  }
  background(255);
  b.display();
  s.display();
  c.display();
  pushMatrix();
  translate(200, -24);
  c.display();
  translate(250, 80);
  c.display();
  translate(150, -20);
  c.display();
  translate(-100, -50);
  c.display();
  popMatrix();
  //n.display();
  //fill(150, 150, 150);
  //rect(130, 428, 50, 300, 50, 10, 0, 0);
  pushMatrix();
  translate(400, -165);
  scale(1.3);
  m1.display();
  popMatrix();
  pushMatrix();
  //translate(150, 0);
  //m3.display();
  pushMatrix();
  scale(1.1);
  translate(-240, 0);
  m2.display();
  popMatrix();
  m1.display();
  popMatrix();
  //t.display();
  g.display();
  s.display();
}

class Background {
  Background() {
  }
  void display() {
    fill(100, 200, 255);
    rectMode(CENTER);
    rect(500, 300, 1000, 600); //blue sky
    fill(255, 243, 0);
    ellipse(700, 40, 50, 50);  //sun
    
  }
}
class Sprite {
  Sprite() {
  }
  float x = 350;
  void display() {
    if (inByte == 0){
    fill(random(150), random(150), 250);
    ellipse (x, 536, 30, 30);
    ellipse (x, 514, 20, 20);
    ellipse (x, 498, 17, 17);
    fill(0);
    ellipse (x+4, 496, 3, 3);
    triangle(x+9, 496, x+9, 499, x+12, 497);
    }
    else {
    fill(0, 0, 150);
    ellipse (x, 536, 30, 30);
    ellipse (x, 514, 20, 20);
    ellipse (x, 498, 17, 17);
    fill(255);
    ellipse (x+4, 496, 5, 5);
    triangle(x+9, 496, x+9, 499, x+12, 497);
  }
  }
  void move() {
    x +=0.05;
  }
}
class Ground {
  Ground() {
  }
  void display() {
    fill(95, 63, 63);
    rect(500, 585, 1000, 35); //brown dirt
    fill(72, 155, 76);
    rect(500, 560, 1000, 15); //green grass
  }
}
class Clouds {
  Clouds() {
  }
  float x = 200.0;
  void display() {
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    pushMatrix();
    translate(500, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    translate(500, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    translate(500, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    translate(500, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    translate(600, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    translate(400, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    translate(1300, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    translate(900, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    translate(750, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    translate(500, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    translate(600, 0);
    fill(255);
    rect(x+30, 140, 110, 56);
    arc(x-20, 143, 50, 50, radians(100), radians(260));
    arc(x, 118, 50, 50, radians(180), radians(350));
    arc(x+54, 132, 80, 60, radians(220), radians(380));
    arc(x+70, 160, 75, 50, radians(298), radians(536));
    arc(x+30, 160, 70, 50, radians(390), radians(520));
    arc(x, 160, 50, 50, radians(420), radians(525));
    popMatrix();
  }
  void move() {
    x-=0.3;
  }
}
class Mountain1 {
  Mountain1() {
  }
  float x = 200;
  void display() {
    fill(125, 125, 125);
    arc(x-114, 553, 60, 480, radians(180), radians(270));
    fill(150, 150, 150);
    rect(x-100, 428, 30, 250, 10, 50, 0, 0);
    arc(x-110, 558, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x-115, 453, 20, 200, radians(180), radians(270));
    fill(150, 150, 150);
    arc(x-110, 458, 20, 200, radians(180), radians(270));
    arc(x-115, 453, 60, 200, radians(270), radians(300));
    arc(x-85, 553, 30, 375, radians(270), radians(360));
    arc(x-91, 500, 20, 100, radians(270), radians(353));
    arc(x-100, 485, 20, 200, radians(270), radians(353));
    arc(x-110, 553, 50, 100, radians(340), radians(360));
    arc(x-140, 553, 60, 100, radians(310), radians(360));
    pushMatrix();
    translate(1500, 0);
    fill(125, 125, 125);
    arc(x-114, 553, 60, 480, radians(180), radians(270));
    fill(150, 150, 150);
    rect(x-100, 428, 30, 250, 10, 50, 0, 0);
    arc(x-110, 558, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x-115, 453, 20, 200, radians(180), radians(270));
    fill(150, 150, 150);
    arc(x-110, 458, 20, 200, radians(180), radians(270));
    arc(x-115, 453, 60, 200, radians(270), radians(300));
    arc(x-85, 553, 30, 375, radians(270), radians(360));
    arc(x-91, 500, 20, 100, radians(270), radians(353));
    arc(x-100, 485, 20, 200, radians(270), radians(353));
    arc(x-110, 553, 50, 100, radians(340), radians(360));
    arc(x-140, 553, 60, 100, radians(310), radians(360));
    translate(500, 0);
    fill(125, 125, 125);
    arc(x-114, 553, 60, 480, radians(180), radians(270));
    fill(150, 150, 150);
    rect(x-100, 428, 30, 250, 10, 50, 0, 0);
    arc(x-110, 558, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x-115, 453, 20, 200, radians(180), radians(270));
    fill(150, 150, 150);
    arc(x-110, 458, 20, 200, radians(180), radians(270));
    arc(x-115, 453, 60, 200, radians(270), radians(300));
    arc(x-85, 553, 30, 375, radians(270), radians(360));
    arc(x-91, 500, 20, 100, radians(270), radians(353));
    arc(x-100, 485, 20, 200, radians(270), radians(353));
    arc(x-110, 553, 50, 100, radians(340), radians(360));
    arc(x-140, 553, 60, 100, radians(310), radians(360));
    translate(500, 0);
    fill(125, 125, 125);
    arc(x-114, 553, 60, 480, radians(180), radians(270));
    fill(150, 150, 150);
    rect(x-100, 428, 30, 250, 10, 50, 0, 0);
    arc(x-110, 558, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x-115, 453, 20, 200, radians(180), radians(270));
    fill(150, 150, 150);
    arc(x-110, 458, 20, 200, radians(180), radians(270));
    arc(x-115, 453, 60, 200, radians(270), radians(300));
    arc(x-85, 553, 30, 375, radians(270), radians(360));
    arc(x-91, 500, 20, 100, radians(270), radians(353));
    arc(x-100, 485, 20, 200, radians(270), radians(353));
    arc(x-110, 553, 50, 100, radians(340), radians(360));
    arc(x-140, 553, 60, 100, radians(310), radians(360));
    translate(1200, 0);
    fill(125, 125, 125);
    arc(x-114, 553, 60, 480, radians(180), radians(270));
    fill(150, 150, 150);
    rect(x-100, 428, 30, 250, 10, 50, 0, 0);
    arc(x-110, 558, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x-115, 453, 20, 200, radians(180), radians(270));
    fill(150, 150, 150);
    arc(x-110, 458, 20, 200, radians(180), radians(270));
    arc(x-115, 453, 60, 200, radians(270), radians(300));
    arc(x-85, 553, 30, 375, radians(270), radians(360));
    arc(x-91, 500, 20, 100, radians(270), radians(353));
    arc(x-100, 485, 20, 200, radians(270), radians(353));
    arc(x-110, 553, 50, 100, radians(340), radians(360));
    arc(x-140, 553, 60, 100, radians(310), radians(360));
    translate(500, 0);
    fill(125, 125, 125);
    arc(x-114, 553, 60, 480, radians(180), radians(270));
    fill(150, 150, 150);
    rect(x-100, 428, 30, 250, 10, 50, 0, 0);
    arc(x-110, 558, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x-115, 453, 20, 200, radians(180), radians(270));
    fill(150, 150, 150);
    arc(x-110, 458, 20, 200, radians(180), radians(270));
    arc(x-115, 453, 60, 200, radians(270), radians(300));
    arc(x-85, 553, 30, 375, radians(270), radians(360));
    arc(x-91, 500, 20, 100, radians(270), radians(353));
    arc(x-100, 485, 20, 200, radians(270), radians(353));
    arc(x-110, 553, 50, 100, radians(340), radians(360));
    arc(x-140, 553, 60, 100, radians(310), radians(360));
    translate(500, 0);
    fill(125, 125, 125);
    arc(x-114, 553, 60, 480, radians(180), radians(270));
    fill(150, 150, 150);
    rect(x-100, 428, 30, 250, 10, 50, 0, 0);
    arc(x-110, 558, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x-115, 453, 20, 200, radians(180), radians(270));
    fill(150, 150, 150);
    arc(x-110, 458, 20, 200, radians(180), radians(270));
    arc(x-115, 453, 60, 200, radians(270), radians(300));
    arc(x-85, 553, 30, 375, radians(270), radians(360));
    arc(x-91, 500, 20, 100, radians(270), radians(353));
    arc(x-100, 485, 20, 200, radians(270), radians(353));
    arc(x-110, 553, 50, 100, radians(340), radians(360));
    arc(x-140, 553, 60, 100, radians(310), radians(360));
    translate(500, 0);
    fill(125, 125, 125);
    arc(x-114, 553, 60, 480, radians(180), radians(270));
    fill(150, 150, 150);
    rect(x-100, 428, 30, 250, 10, 50, 0, 0);
    arc(x-110, 558, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x-115, 453, 20, 200, radians(180), radians(270));
    fill(150, 150, 150);
    arc(x-110, 458, 20, 200, radians(180), radians(270));
    arc(x-115, 453, 60, 200, radians(270), radians(300));
    arc(x-85, 553, 30, 375, radians(270), radians(360));
    arc(x-91, 500, 20, 100, radians(270), radians(353));
    arc(x-100, 485, 20, 200, radians(270), radians(353));
    arc(x-110, 553, 50, 100, radians(340), radians(360));
    arc(x-140, 553, 60, 100, radians(310), radians(360));
    translate(600, 0);
    fill(125, 125, 125);
    arc(x-114, 553, 60, 480, radians(180), radians(270));
    fill(150, 150, 150);
    rect(x-100, 428, 30, 250, 10, 50, 0, 0);
    arc(x-110, 558, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x-115, 453, 20, 200, radians(180), radians(270));
    fill(150, 150, 150);
    arc(x-110, 458, 20, 200, radians(180), radians(270));
    arc(x-115, 453, 60, 200, radians(270), radians(300));
    arc(x-85, 553, 30, 375, radians(270), radians(360));
    arc(x-91, 500, 20, 100, radians(270), radians(353));
    arc(x-100, 485, 20, 200, radians(270), radians(353));
    arc(x-110, 553, 50, 100, radians(340), radians(360));
    arc(x-140, 553, 60, 100, radians(310), radians(360));
    translate(900, 0);
    fill(125, 125, 125);
    arc(x-114, 553, 60, 480, radians(180), radians(270));
    fill(150, 150, 150);
    rect(x-100, 428, 30, 250, 10, 50, 0, 0);
    arc(x-110, 558, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x-115, 453, 20, 200, radians(180), radians(270));
    fill(150, 150, 150);
    arc(x-110, 458, 20, 200, radians(180), radians(270));
    arc(x-115, 453, 60, 200, radians(270), radians(300));
    arc(x-85, 553, 30, 375, radians(270), radians(360));
    arc(x-91, 500, 20, 100, radians(270), radians(353));
    arc(x-100, 485, 20, 200, radians(270), radians(353));
    arc(x-110, 553, 50, 100, radians(340), radians(360));
    arc(x-140, 553, 60, 100, radians(310), radians(360));
    popMatrix();
  }
    void move() {
    x-=0.3;
  }
}
class Mountain2 {
  Mountain2() {
  }
  float x = 200;
  void display() {
    fill(150, 150, 150);
    rect(x+245, 448, 30, 220, 0, 50, 0, 0);
    rect(x+210, 428, 60, 250, 50, 10, 0, 0);
    fill(125, 125, 125);
    arc(x+185, 553, 60, 470, radians(180), radians(262));
    fill(150, 150, 150);
    arc(x+190, 553, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x+195, 553, 50, 400, radians(180), radians(270));
    fill(125, 125, 125);
    rect(x+237, 398, 5, 120, 100, 0, 0, 0);
    fill(150, 150, 150);
    arc(x+195, 553, 40, 385, radians(180), radians(270));
    arc(x+195, 378, 50, 50, radians(270), radians(360));
    arc(x+235, 553, 60, 196, radians(268), radians(360));
    arc(x+205, 553, 20, 100, radians(290), radians(360));
    pushMatrix();
    translate(900, 0);
    fill(150, 150, 150);
    rect(x+245, 448, 30, 220, 0, 50, 0, 0);
    rect(x+210, 428, 60, 250, 50, 10, 0, 0);
    fill(125, 125, 125);
    arc(x+185, 553, 60, 470, radians(180), radians(262));
    fill(150, 150, 150);
    arc(x+190, 553, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x+195, 553, 50, 400, radians(180), radians(270));
    fill(125, 125, 125);
    rect(x+237, 398, 5, 120, 100, 0, 0, 0);
    fill(150, 150, 150);
    arc(x+195, 553, 40, 385, radians(180), radians(270));
    arc(x+195, 378, 50, 50, radians(270), radians(360));
    arc(x+235, 553, 60, 196, radians(268), radians(360));
    arc(x+205, 553, 20, 100, radians(290), radians(360));
    translate(500, 0);
    fill(150, 150, 150);
    rect(x+245, 448, 30, 220, 0, 50, 0, 0);
    rect(x+210, 428, 60, 250, 50, 10, 0, 0);
    fill(125, 125, 125);
    arc(x+185, 553, 60, 470, radians(180), radians(262));
    fill(150, 150, 150);
    arc(x+190, 553, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x+195, 553, 50, 400, radians(180), radians(270));
    fill(125, 125, 125);
    rect(x+237, 398, 5, 120, 100, 0, 0, 0);
    fill(150, 150, 150);
    arc(x+195, 553, 40, 385, radians(180), radians(270));
    arc(x+195, 378, 50, 50, radians(270), radians(360));
    arc(x+235, 553, 60, 196, radians(268), radians(360));
    arc(x+205, 553, 20, 100, radians(290), radians(360));
    translate(500, 0);
    fill(150, 150, 150);
    rect(x+245, 448, 30, 220, 0, 50, 0, 0);
    rect(x+210, 428, 60, 250, 50, 10, 0, 0);
    fill(125, 125, 125);
    arc(x+185, 553, 60, 470, radians(180), radians(262));
    fill(150, 150, 150);
    arc(x+190, 553, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x+195, 553, 50, 400, radians(180), radians(270));
    fill(125, 125, 125);
    rect(x+237, 398, 5, 120, 100, 0, 0, 0);
    fill(150, 150, 150);
    arc(x+195, 553, 40, 385, radians(180), radians(270));
    arc(x+195, 378, 50, 50, radians(270), radians(360));
    arc(x+235, 553, 60, 196, radians(268), radians(360));
    arc(x+205, 553, 20, 100, radians(290), radians(360));
    translate(500, 0);
    fill(150, 150, 150);
    rect(x+245, 448, 30, 220, 0, 50, 0, 0);
    rect(x+210, 428, 60, 250, 50, 10, 0, 0);
    fill(125, 125, 125);
    arc(x+185, 553, 60, 470, radians(180), radians(262));
    fill(150, 150, 150);
    arc(x+190, 553, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x+195, 553, 50, 400, radians(180), radians(270));
    fill(125, 125, 125);
    rect(x+237, 398, 5, 120, 100, 0, 0, 0);
    fill(150, 150, 150);
    arc(x+195, 553, 40, 385, radians(180), radians(270));
    arc(x+195, 378, 50, 50, radians(270), radians(360));
    arc(x+235, 553, 60, 196, radians(268), radians(360));
    arc(x+205, 553, 20, 100, radians(290), radians(360));
    translate(500, 0);
    fill(150, 150, 150);
    rect(x+245, 448, 30, 220, 0, 50, 0, 0);
    rect(x+210, 428, 60, 250, 50, 10, 0, 0);
    fill(125, 125, 125);
    arc(x+185, 553, 60, 470, radians(180), radians(262));
    fill(150, 150, 150);
    arc(x+190, 553, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x+195, 553, 50, 400, radians(180), radians(270));
    fill(125, 125, 125);
    rect(x+237, 398, 5, 120, 100, 0, 0, 0);
    fill(150, 150, 150);
    arc(x+195, 553, 40, 385, radians(180), radians(270));
    arc(x+195, 378, 50, 50, radians(270), radians(360));
    arc(x+235, 553, 60, 196, radians(268), radians(360));
    arc(x+205, 553, 20, 100, radians(290), radians(360));
    translate(500, 0);
    fill(150, 150, 150);
    rect(x+245, 448, 30, 220, 0, 50, 0, 0);
    rect(x+210, 428, 60, 250, 50, 10, 0, 0);
    fill(125, 125, 125);
    arc(x+185, 553, 60, 470, radians(180), radians(262));
    fill(150, 150, 150);
    arc(x+190, 553, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x+195, 553, 50, 400, radians(180), radians(270));
    fill(125, 125, 125);
    rect(x+237, 398, 5, 120, 100, 0, 0, 0);
    fill(150, 150, 150);
    arc(x+195, 553, 40, 385, radians(180), radians(270));
    arc(x+195, 378, 50, 50, radians(270), radians(360));
    arc(x+235, 553, 60, 196, radians(268), radians(360));
    arc(x+205, 553, 20, 100, radians(290), radians(360));
    translate(500, 0);
    fill(150, 150, 150);
    rect(x+245, 448, 30, 220, 0, 50, 0, 0);
    rect(x+210, 428, 60, 250, 50, 10, 0, 0);
    fill(125, 125, 125);
    arc(x+185, 553, 60, 470, radians(180), radians(262));
    fill(150, 150, 150);
    arc(x+190, 553, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x+195, 553, 50, 400, radians(180), radians(270));
    fill(125, 125, 125);
    rect(x+237, 398, 5, 120, 100, 0, 0, 0);
    fill(150, 150, 150);
    arc(x+195, 553, 40, 385, radians(180), radians(270));
    arc(x+195, 378, 50, 50, radians(270), radians(360));
    arc(x+235, 553, 60, 196, radians(268), radians(360));
    arc(x+205, 553, 20, 100, radians(290), radians(360));
    translate(500, 0);
    fill(150, 150, 150);
    rect(x+245, 448, 30, 220, 0, 50, 0, 0);
    rect(x+210, 428, 60, 250, 50, 10, 0, 0);
    fill(125, 125, 125);
    arc(x+185, 553, 60, 470, radians(180), radians(262));
    fill(150, 150, 150);
    arc(x+190, 553, 60, 480, radians(180), radians(262));
    fill(125, 125, 125);
    arc(x+195, 553, 50, 400, radians(180), radians(270));
    fill(125, 125, 125);
    rect(x+237, 398, 5, 120, 100, 0, 0, 0);
    fill(150, 150, 150);
    arc(x+195, 553, 40, 385, radians(180), radians(270));
    arc(x+195, 378, 50, 50, radians(270), radians(360));
    arc(x+235, 553, 60, 196, radians(268), radians(360));
    arc(x+205, 553, 20, 100, radians(290), radians(360));
    popMatrix();
  }
  void move() {
    x-=0.3;
  }
}
//class Mountain3 {
//  Mountain3() {
//  }
//  float x = 200;
//  void display(){
//    fill(150, 150, 150);
//    rect(x-70, 428, 50, 300, 50, 10, 0, 0);
//    popMatrix();
//  }
//  void move() {
//    x-=0.3;
//  }
//}
class Night {
  Night() {
  }
  void display() {
    fill(0);
    rectMode(CENTER);
    rect(500, 276, 1000, 550); //black sky
    fill(255, 180, 0);
    ellipse(100, 50, 50, 50); // moon
    fill(255);
    ellipse(450, 100, 5, 5); //stars {
    ellipse(500, 130, 5, 5);
    ellipse(550, 80, 5, 5);
    ellipse(300, 80, 5, 5);
    ellipse(350, 80, 5, 5);
    ellipse(375, 23, 5, 5);
    ellipse(80, 100, 5, 5);
    ellipse(430, 50, 5, 5);
    ellipse(390, 180, 5, 5);
    ellipse(250, 130, 5, 5);
    ellipse(800, 130, 5, 5);
    ellipse(950, 100, 5, 5);
    ellipse(200, 400, 5, 5);
    ellipse(500, 400, 5, 5);
    ellipse(150, 350, 5, 5);
    ellipse(50, 380, 5, 5);
    ellipse(800, 390, 5, 5);
    ellipse(900, 320, 5, 5);
    ellipse(850, 420, 5, 5);
    ellipse(460, 300, 5, 5);
    ellipse(210, 305, 5, 5);
    ellipse(700, 80, 5, 5); //}
    
  }
}
class Tower {
  Tower() {
  }
  void display() {
    fill(150, 150, 150);
    rect(500, 448, 30, 230, 10, 10, 0, 0);
    arc(391, 553, 15, 35, radians(180), radians(280));
    arc(610, 553, 15, 35, radians(260), radians(360));
    arc(405, 553, 30, 60, radians(180), radians(280));
    arc(596, 553, 30, 60, radians(260), radians(360));
    arc(425, 553, 50, 100, radians(180), radians(280));
    arc(576, 553, 50, 100, radians(260), radians(360));
    arc(445, 553, 60, 150, radians(180), radians(280));
    arc(556, 553, 60, 150, radians(260), radians(360));
    arc(465, 553, 60, 210, radians(180), radians(280));
    arc(536, 553, 60, 210, radians(260), radians(360));
    arc(485, 553, 60, 300, radians(180), radians(270));
    arc(516, 553, 60, 300, radians(270), radians(360));
  }
}

Final Project: catch me if you can

Date: 18 May 2017

Instructor: Dan

Partner: Jung Hwan Park

Purpose: Build an interactive project between reality and the virtual world.

Materials:

  • jumper cables
  • arduino
  • joystick
  • breadboard
  • USB cable
  • leap motion
  • processing
  • cardboard
  • tape

Procedure:

At first, for the in-class speed dating, I had come up with an idea. I wanted to build a 3D space and be able to pick and move the objects around in the space. Since I had learned about the leap motion from midterm, but was not able to use it, I decided to use it to complete my project. Since Jacob and I have worked together before, while we were discussing our ideas, we decided to work together on my idea and build around it. We wasn’t sure if it was possible to create the space, nor if we had the capabilities, but we worked with it.

After deciding to work together, we had to develop more on our project. Since I knew and felt that there wasn’t really a purpose behind my project, I searched and thought of different projects we could do. For the in-class presentation, I came up with 3D tetris; however, after giving it a thought, we understood that would have been too hard. So we decided to build the space first and then create a purpose once we had the base for the project.

First, we had an office hour with Moon to understand more about the leap motion. Moon used an example code and explained the more important parts for us. He build a code from scratch explaining line by line. Later, I had to go back and study the code again to understand it. In the code, we created if functions: if our hand was within the radius of the ball, the ball would fill red but if we grabbed on it, the ball would turn yellow.

Then, other than the leap motion, it was also very important to understand the idea of 3D space. I looked up an example of 3D space and tried understanding a little bit of it. Then, I went to Moon again to ask questions about it. The example 3D space I had found included the function cam(), but since Moon thought it was too complicating for us, he once again helped us build another code for 3D space without the function cam(). Instead, we used mapping, rotationX and rotationY.

I added the leap motion and the 3D space together to see how it looked like.

As we were working on leap motion and 3D space, Jacob and I decided we wanted to throw the ball in the space because it would add more to the interaction. However, when we asked Moon, he said it would be too challenging and that we could try doing it, but by taking steps. First, we had to have a bouncing ball in the 3D space. So he had us look up 2D bouncing ball and then he helped us with 3D bouncing ball in a space. Even the 2D was a little complicating because we didn’t really understand some of the functions, but we slowly started to understand and Moon helped us answer some questions we didn’t understand.

By then, we understood that throwing the ball would kind of be hard. He advised us to shoot a ball , so helped us shoot a ball that gradually decelerated with a keyPressed function.

Once we had all three basic codes, we decided to combine them together. I had combined 3D space and leap motion together previously, but combining all of them was complicating. However, I noticed that our leap motion was still in 2D and I tried very hard to change it into 3D. I later found out it was actually a lot more simple, because all I had to do was add the z-axis.

After changing it to 3D we still encountered a lot of troubles where our code wasn’t working properly. We had all the three codes together, but the bouncing ball and the ball we wanted to grab wouldn’t work together. Jacob and I kept trying to add them together separately, when he managed to make it work. I was very confused because, when we creating the three different codes, we actually created two names for the same function. Jacob understood that and was able to fix the code.

After combining everything, then we had to build our arduino side: the joystick. Jacob worked on the joystick and managed to get it working. Then, Jacob also worked on serial communication, while I created the design of the box. We struggled a little bit with the serial communication because the location of the ball and our hands would keep on shifting. We found out later that we had to add another mapping for it to work properly.

After the serial communication, we also had many other minor problems that we had to fix. First, since the sphere was bouncing based on the center, it seemed like the ball was sinking. So we made the ball stop sinking and in our mapping we added the radius of the ball which was 30 plus 1 to make sure the ball wouldn’t sink it. When we added the serial communication, our code started to blink. We found out that when we pressed the joystick, the value of the y-axis would also change, causing the 3D space to rotate every time we clicked. We added another mapping to prevent it from rotating when we pressed the joystick.

Conclusion

This project took Jacob and I a long time because we had to understand it and correct the problems we encountered on our project. However, as hard and long as it was, it was also a very interesting project and I learned a lot from it. At first, we didn’t know how to start our project because of our lack of experience, but towards the end, since we understood a lot more, we were able to solve the problems with our capabilities. We weren’t able to do many things because of our experience, but we were still able to create something based on what we already knew. What I most want to improve about this project is that I want to give it more purpose. I believe it is an interactive project, but as Antonius suggested, I need more of a purpose to make it better.

Thing to share / learned:

  1. Come with a proper idea and goal, don’t leave everything to last minute. Although we had an idea, we didn’t really have a purpose. We just added things as we thought would be fun.
  2. How to build a 3D space and have different coloured walls.
  3. 3D space can be very different from 2D, it can complicate a lot of things by just adding a z to it.
  4. Leap motion is very hard to use because it isn’t always too precise, also it was hard to control the z-axis by hand.
  5. so many different functions!!
CATCH ME IF YOU CAN
import de.voidplus.leapmotion.*;    //import from library
import processing.serial.*;

LeapMotion leap;    //declaring the object

String myString = null;
Serial myPort;

int NUM_OF_VALUES = 3;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;  /** this array stores values from Arduino **/

int worldSize = 400; 

//the center of the palm 
float handX;  
float handY;
float handZ;

//handGrab
float hGrab;

float sphX;   
float sphY;
float sphZ;
float sphRad;
color sphColor;

//velocity of the sphere
float velX;    
float velY;
float velZ;

int previousSensorValues;    

void setup () {
  size (600, 600, P3D);  //3D space
  background (255);

  setupSerial();

  leap = new LeapMotion(this);    //initializing the object

  sphX = 0;                       //x, y, z of the center of the circle
  sphY = 0;
  sphZ = 0;
  sphRad = 30;                    //radius of the sphere
  sphColor = color(255, 0, 0);    //color of the sphere
  sphereDetail(30);               //detail of the sphere

  //velocity at random with the range of -3 and 3
  velX=random (-3, 3);
  velY=random (-3, 3);
  velZ=random (-3, 3);
}

// update!
// check the value!
// and then display

void draw () {
  background (255);

  updateSerial();        //serial communication
  printArray(sensorValues);


  leapMotionUpdate();    //call methods of the object

  //center the hand
  float posX = map(handX, 0, width, -200, 200);
  float posY = map(handY, 0, height, -200, 200);
  float posZ = map(handZ, 80, 20, -150, 150);

  //println (hGrab);  //shows the value of 1.0 when I grab the circle

  float distance = dist(sphX, sphY, sphZ, posX, posY, posZ); //distance between sphere and hand
  //println(distance);
  if (distance < sphRad) {
    sphColor = color(0, 255, 0);   // if distance is smaller than radius then the circle turns green
    if (hGrab > 0.8) {  //open hand = 0, closed hand = 1
      sphColor = color(255, 255, 0);  //if you grab the button, then its gonna turn yellow
      sphX = posX;  //position of sphere = position of hand
      sphY = posY;
      sphZ = posZ;
    }
  } else {
    sphColor = color(0, 0, 255);  //color it blue
  }

  //sphere increases velocity on X, Y, and Z
  sphX= sphX+velX;
  sphY= sphY+velY;
  sphZ= sphZ+velZ;

  //X - cannot move outside of side walls
  float area = sphRad + 1;
  if (sphX > (worldSize/2)-area) {
    sphX = worldSize/2-area;
    velX = -velX;
  } else if (sphX<(-worldSize/2)+area) {
    sphX = -worldSize/2+area;
    velX = velX*-1;
  }
  
  //Y - cannot move outside of the ceiling and floor
  if (sphY > (worldSize/2)-area) {
    sphY = worldSize/2-area;
    velY = -velY;
  } else if (sphY<(-worldSize/2)+area) {
    sphY = -worldSize/2+area;
    velY=velY*-1;
  }
  
  //Z - cannot move outside the back and front
  if (sphZ > (worldSize/2)-area) {
    sphZ = worldSize/2-area;
    velZ = -velZ;
  } else if (sphZ<(-worldSize/2)+area) {
    sphZ = -worldSize/2+area;
    velZ = velZ*-1;
  }

  //speed slowly slowing down
  velX *= 0.99;
  velY *= 0.99;
  velZ *= 0.99;

  //translate the world to the middle of the space
  translate(width/2, height/2);   

  // to rotate on x-axis and y-axis
  float rotAngleY = map(sensorValues[0], 261, 762, -PI/4, PI/4);
  float rotAngleX = map(sensorValues[1], 259, 763, PI/4, -PI/4);
  
  //instead of joytick, use mouseX and mouseY to look at the space
  //float rotAngleY = map(mouseX, 0, width, -PI/4, PI/4);
  //float rotAngleX = map(mouseY, 0, height, PI/4, -PI/4);

  //since the values are constantly changing, to prevent the world from moving around by itself
  if (sensorValues[1]>=763) {
    rotAngleX = map (sensorValues[1], 259, 763, 0, 0);
  }
  if ((sensorValues[0]>506)||(sensorValues[0]<499)) {
    rotateY(rotAngleY);
  }
  if ((sensorValues[1]>511)||(sensorValues[1]<499)) {
    rotateX(rotAngleX);
  }

  //world
  noFill();
  noStroke();
  drawBox(0, 0, 0, 400, 400, 400);    

  //side walls
  fill(50);
  noStroke();
  drawBox(200, 0, 0, 2, 400, 400);
  drawBox(-200, 0, 0, 2, 400, 400);

  //background
  fill(155);
  noStroke();
  drawBox(0, 0, -200, 400, 400, 2);  

  //ceiling and floor
  fill(220);
  drawBox(0, 200, 0, 400, 2, 400);
  drawBox(0, -200, 0, 400, 2, 400);

  //sphere that follows the position of hand
  noStroke();    
  fill(0);
  drawSphere (posX, posY, posZ, 10);

  //draw the sphere
  stroke(sphColor);
  noFill();
  drawSphere (sphX, sphY, sphZ, 30);
}

//function to draw multiple spheres
void drawSphere (float x, float y, float z, int size) {
  pushMatrix ();
  translate (x, y, z);
  sphere(size);
  popMatrix();
}

//function to draw multiple boxes
void drawBox(float x, float y, float z, float w, float h, float d) {
  pushMatrix();
  translate(x, y, z); // this way you can change the position of the box
  box(w, h, d); // you can only change the size of the box (w,h,d)
  popMatrix();
}

//if a key on the keyboard is pressed, then the sphere will randomly change velocity
void keyPressed() {
  velX=random (-15, 15);
  velY=random (-15, 15);
  velZ=random (-15, 15);
}

//serial communication
void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 2 ], 9600);
  // WARNING!
  // You will definitely get an error here.
  // Change the PORT_INDEX to 0 and try running it again.
  // And then, check the list of the ports,
  // find the port "/dev/cu.usbmodem----" or "/dev/tty.usbmodem----" 
  // and replace PORT_INDEX above with the index number of the port.

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = 'n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}

void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = 'n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
  if (sensorValues[2] == 1) {    //whenever the value=1, the sphere will change velocity
    velX=random (-15, 15);
    velY=random (-15, 15);
    velZ=random (-15, 15);
    sensorValues[2] = previousSensorValues;
  }
  if (previousSensorValues == 1) {    //when the previous value=1, the next value has to equal to 0
    sensorValues[2] = 0;
  }
}

LEAP MOTION
// ======================================================
// Table of Contents:
// ├─ 1. Callbacks
// ├─ 2. Hand
// ├─ 3. Arms
// ├─ 4. Fingers
// ├─ 5. Bones
// ├─ 6. Tools
// └─ 7. Devices
// ======================================================

// ======================================================
// 1. Callbacks

void leapOnInit() {
  // println("Leap Motion Init");
}
void leapOnConnect() {
  // println("Leap Motion Connect");
}
void leapOnFrame() {
  // println("Leap Motion Frame");
}
void leapOnDisconnect() {
  // println("Leap Motion Disconnect");
}
void leapOnExit() {
  // println("Leap Motion Exit");
}


void leapMotionUpdate () {
  // ...

  int fps = leap.getFrameRate();
  for (Hand hand : leap.getHands ()) {


    // ==================================================
    // 2. Hand

    int     handId             = hand.getId();
    PVector handPosition       = hand.getPosition();
    PVector handStabilized     = hand.getStabilizedPosition();
    PVector handDirection      = hand.getDirection();
    PVector handDynamics       = hand.getDynamics();
    float   handRoll           = hand.getRoll();
    float   handPitch          = hand.getPitch();
    float   handYaw            = hand.getYaw();
    boolean handIsLeft         = hand.isLeft();
    boolean handIsRight        = hand.isRight();
    float   handGrab           = hand.getGrabStrength();
    float   handPinch          = hand.getPinchStrength();
    float   handTime           = hand.getTimeVisible();
    PVector spherePosition     = hand.getSpherePosition();
    float   sphereRadius       = hand.getSphereRadius();

    handX = handPosition.x;
    handY = handPosition.y;
    handZ = handPosition.z;
    hGrab = handGrab;  //renaming it because there is another handGrab and they would interfere with each other

    // --------------------------------------------------
    // Drawing
    //hand.draw(); //draws out the hand


    // ==================================================
    // 3. Arm

    if (hand.hasArm()) {
      Arm     arm              = hand.getArm();
      float   armWidth         = arm.getWidth();
      PVector armWristPos      = arm.getWristPosition();
      PVector armElbowPos      = arm.getElbowPosition();
    }

    // ==================================================
    // 4. Finger

    Finger  fingerThumb        = hand.getThumb();
    // or                        hand.getFinger("thumb");
    // or                        hand.getFinger(0);

    Finger  fingerIndex        = hand.getIndexFinger();
    // or                        hand.getFinger("index");
    // or                        hand.getFinger(1);

    Finger  fingerMiddle       = hand.getMiddleFinger();
    // or                        hand.getFinger("middle");
    // or                        hand.getFinger(2);

    Finger  fingerRing         = hand.getRingFinger();
    // or                        hand.getFinger("ring");
    // or                        hand.getFinger(3);

    Finger  fingerPink         = hand.getPinkyFinger();
    // or                        hand.getFinger("pinky");
    // or                        hand.getFinger(4);

    for (Finger finger : hand.getFingers()) {
      // or              hand.getOutstretchedFingers();
      // or              hand.getOutstretchedFingersByAngle();

      int     fingerId         = finger.getId();
      PVector fingerPosition   = finger.getPosition();
      PVector fingerStabilized = finger.getStabilizedPosition();
      PVector fingerVelocity   = finger.getVelocity();
      PVector fingerDirection  = finger.getDirection();
      float   fingerTime       = finger.getTimeVisible();

      // ------------------------------------------------
      // Drawing

      // Drawing:
      // finger.draw();  // Executes drawBones() and drawJoints()
      // finger.drawBones();
      // finger.drawJoints();

      // ------------------------------------------------
      // Selection

      switch(finger.getType()) {
      case 0:
        // System.out.println("thumb");
        break;
      case 1:
        // System.out.println("index");
        break;
      case 2:
        // System.out.println("middle");
        break;
      case 3:
        // System.out.println("ring");
        break;
      case 4:
        // System.out.println("pinky");
        break;
      }


      // ================================================
      // 5. Bones
      // --------
      // https://developer.leapmotion.com/documentation/java/devguide/Leap_Overview.html#Layer_1

      Bone    boneDistal       = finger.getDistalBone();
      // or                      finger.get("distal");
      // or                      finger.getBone(0);

      Bone    boneIntermediate = finger.getIntermediateBone();
      // or                      finger.get("intermediate");
      // or                      finger.getBone(1);

      Bone    boneProximal     = finger.getProximalBone();
      // or                      finger.get("proximal");
      // or                      finger.getBone(2);

      Bone    boneMetacarpal   = finger.getMetacarpalBone();
      // or                      finger.get("metacarpal");
      // or                      finger.getBone(3);

      // ------------------------------------------------
      // Touch emulation

      int     touchZone        = finger.getTouchZone();
      float   touchDistance    = finger.getTouchDistance();

      switch(touchZone) {
      case -1: // None
        break;
      case 0: // Hovering
        // println("Hovering (#" + fingerId + "): " + touchDistance);
        break;
      case 1: // Touching
        // println("Touching (#" + fingerId + ")");
        break;
      }
    }


    // ==================================================
    // 6. Tools

    for (Tool tool : hand.getTools()) {
      int     toolId           = tool.getId();
      PVector toolPosition     = tool.getPosition();
      PVector toolStabilized   = tool.getStabilizedPosition();
      PVector toolVelocity     = tool.getVelocity();
      PVector toolDirection    = tool.getDirection();
      float   toolTime         = tool.getTimeVisible();

      // ------------------------------------------------
      // Drawing:
      // tool.draw();

      // ------------------------------------------------
      // Touch emulation

      int     touchZone        = tool.getTouchZone();
      float   touchDistance    = tool.getTouchDistance();

      switch(touchZone) {
      case -1: // None
        break;
      case 0: // Hovering
        // println("Hovering (#" + toolId + "): " + touchDistance);
        break;
      case 1: // Touching
        // println("Touching (#" + toolId + ")");
        break;
      }
    }
  }


  // ====================================================
  // 7. Devices

  for (Device device : leap.getDevices()) {
    float deviceHorizontalViewAngle = device.getHorizontalViewAngle();
    float deviceVericalViewAngle = device.getVerticalViewAngle();
    float deviceRange = device.getRange();
  }
}

ARDUINO
/*
  Thumb Joystick demo v1.0
  by:http://www.seeedstudio.com
  connect the module to A0&A1 for using;
*/
int buttonPin = 0;

void setup() {
  Serial.begin(9600);
}

void loop() {

  int pot1 = analogRead(A0);
  int pot2 = analogRead(A1);

  if (pot2 == 1023) {
    buttonPin = 1;
  } else {
    buttonPin = 0;
  }

  Serial.print(pot1);
  Serial.print(",");
  Serial.print(pot2);
  Serial.print(",");
  Serial.print(buttonPin);
  Serial.println();

  delay(100);

}

Mixed Reality Interview: Maggie, Shawn, Daniela, and Ale

For our mixed reality interview we decided to do a talk show between an Abu Dhabi Student and an NYU Shanghai student. We pretty much improvised our script, but we used Ideascale to compile a list of complaints that Abu Dhabi students had actually made in the past. Then we used those complaints to formulate the material for our storyboard.

Both Ale, and I (Maggie) 3D Scanned ourselves with the Skanect with the help of Kyle. We came to class with our bodies rigged and ready to go.

We wanted to use the actual ideascale page as the background of our interview, and be able to scroll through all of the material, but instead we ended up using a static image.

Here are a few shots of what we ended up with.

Screen Shot 2017-05-21 at 8.50.10 PM

 

 

Screen Shot 2017-05-21 at 8.51.18 PM

 

Screen Shot 2017-05-21 at 8.50.41 PM

 

Bomb Diffusing! – Interaction Lab 12 Documentation

Date: May 5th, 2017

Instructor: Marcela

Partner: Sjur

Material’s Used:

  • Arduino
  • Jumper Cables
  • Breadboard
  • USB Cable
  • 1x LED
  • 2x 10K Resistors
  • 1x Wire Cutters

For this lab we were assigned with using both Arduino and Processing to create a media controller. My partner Sjur and I decided to stray away and create something a little different than just a basic media controller. We decided to create a very simple bomb diffusing simulation. The circuitry is displayed below:

The Bomb!

IMG_20170505_120137

More jumper cables for added confusion!

IMG_20170505_122756

With our very limited time, the concept of the game was that the user would cut different wires on the breadboard until the bomb exploded. Antonius helped us create a simple physical switch with jumper cables and a resistor. So if the jumper cable was disconnected from the breadboard, or cut, the bomb would be activated. Only one of the jumper cables on our breadboard actually activated the bomb, but we put multiple other different jumper cables on there as decoys to the real thing.

When the bomb went off the processing would play a video I found and quickly edited using Adobe Premiere. In order to make it more realistic, we used a real wire cutter.

IMG_20170505_123734

Our lives are in your hands, Sjur…

Final Project Reflection

As for my final project, things almost go as I first planned. In general, the only thing that I am not satisfied with is the size of the container, if I were to continue on my project, I would definitely just  make a much larger one.

I first bought a fish bowl and I 3D print the cover of the whole project. The idea behind the cover is just to make the cover being able to connect two bowl as well as having space for grow my “small aquaponic mesh pots.” There goes the inner part.

At first, I was thinking about buying a small glass container on Taobao, which serves the function of holding fruits, but later, I realized that the glass jar is simply too heavy and also it is also hard to find a glass jar that is the right size compared with plastic bowl whose use is more widely. Then, I thought about the take-out container that we always have from Eleme. 54

I measured the size of the glass bowl and, the above us just right to be put on the edge of the glass  fish container. The thing that I am not quite satisfied here is the size of the hole of the cover. But this is hard to improve unless I have a really large fruit container in the first place.

Then I started to think about the plants that I would like to have. Since the initial idea behind my project is that I’d love to see how living plants works with ripe fruits. I would imagine the plants that I have here look good, with flowers being great. I would also image the plants that I grow could have interactions with my fruits. There come two ideas that I think is interesting. I was wondering whether it is possible that by surrounding ripe fruits with certain plants to slower the process of the ripe fruits going rot.  Also, I personally would be quite bothered by the flies that around the fruit bowl in summer time, and therefore, I was thinking whether it is possible to grow certain kinds of plants that could release certain gas that would chase away the flies. And apart from the idea of interaction itself, the plants also need to be able to fit into the aquaponic system. I didn’t find any plants that could fulfill the idea of slower the ripe process as well as aquaponic friendly. However, I am lucky enough to find this Mozzie Buster. This particular plant could largely reduce the flies around the area and with the temperature goes higher, it works better by releasing more gas. It smells good to me, as a human being.

3

And, what’s better is that this Mozzie Buster likes water and being in a aqua environment. The reason that I am not using this in my final project is because this plant is too large that I could not fit it into my mesh pots. But I would definitely experiment more with that if I were to have a huge container. Moreover, I also experiments with plants that look good and likes the aqua environments.

I bought some Pennywort online. They look cute and they have good meanings behind, and they are easy to grow in aqua environment. When I get the Pennywort, they were just roots in soil, and I took them out, watered them gently. And then I put some rocks onto the container, then the Pennywort roots, and then covered the roots with more rocks. Then fill the container with water to submerge the roots in water thoroughly.

9 7 6 1

I changed the water once two days, because the Pennywort roots in this stage of their lives are still quite weak, that it might not being able to survive it with frequent water change. I am happy with how the Pennywort growing. They are one of the perfect plants for my project, because they are small and cute.

I also tried two water lilies. I didn’t know water lilies have mini species. They grow slow, therefore, by the day I present, they were still in the sprouting stage. unnamed

I also testes with the water plants in the Lab’s fish tank system, but they didn’t do well, as you could see in my final project. 10

I should have some fish swimming in the bowl, but after my research, I got to know that even one fish would love to have 1 L of water, but my container is just too small to make the fish happy, and therefore I didn’t borrow fish from Dan.

Even at this point of my project, I still quite love the idea to have fishes swimming in the fruit bowl. It is lively and interesting just to look at.