Week 1: Response to E.M. Forster (Moon)

In the story “The Machine Stops” is a story in the future that people live under the control of a Big Machine, and get used to the life that they don’t need to work, don’t need to move and even don’t need to work. They communicate with each other through some “magic plates”, and only get what the machine want them to get from phrases. The story makes me to rethink the relationship between human and machine. Nowadays most  our concerns about technology development is that the robots may take over our jobs and more and more people will face the risk of losing jobs. However, this story reminds me that how to keep the independence is more vital.

In GPS class, we have learned Descartes’ idea “cogito ergo sum” that only when we doubt on something, we can make sure it exists actually. Though someone may argue that his opinion is too idealistic, but this does has some significants in the international decade. In the story, because every room around the world is same and the communication tools are convenient, most people underground are like Vashti that they don’t go out if not necessary, the result of which is people lose the sense of space and distance. We need to feel the world by touching it, and one problem coming form technology development is when we get used to relying on machine, we will lose the chance to realize the world, and at that time, by Descartes’ opinion, the world doesn’t exist for us anymore.

What is worse, with the development of AI (Artificial intelligence) we can find that robots will definitely pass the Turing test in the future, which means that they can think as a human. And if we are used to relying on machine, it’s likely that we will also rely on machine in thinking at that time. In the story the “leader” says that people should refuse “first-hand” idea and only receive the idea that the machine allows. In my opinion, after giving up the ability to think, we are the same as machines in this generation, in other words, there no significants for us to exist on the earth.

Therefore, I think during the development of technology, what we need to do is keep our independence and make sure it’s we that control the machine rather than being controlled by machine.

Interaction Lab(Moon) Final Project: Magic Ink Brush- Hanlu Zhang(Mary)

Final Project: Magic Ink Brush

  • Date: December 15th, 2017
  • Instructor: Moon
  • Introduction:

 

Magic Ink Brush is a project that users can draw items in Chinese painting with effects by moving the brush. Through this project users can experience being a maser of Chinese painting and feel the beauty of traditional Chinese art.

My inspiration has two parts, one is a Chinese Cartoon called “Incredible Ink”, in Chinese is “Shenbimaliang”, the other one is traditional Chinese Brush painting. ( more detailed explanation is in my Final idea post http://ima.nyu.sh/documentation/wp-admin/post.php?post=125568&action=edit)

  • Development:

At the beginning, I want to use accelerometer sensor to detect the movement of brush. At the beginning, I have no idea of values from accelerometer, with Professor Moon’s suggestion, I send the original values from Arduino to Processing, then map the six value x, y, z, ax, ay, az with the length of rectangle showing on the screen. By this method I can see the change of value directly. However, I found that the value of accelerometer sensor are not that stable, and the way to place the sensor also influence the value because of gravity. With the consideration that user may hold the brush in different ways so that it will be hard to map the movement and the effects on screen. What’s more, wires connected on the brush will limit users’ movements. Because of these considerations, Though I had tested the accelerometer for half a week, I decided to change the way of detection to color-tracking.

The following problem is without accelerometer, what to put on Arduino part. At the beginning, I want to have a board with four seasons, when user put the brush on one season (mimicking the gesture of dipping in Ink) , processing will switch to that mode. Then Moon suggested me that I can connected the dipping with choosing what to draw on screen, or the Arduino part may be too weak compared with processing. Therefore I form my final idea, and below are detailed information.

  • Arduino Part

Arduino part is to control what user can draw on the screen. For the control board,  I used Adobe illustrator design it and did laser cutting. I use two material cardboard and MDF. Cardboard are better in sense of layer a because different cutting depth will show different color, but the images are not obvious enough, and MDF is on the contrary. Consider of the user experience, I choose the MDF one.

For the circuit part, to make the brush wireless, I borrow the structure of switch that there will be a gap between construction tapes on control board, and the brush’s nib will also have tapes. So when brush connect the gap, circuit close and the value from Arduino will send the values read by the pins to processing.

Finally the Arduino part looks like this:

  • Processing Part

The first thing is color tracking, first I did the color tracking based on the sample of openCV, but the frameRate was to low when using that code, then Jiwon gave me a code wrote by moon before that can realize color-tracking with higher speed. After changing to this code, the frameRate was still very low, and Moon told me that this was because the background picture was too large and it will be better if I use a lower quality of image.

Second thing is to detect the change of brush’s position. Moon suggested me that I can detect the posX/posY first, and compare them with pPosX/pPosY, then let the pPosX/pPosY=posX/posY, and I can find the change of position. Also because the movement in camera is opposite with the real movement, I also use map() function to map the movements.

To make the rate of drawing not too high, I write that

 So it will detect the value for every 20 frames.

The third thing is the effects on screen, to make the sense of Chinese printing, I put a translucent rectangle on the whole screen so that the everything moving on the screen will leave some trace like ink water. Items in my project has four kinds: bird, butterfly, fish, and weather, most effects are made by functions sin(), cos(), noise() and rotate(). I also use some resources form openProcessing, for example the movement of fish, and the heart formed by moving butterfly is modified based on a resources. To make the connection between my project and Chinese painting stronger, the items I chose all have some special meanings in Chinese culture. For example, butterfly represent love because the story “Butterfly Lovers”, and magpies and fishes are both the symbol of  luck.

I created classes for each effects, and when to make them appear on the screen, if I only add things with out removed, my laptop may breakdown because datas are super big. With the help of Jack, I create arrayList for each class, and when I don’t need the item, I can remove them form the list, and also I don’t need to limit the amount of each item.

The forth thing is switching modes. As I hope that when the brush connected the gap, the mode will change, I void boolean value that when the value from Arduino is 1, boolean of that mode become true and mode change, when next change happen, all the boolean value are reset to false. And here’s a problem that as I hope when the brush is on the board, the mode is 0 and when I take it up it can change to mode 1, then I choose the item, mode change with that. But when I take up the brush, the value from that pin will always be 0 so that the mode can’t change to others. So Jack suggested me that for this mode, I can test the value in both current and previous frame and the mode will change only the two value is different. The code is below:

  • User tests

Because of time limitation, user tests are most done in the End of Semester Show

Feedbacks from users are:

  1. The green paper on the brush looks strange and will affect people’s understand, as some people thinks it’s an umbrella rather than a brush.
  2. The place to switch the mode is too small and it’s not easy for some people to understand where to dip the brush.
  3. Some users think that it will make sense if the drawing is on horizontal plane.
  4. Some users don’t know when they are drawing , where should they put the brush so that the computer can detect it.
  • Future development
  1. I can use 3d print to create the green part and make it more related to brush so it will looks better
  2. I can give more effective instructions to users so they can understand how to use this (maybe image or video demo will be better?).
  3. I can change the drawing to the horizontal plane and has a camera on the top to detect the color
  4. I can develop the Arduino part like hide all the wires in the box and make the switches more obvious
  • Courses learned
  1. Combination of class and arraylist
  2. Use of variables and comparing the current value and previous value.
  3. How to use color-tracking and the different kinds of color-tracking.
  4. Choose the proper ways and devices base on the need.
  5. How to consider users’ experience.
  • Special thanks
  • Thanks professor Moon for helping me a lot in idea development and solving my problems during the whole process of making my project, as well as helping me keep the brush that I left in lab.
  • Thanks Jiwon and Jack who help me a lot in debug and doing laser cut.
  • Thanks all the fellows that offer help in all aspects during the process.
  • Thanks my roomie Demi for being my first user and give me many suggestions
  • Thanks all my classmates, it’s my pleasure to study with so many excellent and creative people.
//Processing

import processing.sound.*;
import processing.video.*;
import processing.serial.*;

Capture cam;
color pickedColor;

String myString = null;
Serial myPort;

int NUM_OF_VALUES = 5;   /** YOU MUST CHANGE THIS ACCORDING TO YOUR PROJECT **/
int[] sensorValues;      /** this array stores values from Arduino **/


int tolerance = 25;
float posX = 0;
float posY = 0;
float pPosX = 0;
float pPosY = 0;
float accX = 0;
float accY = 0;
int psensor = 1;
int csensor;

int mode = 0;//mode

boolean sensor=false;
boolean sensor0=false;
boolean sensor1 =false;//bird
boolean sensor2 =false;//butterfly
boolean sensor3 =false;//fish
boolean sensor4 =false;//weather
boolean birdPlay = false;
boolean butterPlay = false;
boolean fishPlay = false;
boolean weatherPlay = false;

PImage background;
PImage swallowP;
PImage petal;
PImage duck;
PImage bird;
PImage fishTwo;
PImage butterfly1;
PImage butterfly2;
PImage butterfly3;
PImage cloud;
PImage fog;
PImage title;
PImage title2;
PImage instruction;

SoundFile fishS;
SoundFile butterflyS;
SoundFile birdS;
SoundFile weatherS;


ArrayList<PetalGroup> petals = new ArrayList<PetalGroup>();
ArrayList<Duck> ducks = new ArrayList<Duck>();
ArrayList<Fog> fogs = new ArrayList<Fog>();
ArrayList<Swallow> swallows = new ArrayList<Swallow>();
ArrayList<Fish> fish = new ArrayList<Fish>();
ArrayList<Bird> birds = new ArrayList<Bird>();
ArrayList<Fish2> fish2 = new ArrayList<Fish2>();
ArrayList<ButterflyU> butterflyU = new ArrayList<ButterflyU>();
ArrayList<ButterflyL> butterflyL = new ArrayList<ButterflyL>();
ArrayList<ButterflyR> butterflyR = new ArrayList<ButterflyR>();
ArrayList<Cloud> clouds = new ArrayList<Cloud>();

void setup() {
  size(1280, 720);
  cam = new Capture(this, 320, 240);
  cam.start();
  pickedColor = color(121, 180, 71);
  setupSerial();

  //load picture
  background = loadImage("background.png");
  swallowP = loadImage("swallow.png");
  petal = loadImage("petal.png");
  duck = loadImage("duck.png");
  bird = loadImage("bird.png");
  fishTwo = loadImage("fish2.png");
  butterfly1 = loadImage("butterfly1.png");
  butterfly2 = loadImage("butterfly2.png");
  butterfly3 = loadImage("butterfly3.png");
  cloud = loadImage("cloud.png");
  fog = loadImage("fog.png");
  title = loadImage("title.png");
  title2 = loadImage("title2.png");
  instruction = loadImage("instruction.png");
  // butterflyL.add(new ButterflyL(width / 2, height / 2));

  birdS = new SoundFile (this, "bird.mp3");
  weatherS = new SoundFile (this, "weather.mp3");
  fishS = new SoundFile (this, "fish.mp3");
  butterflyS= new SoundFile (this, "butterfly.mp3");
}


void draw() {

  accX = 0;
  accY = 0;
  updateSerial();
  printArray(sensorValues);
  csensor = sensorValues[0];

  if (sensorValues[0] ==1) {
    reset();
    sensor= true;
  }
  if ((psensor == 1) && (csensor==0)) {
    reset();
    sensor0= true;
  }

  if (sensorValues[1] ==1) {
    reset();
    sensor1= true;
  }
  if (sensorValues[2] ==1) {
    reset();
    sensor2= true;
  }
  if (sensorValues[3] ==1) {
    reset();
    sensor3= true;
  }
  if (sensorValues[4] ==1) {
    reset();
    sensor4= true;
  }

  if ( sensor == true) {
    mode = 0;
  }
  if ( sensor0 == true) {
    mode = 1;
  }
  if ( sensor1 == true) {
    mode = 2;
  }
  if ( sensor2 == true) {
    mode = 3;
  }
  if ( sensor3 == true) {
    mode = 4;
  }
  if ( sensor4 == true) {
    mode = 5;
  }

  if (frameCount % 20 == 0) {
    updateTracking();
  }

  drawBackground();

  switch(mode) {
  case 0 :
    drawBackground();
    image(title, 150, 250);
    image(title2, 170, 450);
    break;

  case 1:
    drawBackground();
    scale(1.05, 1.05);
    image(instruction, 80, 300);
    break;

  case 2:
    drawBackground();
    ////    class swallow
    if (birdPlay == false) {
    birdS.play();
    birdPlay = true;
  }
    for (int i=0; i<swallows.size(); i++) {
      Swallow s = swallows.get(i);     
      s.display();
      if (s.x <-100) {
        swallows.remove(i);
      }
    }

    //class duck
    for (int i=0; i<ducks.size(); i++) {
      Duck d = ducks.get(i);
      d.display();
      if (d.x>700) {
        ducks.remove(i);
      }
    }
    //class bird
    for (int i=0; i<birds.size(); i++) {
      Bird b = birds.get(i);
      b.display();
      if (b.x>1300) {
        birds.remove(i);
      }
    }
    if (accX<-500) {
      swallows.add(new Swallow(1300, random(50, 120)));
    } else if (accX>500) {
      //fogs.add(new Fog(random(200,400), random(50, 120)));
      ducks.add (new Duck(random(200, 500), random(350, 450)));
    }
    if (accY<-200) { 
      birds.add (new Bird(random(700, 800), random(450, 500)));
    }
    // birdPlay = true;
    break;

  case 3:
    drawBackground();
    if (butterPlay == false) {
      butterflyS.play();
      butterPlay =true;
    }
    for (int i=0; i<butterflyU.size(); i++) {
      ButterflyU bU = butterflyU.get(i);
      bU.display();
      if (bU.t>380) {
        butterflyU.remove(i);
      }
    }

    for (int i=0; i<butterflyL.size(); i++) {
      ButterflyL bL = butterflyL.get(i);
      bL.display();
      if (bL.t>600) {
        butterflyL.remove(i);
      }
    }

    for (int i=0; i<butterflyR.size(); i++) {
      ButterflyR bR = butterflyR.get(i);
      bR.display();
      if (bR.t>300) {
        butterflyR.remove(i);
      }
    }
    if (accY<-200) { 
      for (int i = 0; i < 3; i++) {
        butterflyU.add(new ButterflyU(600, 400));
      }
    } else if (accX>500) { 
      butterflyL.add(new ButterflyL(random(200, 400), random(50, 150)));
    } else if (accX<-500) { 
      butterflyR.add(new ButterflyR(random(900, 1000), random(200, 500)));
    }


    break;

  case 4:
    drawBackground();
    if (fishPlay == false) {
      fishS.play();
      fishPlay =true;
    }
    //left to right fish
    for (int i = 0; i < fish.size (); i++) {
      Fish f = (Fish) fish.get(i);
      fill(0, 50);
      f.display();
      if ((f.loc.x>600)||(f.loc.y>500)) {
        fish.remove(i);
      }
    }
    for (int i = 0; i < fish2.size (); i++) {
      Fish2 f2 = (Fish2) fish2.get(i);
      f2.display();
      if (f2.count>240) {
        fish2.remove(i);
      }
    }

    if (accX>500) { 
      for (int i = 0; i < 10; i++) {
        fish.add(new Fish());
      }
    }
    if (accY<-200) { 
      fish2.add (new Fish2(random(200, 500), random(350, 500)));
    }
    break;

  case 5:
    drawBackground();
    if (weatherPlay == false) {
      weatherS.play();
      weatherPlay =true;
    }
    //class Petal
    for (int i=0; i<petals.size(); i++) {
      PetalGroup p = petals.get(i);
      p.display();
      if (p.dropped > 10) {
        petals.remove(i);
      }
    }
    //class fog
    for (int i=0; i<fogs.size(); i++) {
      Fog f = fogs.get(i);
      f.display();
      if (f.Fx<-800) {
        fogs.remove(i);
      }
    }
    //class cloud
    for (int i=0; i<clouds.size(); i++) {
      Cloud c = clouds.get(i);     
      c.display();
      if (c.x> 1000) {
        clouds.remove(i);
      }
    }

    if (accX<-500) { 
      petals.add (new PetalGroup());
    } else if (accX>600) {
      fogs.add(new Fog(random(300, 500), random(180, 200)));
    } else if (accY<-200) {
      clouds.add(new Cloud(random(20, 100), random(50, 1-0)));
    }
    break;
  }


  //fill(pickedColor);
  //rect(10, 10, 50, 50);
  // test the center position
  stroke(0, 80);
  ellipse(posX, posY, 10, 10);
  // show the frame rate
  //fill(0);
  //text(frameRate, width/2, height/2);
  //text(posX, width/2, height/2+20);
  //if (mousePressed ) {
  //  println ( mouseX+"+"+mouseY);
  //}

  pPosX = posX;
  pPosY = posY;
  psensor=csensor;
  //}
  //test color 
  //  draw the webcam image 
  // image(cam, 0, 0);

  ////image(detectedAreaImg, 0, 0);
  ////draw the picked color
  //  noStroke();
  //}
  ////test color
  //void mousePressed() {
  //  pickedColor = color(255, 0, 0);
  //  pickedColor = cam.get(mouseX, mouseY);
  //  println(red(pickedColor) + ", " + green(pickedColor) + ", " + blue(pickedColor));
  //}
}

void drawBackground() {
  fill(255, 70);
  pushMatrix();
  imageMode(CORNER);
  rect(-10, -10, width+20, height+20);
  image(background, 0, 0);
  popMatrix();
  stroke(255, 90);
}

void reset() {
  drawBackground();
  sensor =false;
  sensor0 =false;
  sensor1 =false;
  sensor2 =false;
  sensor3 =false;
  sensor4 =false;
  weatherPlay = false;
  birdPlay = false;
  fishPlay = false;
  butterPlay= false;
  butterflyS.stop();
  birdS.stop();
  weatherS.stop();
  fishS.stop();
}
void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 4 ], 9600);
  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = 'n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}

void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = 'n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}

class Duck {
  float x = random(200,500);
  float y = random(300,400);
  float yoff;

  Duck (float px, float py) {
    x= px;
    y = py;
  }

  void display() {
    pushMatrix();
    imageMode(CENTER);
    image(duck, x, y, 100, 100);
    popMatrix();
    float amp = 10;
    float freq = frameCount * 0.01;
    float Yoff= noise(freq) * amp; 
    y+= Yoff*0.1;
    yoff += 0.01; 
    x = x + 2;
  }
}

class Swallow {
  float x = 1000;
  float y;
  float yoff;

  Swallow (float px, float py) {
    x= px;
    y = py;
  }

  void display() {
    pushMatrix();
    imageMode(CENTER);
    image(swallowP, x, y, 80, 80);
    popMatrix();
    float amp = 10;
    float freq = frameCount * 0.01;
    float swallowYoff= noise(freq) * amp; 
    y-= swallowYoff*0.1;
    yoff += 0.1; 
    x = x - 4;
  }
}

class Bird {
  float x;
  float y;

  Bird (float px, float py) {
    x= px;
    y = py;
  }

  void display() {
    pushMatrix();
    imageMode(CENTER);
    image(bird, x, y, 60, 60);
    popMatrix();
    //int count = 0;
    x = x +3;
    y = y + (sin(frameCount*.5));
    println(y);
   // count++;
  }
}

class ButterflyU {
  float x = random(200, 500);
  float y = random(300, 400);
  float yoff;
  float t = 0;
  float rotation=0;
  boolean a = false;
  ButterflyU (float px, float py) {
    x= px;
    y = py;
  }

  void display() {
    pushMatrix();
    translate(x, y);
    if (t<180) {
      t+=0.3;
      println(t);
    }
    float x = 16 * pow(sin(radians(t)), 3);
    float y = 13 * cos(radians(t)) - 5 * cos(2 * radians(t)) - 2 * cos(3 * radians(t)) - cos(4 * radians(t));
    noStroke();
    fill(0);
    imageMode(CENTER);
    pushMatrix();
    translate(x*12, -y*12);
    rotate(30+t * 0.008);
    image(butterfly1, 0, 0);
    popMatrix();
    pushMatrix();
    translate(-x*12, -y*12);
    rotate(-t * 0.008);
    image(butterfly1, 0, 0);
    popMatrix();
    popMatrix();
    t++;
  }
}

class ButterflyL {
  float ax;
  float ay;
  float angle;
  float t = 0;
  ButterflyL (float px, float py) {
    ax= px;
    ay = py;
  }

  void display() {
    imageMode(CENTER);
    image(butterfly2, ax, ay);
    pushMatrix();
    translate(ax, ay);
    rotate(radians(angle));
    image(butterfly1, 0, 100);
    angle+=.5;
    popMatrix();
    ax+=1;
    t++;
  }
}

class ButterflyR{
  float rx;
  float ry;
  float angle;
  float t = 0;
  ButterflyR (float px, float py) {
    rx= px;
    ry = py;
  }

  void display() {
     pushMatrix();
    imageMode(CENTER);
    image(butterfly3, rx, ry);
    image(butterfly1, rx+15,ry+100*sin(t));
    popMatrix();
    rx-=1;
    t+=.01;
  }
}
class Fish {
  PVector loc;
  PVector vel;
  /* Just to add some individuality to the fish wiggle */
  float s = random(-90, 90);
  float d = random(0.1, 0.2);

  Fish() {
    loc = new PVector(random(300, 400), random(300, 400));
    /* Make a random velocity */
    vel = new PVector(random(.5, 1), random(.5, 1));
  }

  void display() {
    loc.add(vel);
    pushMatrix();
    translate(loc.x, loc.y);
    scale(d);
    /* Get the direction and add 90 degrees. */
    rotate(vel.heading2D()-radians(90));
    beginShape();
    for (int i = 0; i <= 180; i+=20) {
      float x = sin(radians(i)) * i/3;
      float angle = sin(radians(i+s+frameCount*5)) * 50;
      vertex(x-angle, i*2);
      vertex(x-angle, i*2);
    }
    /* 
     Started from the top now we are here. We need to now start to draw from where the first for loop left off. 
     Otherwise un ugly line will appear down the middle. To see what I mean uncomment the below line and comment
     the other line.
     */
    for (int i = 180; i >= 0; i-=20) {
      //for (int i = 0; i < 180; i+=20){
      float x = sin(radians(i)) * i/3;
      float angle = sin(radians(i+s+frameCount*5)) * 50;
      vertex(-x-angle, i*2);
      vertex(-x-angle, i*2);
    }
    endShape();
    popMatrix();
  }
}

class Fish2 {
  float x;
  float y;
  float count = 0;
  float rotation = 0;
  Fish2 (float px, float py) {
    x= px;
    y = py;
  }

  void display() {
    pushMatrix();
    translate(x,y);
    rotate(-radians(rotation));
    imageMode(CENTER);
    image(fishTwo, 0,0, 120, 120);
    popMatrix();
    count++;
    rotation ++;
    //println(count);
  }
}

void updateTracking() {
  if (cam.available() == true) {
    cam.read();
    cam.loadPixels();
  }

  int sumX = 0;
  int sumY = 0;
  float avgX = 0;
  float avgY = 0;
  int count = 0;

  int w = cam.width;
  int h = cam.height;
  for (int y = 0; y < h; y++) {
    for (int x = 0; x < w; x++) {
      int i =  x + y*w; // IMPORTANT

      int r = int( red(cam.pixels[i]) );
      int g = int( green(cam.pixels[i]) );
      int b = int( blue(cam.pixels[i]) );

      if (r > red(pickedColor)-tolerance && r < red(pickedColor)+tolerance &&
        g > green(pickedColor)-tolerance && g < green(pickedColor)+tolerance &&
        b > blue(pickedColor)-tolerance && b < blue(pickedColor)+tolerance )
      {
        cam.pixels[i] = color(100,230,114);
        sumX += x;
        sumY += y;
        count++;
      } else { 
      }
    }
  }
  cam.updatePixels();
  // get average position to get the center position
  if (count != 0) {
    avgX = sumX / count;
    avgY = sumY / count;
  }

  posX = map(avgX,0,320,width,0);
  posY = map(avgY,0,240,0,height);
  
  accX = posX - pPosX;
  accY = posY - pPosY;
//  println(accX+"+"+accY);

}
class Petal {
  float x = random(600, 900);
  float y = random(400, 500);
  float xoff;
  Petal (float px, float py) {
    x= px;
    y=py;
  }

  void display() {
    pushMatrix();
    imageMode(CENTER);
    image(petal, x, y, 20, 20);
    popMatrix();
    float amp = 10;
    float freq = frameCount * 0.01;
    float xoff= noise(freq) * amp; 
    xoff += 0.02; 
    x+= xoff*0.1;
    y = y + random(1, 3);
  }
}

class PetalGroup {
  Petal[] tenPetals = new Petal[10];
  int dropped = 0;

  PetalGroup() {
    for (int i=0; i<tenPetals.length; i++) {
      tenPetals[i]= new Petal(random(800, 1090), random(500, 600));
    }
  }

  void display() {
    for (int i=0; i<tenPetals.length; i++) {
      tenPetals[i].display(); 
      if (tenPetals[i].y > 800) {
        dropped++;
      }
    }
  }
}

class Fog {
  float Fx = random(200, 500);
  float Fy = random(300, 400);
  float yoff;

  Fog (float px, float py) {
    Fx= px;
    Fy =py;
  }
  void display() {
    imageMode(CENTER);
    float freq, amp;
    amp = 80;
    freq = frameCount * 0.01;
    float xOffset1 = sin(freq) * amp;
    image(fog, Fx-80, Fy+(xOffset1*.7));
    //image(fog2, Fx+160, Fy+(xOffset1*.5));
    //float xOffset2 = sin(freq*1.2) * amp;
    //image(fog1, Fx+80, Fy+(xOffset2*.4));
    //image(fog2, Fx+160, Fy+(xOffset2*.6));
    Fx-=1;
  }
}

class Cloud {
  float x;
  float y;
  float yoff;


 Cloud (float px, float py) {
    x= px;
    y = py;
  }
  void display() {
float freq, amp;
    amp = random(30,40);
    freq = frameCount * 0.2;
    float xOffset1 = 0.3*sin(freq*.3) * amp;
    pushMatrix();
    imageMode(CENTER);
    image(cloud, x, y+xOffset1);
    popMatrix();
    x = x +random(0.8,1.5);
  }
}

//Arduino

void setup()
{
  Serial.begin(9600);
  pinMode(9, INPUT);
  pinMode(10, INPUT);
  pinMode(11, INPUT);
  pinMode(12, INPUT);
  pinMode(13, INPUT);

}

void loop() {
  int sensor1 = digitalRead(9);
  int sensor2 = digitalRead(10);
  int sensor3 = digitalRead(11);
  int sensor4 = digitalRead(12);
  int sensor5 = digitalRead(13);

  Serial.print(sensor1);
  Serial.print(",");
  Serial.print(sensor2);
  Serial.print(",");
  Serial.print(sensor3);
  Serial.print(",");
  Serial.print(sensor4);
  Serial.print("," );
  Serial.print(sensor5);
  Serial.println( );
  delay(50);

}

Interaction Lab(Moon) Recitation 11: Media Controller, Hanlu Zhang (Mary)

  • Date: 1st, Dec. 2017
  • Instructor: Moon
  • Aim: Create a Processing sketch that controls media elements (images, videos and audio) using a physical controller made with Arduino.
  • Process
  • I decide to make a sketch that when you water the sensor, the tree will recovered from green.
  • So I choose the  Moisture Sensor, which can detect the moisture of tissue. Then I search a picture of tree with green tree from (https://www.google.com.hk/imgres?imgurl=https%3A%2F%2Fwww.clipartsgram.com%2Fimage%2F1586397434-19802256-summer-tree-with-green-leaves-illustration.jpg&imgrefurl=https%3A%2F%2Fwww.clipartsgram.com%2Fclipart-of-tree-of-life-17181&docid=v69yXtQuyY7RUM&tbnid=CfBFPnCAflNDHM%3A&vet=10ahUKEwjOwseT8ufXAhVEW5QKHfKJDKo4kAMQMwgQKA4wDg..i&w=450&h=450&bih=706&biw=1439&q=hand%20drawn%20tree&ved=0ahUKEwjOwseT8ufXAhVEW5QKHfKJDKo4kAMQMwgQKA4wDg&iact=mrc&uact=8)
  • Then I connect the moisture sensor into circuit, as the value from sensor is from 0-1023, so I change the output to (digital read/4) so that the value from Arduino will be 0-255
  • Then I add the picture into Processing, using a filter called GRAY, and the tree become black and white. Then I draw the picture again, and this time the transparency will be the value from Arduino. Therefore, the more wet the sensor detect, the more green this tree will be. Here’s the Demo of my project.
  •  Course Learned
  • The use of filter
  • serial communication

Interaction Lab(Moon) Recitation 10: Object Oriented Programming, Hanlu Zhang (Mary)

  • Date: 26th Nov, 2017
  • Instructor: Moon
  • Aim: Review the use of variables, do some practice by using array[] and class
  • Process
  • First Jack and another fellow help us review the use of variables, showing us the structure of class and array, as well as the relationship between them. Then they show us some examples of mobiles, and we are ask to make a small project by using variables.
  • I decide to make a project that when mouse pressed, a flower will appear at the mouse position and rotate.
  • First I create the class of flower, with two functions which are display() and move(). And write the main code of flower. move and flower. display. Then I found it doesn’t work, with the help of Ariel, I found it was because I should use array rather than class to create serval flowers.
  • After changing my structure to array, here comes another problem, only the first flower can appear. Then Ariel told me that I need to write a delay() after the code as mouse will need some time to press and the process may count more than once.
  • Finally, here is my project:
flower[] myFlowerArray = new flower[15];
int counter = 0;
void setup() {
  size(500, 600);
  for (int i = 0; i < myFlowerArray.length; i++) {
    myFlowerArray[i]= new flower(random(0,width), random(0, height), 0);
  }
}
void draw() {
  background(255);

  if ( mousePressed && counter < myFlowerArray.length) { 
    myFlowerArray[counter].move();
    counter += 1;
    delay(100);
  }


  for (int i = 0; i < counter; i++) {
    myFlowerArray[i].rotateFlower();
    myFlowerArray[i].display();
    println(i);
  }
}

class flower {
  float transX;
  float transY;
  float rotate;

  flower (float px, float py, float rot) {
    transX= px;
    transY = py;
    rotate = rot;
  }

  void display() {
    pushMatrix();
    translate(transX,transY);
    rotate(radians(rotate));
    noStroke();
    fill(#FFEC8B);
    ellipse(0, 0, 30, 30);
    fill(#FFB6C1);
    ellipse(0, 40, 40, 40);
    ellipse(0, -40, 40, 40);
    ellipse(40, 0, 40, 40);
    ellipse(-40, 0, 40, 40);
    popMatrix();
  }
  void move() {
   
    this.transX = mouseX;
    this.transY = mouseY;
  }
  void rotateFlower(){
     rotate++;
  }
}

Interaction Lab(Moon) Final Project Idea: Hanlu Zhang (Mary)

  • Instructor: Moon
  • Project Name: Magic Ink Brush
  • Introduction

My idea is to make a interactive device that everyone can enjoy the process of Chinese painting, as well as to communicate traditional Chinese culture. The project contain a physical Ink brush and the screen, and users will hold a Ink Brush and their different movements can cause different painting effects on screen.

  • Inspirations

My inspiration comes from two parts: A Chinese story and Ink painting (and related cartoon).

The first part is a Chinese story called “Incredible Ink” and in Chinese is “Shenbimaliang”, in which story the little boy get a magic ink brush that what he draw will become real items, which gives me the idea of making a project that user can draw some amazing effects with simple gestures.

The second part is from the traditional Chinese brush painting, which I used to learn serval years. I also inspired by some excellent brush painting style cartoon made by Chinese in 1960s like “Where is Mama“ and “buffalo boy“ and the most praised one called Feeling from Mountain and Water”

(ScreenShots from”Where is Mama“ and “buffalo boy“)

The video of “Feeling from Mountain and Water”

I think the inheritance of traditional culture need to attract people’s attention first, and combine it with technology is attractive way. So this time I want to try whether this project can make some people become interested in Chinese culture. 

  • Arduino Part

Arduino part contains to objects, one is brush with accelerometer and a button, the other one is a plate with four area represent four seasons. The different gestures detected by sensor will send to the processing part for different effects, and when the brush touch the plate, current mode will change which imitate the process of dipping in ink. The button is to control when to paint, if user don’t click the button, nothing will be draw on the screen.

  • Processing Part

Processing part is concentrate on visual effects. The whole project will have four modes that is the four seasons. There will be some basic items that exist in each mode like house, mountain, some plants. But each mode have some different containers which is related to the datas from Arduino. For example, if the user draw horizontally, swallows will fly over the sky in spring mode while frogs will jump out of river in summer mode, leaves will be blowed by the wind in Autumn and an eagle may appear in winter. And I will add some sound effect to make it more complete.

(sample seasons and animals)

  • Difficulties
  • The use of accelerometers may be a difficult part that the output value is complexed.
  • Visual effects: as I don’t know what kind of final effects I can figure out in the end and how to make all the pictures apply to one style.

 

 

Interaction Lab(Moon) Recitation 9: 3D Modeling, Hanlu Zhang (Mary)

  • Instructor: Moon
  • Aim:
  • Using Tinkercad, design a 3D model of a wearable device, a game controller or a mechanism (for a component) that utilizes one of provided components or equipment.
  • Process

My idea is to design a device with Ultrasonic Distance and can be installed on mobiles,  which can prevent people from running into walls or trees if they playing on their phones when walk on street.

Using iPhone 6 as an example, first I find the data of iPhone 6 and make a model of it.

combining with another cube, I create a module that can contain iPhone6.

Then I start to create the other part that can contain the distance sensor, I find the size of Ultrasonic Distance in provided material, as well as measure some data by Vernier scale.

Using the data of sensor size, I create a module of the sensor, I also leave a hole for wires getting out of the module.

Combine with another cube, I got the container of sensor. Finally I combine the two modules I have and get the device for distance sensor and iPhone.

  • Course Learned
  • How to use Thinkcad
  • Possible use of 3D module

 

Interaction Lab(Moon) Recitation 8: Drawing Machines, Hanlu Zhang (Mary)

  • Instructor: Moon
  • Partner: Lichen Dong
  • Aim:
  • Build a circuit to control the stepper.
  • Build a circuit that can control stepper by potentiometer and the MotorKnob example in Arduino.
  • Write a sketch on Processing that send values to Arduino to control the motor
  • Corporate with my partner to combine two motors  into a mechanical arm that can hold a marker. Use processing to control the movement of each motor, and make adjustments with my partner to draw on the paper
  • Material
  • 1 42STH33-0404AC stepper motor
  • 1 SN75440NE ic chip
  • 1 power jack
  • 12 V power supply
  • 1 potentiometer from your kit
  • 1 Arduino and USB cable from your kit
  • Laser-cut mechanisms
  • Pen that fits the laser-cut mechanisms
  • Schematics
  • (http://ima.nyu.sh/interaction-lab/lab-8-drawing-machines/)
  • Process
  • Part1
  • First I connect the circuit according to the schematics and use power to make it rotate.
  •    
  • Part2
  • Then I add a potentiometer into the circuit to control the stepper.
  • However, the stepper can rotate but can’t be controlled by the potentiometer, then with professor Antonius’ reminder, I found that the out put from potentiometer is 0-1023 while the input that stepper need is from 0-200. So I use map function to make the data suitable, and then the stepper can be controlled
  • .
  • Test video:
  • Part 3
  • Then I remove the potentiometer and control the stepper with data from Processing. I create a sketch which is 200*200, and map the mouse position with angle of stepper
  • Then I put together the parts of the laser-cut mechanisms like the picture below
  • And the test movie is below:
  • Part 4
  • Finally, I combine my mechanisms with my partner’s, and control the pen together. Here’s the video that my partner control two computer together while I’m taking this video.
  • Coursed learned:

  • Use of map() function
  • Ways of interaction

Interaction Lab(Moon) Midterm Project: User Test Part II

  • Project Name: Xylophone–Music Learner
  • Producer: Lichen Dong
  • Using experience: Lichen’s project it’s constructed by battens that represent different keys in the Xylophone, and when you knock the key, a drop of color will drop in the barrel below. The project looks very nice as it reminds me of one of my toy in childhood, and it also reminds me a traditional instrument called “Yangqin“. And when I knock the keys with a stick, user experience is good as it feels that I’m a real musician.
  • Suggestions
    • My first suggestion will be that it’s will be better to add some performing instructions so that I can play some songs on the Xylophone, which will let me gain more pleasure from playing it.
    • Maybe it will looks better to paint the keys in the same color with drops so the physical part and processing part will be more connected
    • Maybe it’s also possible to make the project more interactive that when the color drop, users click the corresponding key to make a sound.
  • Test video (video by Lichen)

Interaction Lab(Moon) Midterm Project: User Test Part I

  • Midterm Project: Star Puzzle
  • Name: Hanlu Zhang(Mary)
  • Date: October.2 2017
  • Instructor: Moon
  • Questions I have for the testers
    • Is it easy for you to understand the project?
    • Are the instructions clear enough?
    • Is the degree of difficulty proper?
  • User’s feedbacks
    • Problems1: The dots are too little to be clicked so sometime just miss them and have to restart the whole levels
      • Possible reasons:
      • The dots is indeed a little bit too small;
      • When mouse is on the dot, the change of color is not obvious enough
      • The original color of dots is not obvious enough.
      • Possible solutions:
      •  Make the dots bigger. But consider the visual effects, it will be a better choice to keep the size of dots but change the range that the click can be counted as clicked.
      • Change the original color and the color after being selected to make it more obvious.
    • Problems2: It’s hard to know what’s the star is, especially in Level3 and Level 4, one of which has lines inside the star and the other has three stars in one scene
      • Possible reasons:
      • The introductions and informations I give at the beginning is not enough that user can’t understand what the game is.
      • The shortage of instructions before each level.
      • The shortage to tell people what does each shape look like rather than say “star”
      • Possible solutions
      • Give instructions before each level like : Draw a normal star/Draw a star with 7 points/Draw a star with lines inside/Draw 3 stars in this level. And it will be easier for users to understand.
      • At the start page try to explain a little bit more or add another state for rules.
    • Problem3: The feedback to users are too few that users will feel confused or annoyed when all the lines disappear.
      • Possible reasons:
      • When they make some mistakes, I don’t give enough feedback to show that the lines are disappeared because of you click a wrong dots or don’t click on and dots.
      • Sometimes users make too much mistakes and lose the patience to continue.
      • Possible solutions:
      • Give more feedbacks like when connected a wrong lines, the reminder that “wrong lines”will appear, and also says “no dots selected” when clicking on background.
      • To make it more interesting, some effects can be added. For example, for user whoo have restarted the current level twice, when the mouse is near dots, it will “shake” unless it’s the right one.

Interaction Lab(Moon) Midterm Project: Star Puzzle- Hanlu Zhang(Mary)

Midterm Project: STAR PUZZLE

  • Date: October.2 2017
  • Instructor: Moon
  • Introduction

Star Puzzle is a game that user will connect dots to draw stars and each line can only be drawn once. If a line is drawn twice or a line doesn’t exist is drawn or the user click somewhere that doesn’t have a star, the current level will be restart. The game has 4 levels in total.

  • Inspiration 

At first I want to make a game that users can draw Chinese constellations, however I find them to abstract to become a visual game. Then, during my research online, I saw a game that let users to draw a shape in one continual line(https://www.openprocessing.org/sketch/4827#), which inspire me a game that I always play with my mom when I was young that we try to draw stars with different number of points in a continual line. So I decided to make a game that users need to figure how to draw all the lines of the star in one time.

  • Process 
  1. Background

As the game is about drawing stars, so I want to make the background looks like real sky. Therefore, I set an array with 1500 dots with random position and random size between 0.1 to 3. The opacity is random and will run circular as it’s under the “void”, so the stars look like that they are “blinking”.

To make the background more interactive, I also use “dist” function to detect the distance between  the mouse position and stars. if the mouse is near the star, the dot will be a little bit bigger.

here is the code:

And the effect is the video below:

 

2. Setup

The main elements in my projects are dots and lines. So I first set up arrays for dots and lines in different levels.

Then I setup some variables for detecting:

select dot: the dot user click on to star, I define this with the mousePressed and distance between mouse and dot’s center.

totalLines: when mouse click, totalLines +1, which can detect how many lines user has been drawn in total

countLines: when a right line is drawn, countLines +1, which can detect how many correct lines has been drawn.

solve: which to show whether the user finish the current level.

draw: to decide whether the line can be draw.

3.Detection

First is for whether the lines can be draw. i will detect it each time mouseReleased. When solve == false, which means current level haven’t been solved, and a dot is selected, then drawing == true that lines can be drawn. The the dot be selected first is startDot and the later one is endDot.

And in void draw, I also detect the endDot, if endDot doesn’t exist, the line is between mouse and startDot. If endDot exist, the line between them is drawn and endDot becomes the new startDot. The code is below

 

Second is whether the current level has been finished. I use if statement to detect whether user have finished the current level. The code is:

countLine = totalLine means the user doesn’t draw any lines twice and countLines = currentLines.length (I will explain this array later) means all the lines in current level has been drawn. When these two statements are fulfilled, the user has drawn all the lines correctly and can enter next level.

  • Problems and Developments
  1. As I plan to design several puzzles, I have different arrays in both lines and dots, but when I first write the code, I design everything based on level which only contains lines1 and dots1. So after I finishing the structure of my code, I became confused that whether I have to rewrite all the code for level2 and again for level 3, which may make my code super long. Then with the help of Jiwon, I understand that I can set up another arrays that are “Dot[] currentDots;” and “Line[] currentLines;” and declare them with l the lines array and dots array in it, So that I can use currentLines/Dots instead of specific arrays and my processing become more organized.
  2.  As I have different level, but I don’t know how to make the action of “moving to next level”. As Kaye has similar part in her code, so she shared the use of mode and case functions. Then, when  solve == true, mode will switch to the next one, and all the variables can be reset.

To make is more complete, I also create start and end scenes

3. After setting the start page, I set the if statement that if mousePressed in start page, first level will start. However, when I test it, I found that sometimes it detected the mouseReleased, but doesn’t detect the Press. Then Professor moon tell me that this happens because I write the “if mousePressed” function in the “void draw”, which has some distance with the function “void mousePressed” I write in the later part. So sometimes, processing may miss the order as the frameRate has refresh before it perform the order.

So I move this code under the mousePressed and I also move the mousePressed function in mode 5(ending part) into the same place to avoid this problem.

  • User Test

After finishing the first draft, I ask some friends to test it, and I received some responses from them. Professor moon and fellows in lab also give me some suggestions to make it better.

1. First response is that the current level should be reset automatically when user makes a mistake so that they don’t need to restart from the first level. So that I set another if statement in the checkResult part that if countLines doesn’t equal to totalLines-1, which means user draw lines more than ones, current level reset. However for the situation that user click somewhere that doesn’t have dots is a little bit tricky. With professor Moon’s advice, I create a new variable called “count” and if one dot is selected, count will plus 1 . Each time mouseReleased, processing will check the count, if count = 0, which means no dot is selected, the current level will be reset.

The effect is like the video below:

2. In the first draft, the shape that user need to draw in the level will show in the background and they will draw the lines on that graph like that:

Then professor Moon suggest me delete the graph in the background to make the visual effects better and also make the game harder, so in the final draft, each level will start with dots only.

3. Some of my friend told me that the levels in first draft is too simple and suggest me to add a harder level so that people who are good at this kind of game won’t feel boring. So I add a new level that have 12 points and 18 lines, which looks like that:

  • Demo

Here is the demo of my project

  • Lesson Learned
  1. The use of array and class

In my project, I use several arrays to list the lines and dots in different levels, which makes my project organized, as well as make the logic more clear. The use of class makes it possible to control each line or dots in array, making my project easier to code.

2. Create new functions

I create some new functions like “selected””drawNightSky””setupNightSky”, which makes the code under setup and draw shorter and more clear to understand.

3. Smart use of “println();”

From this project, I feel the importance of “println();”. First, when I find the position of dots, I just insert the background photo and let processing “println” the mouse position when I click the dots’ positions.

Second I learn from Professor Moon and fellow Jiwon that when bug happens, I can use “println” important datas that are used to control the following actions to see whether data match the order. Or I can write “println something” after some certain code to say whether the order is working so that I can find where the problem is.

4. Be careful! Be patient!

During processing, I make some stupid mistakes like confusing the “currentLines” and “currentDots”, or type semicolon after “if()”. These mistakes takes a lot of time to be found and fixed, so being careful can save time.

And when something wrong happens in my code, being patience is important or it will spend more time to fining the bug.

  • Conclusion

From making this project, I practiced what I have learned in class, as well as having experience of the whole process of making a project. There are still some problems existing in my project like the instructions are not clear enough or the lack of feedback to user, which I will explain more and give some possible solutions in the later documentation. But the whole process really bring me lots of fun.

  • Special Thanks

Thanks Professor Moon for giving me suggestions of using “count” to detect whether dots are selected and suggestions in visual effects. Thanks Jack for helping me organize my logic and structure before starting coding. Thanks Jiwon for suggestions of using “currentLines/dots”. Thanks Kaye for sharing the use of mode/case with me. Thanks my roommate Demi who encourage and support me during the whole process. Thank you all !

PImage img1;
PImage img2;
PImage img3;
PImage img4;
PImage img5;
PImage img6;
PImage img7;
int mode = 0;
color dotColor = color(255);
color dotColorOver = color(200, 200, 0);
color dotColorClicked = color(200, 200, 0);
int countLines;

Dot[] currentDots;
Line[] currentLines;

Dot[] dots1 = new Dot[5];
Line[] lines1 = new Line[5];
//level one dots and lines
Dot[] dots2 = new Dot[7];
Line[] lines2 = new Line[7];
//level two dos and lines
Dot[] dots3 = new Dot[12];
Line[] lines3 = new Line[18];
//level four dos and lines
Dot[] dots4 = new Dot[13];
Line[] lines4 = new Line[15];
//level four dos and lines

int selectedDot=99, startDot = 99, endDot = 99;
int totalLines = 0;

boolean drawing;
boolean solved;

// night sky
int numofCircles = 1500;
float []size = new float[numofCircles];
float []x = new float[numofCircles];
float []y = new float[numofCircles];
color[] clr = new int [numofCircles];

void setup() {
  size(1000, 780);
  frameRate(15);
  drawing=false;
  solved=false;
  SetupDotsAndLines();

  //NightSky
  setupNightSky();
  img1 = loadImage("Title.png");
  img2 = loadImage("Start.png");
  img3 = loadImage("LevelTitle1.png");
  img4 = loadImage("LevelTitle2.png");
  img5 = loadImage("LevelTitle3.png");
  img6 = loadImage("LevelTitle4.png");
  img7 = loadImage("End.png");
  println("length of currentDots: " + currentDots.length);
}

void draw() {
  drawNightSky();
  switch(mode) {
  case 0:
    image(img1, 0, 0);
    image(img2, 250, 400);
    break;
  case 1: //level 1
    image(img3, 0, 0);
    break;

  case 2://level2
    image(img4, 0, 0);
    break;
  case 3://level3
    image(img5, 0, 0);
    break;
  case 4://level4
    image(img6, 0, 0);
    break;
  case 5://end
    image(img7, 200, 250);
  }
  // draw the dots
  if (mode > 0 && mode < 5) {
    for (int i=0; i<currentDots.length; i++) {
      currentDots[i].display();
    }
    for (int i=0; i<currentLines.length; i++) {
      if (currentLines[i].visible == true) { 
        currentLines[i].display();
      }
    }
  }

  //draw lines
  if (drawing==true) { 
    stroke(#FFEC8B, 250);
    if (selectedDot != 99) {
      line(currentDots[selectedDot].x, currentDots[selectedDot].y, mouseX, mouseY);
    }
  }
}

void mousePressed() {
  println("startDot: " + startDot + ", endDot: " + endDot + ", selectedDot: " + selectedDot);

  if (mode == 0) {
    mode = 1;
    totalLines = -1;
    println("total lines in case 0: " + totalLines);
  } else if (mode == 5) {
    selectedDot=99;
    startDot = 99;
    endDot = 99;
    totalLines = 0;
    solved=false;
    SetupDotsAndLines();
    for (int i = 0; i < currentLines.length; i++) {
      currentLines[i].visible = false;
      mode = 0;
    }
  }
}

void mouseReleased() {
  if (solved==false) {
    totalLines++;
    println("totalLines: " + totalLines);
    if (selectedDot!=99) {
      currentDots[selectedDot].c=dotColor;
    }
    drawing=false;  

    int count = 0;
    for (int i = 0; i < currentDots.length; i++) {
      if (currentDots[i].selected()==true) {
        count++;
      }
    }
    println(count);
    if (count == 0) { 
      selectedDot=99;
      startDot = 99;
      endDot = 99;
      totalLines = 0;
      for (int i = 0; i <currentLines.length; i++) {
        currentLines[i].visible = false;
      }
    }

    for (int i=0; i<currentDots.length; i++) {
      if ((currentDots[i].selected()==true) && (i!=startDot)) {
        drawing=true;
        currentDots[i].c=dotColorClicked;
        selectedDot=i;
        break;
      }
    }

    if (drawing==true) {
      if (startDot==99) {
        startDot=selectedDot;
        //println("In mouse released - startDot: " + startDot);
      } else if (endDot==99) {
        endDot=selectedDot;
        //println("In mouse released - endDot: " + endDot);

        drawLine(startDot, endDot);
        checkResult();
      }
      startDot=selectedDot;
      endDot=99;
    } else {
      startDot=99;
      endDot=99;
    }
  }
}

void mouseMoved() {
  if (solved==false) {
    for (int i=0; i<currentDots.length; i++) {
      if (selectedDot==i) {
      } else
        if (currentDots[i].selected()==true) {
          currentDots[i].c=dotColorOver;
        } else {
          currentDots[i].c=dotColor;
        }
    }
  }
}

/* mouseDrag is the same as mouseMove */
void mouseDragged() {
  mouseMoved();
}
void SetupDotsAndLines() {
  currentDots = dots1;
  currentLines = lines1;

  //level1 dots 
  dots1[0] = new Dot(502, 181);
  dots1[1] = new Dot(255, 368);
  dots1[2] = new Dot(347, 666);
  dots1[3] = new Dot(660, 667);
  dots1[4] = new Dot(755, 369);
  println("created dots1");
  //level1 lines
  lines1[0] = new Line(0, 2);
  lines1[1] = new Line(0, 3);
  lines1[2] = new Line(1, 3);
  lines1[3] = new Line(1, 4);
  lines1[4] = new Line(2, 4);
  println("created lines1");

  currentDots = dots2;
  currentLines = lines2;
  //level2 docts 
  dots2[0] = new Dot(505, 182);
  dots2[1] = new Dot(307, 283);
  dots2[2] = new Dot(259, 498);
  dots2[3] = new Dot(394, 672);
  dots2[4] = new Dot(618, 673);
  dots2[5] = new Dot(752, 497);
  dots2[6] = new Dot(705, 283);
  println("created dots2");

  //level2 lines
  lines2[0] = new Line(0, 3);
  lines2[1] = new Line(0, 4);
  lines2[2] = new Line(1, 4);
  lines2[3] = new Line(1, 5);
  lines2[4] = new Line(2, 5);
  lines2[5] = new Line(2, 6);
  lines2[6] = new Line(3, 6);
  println("created lines2");

  currentDots = dots3;
  currentLines = lines3;
  //level3 docts
  dots3[0] = new Dot(502, 137);
  dots3[1] = new Dot(256, 278);
  dots3[2] = new Dot(257, 560);
  dots3[3] = new Dot(502, 702);
  dots3[4] = new Dot(745, 562);
  dots3[5] = new Dot(746, 279);
  dots3[6] = new Dot(422, 279);
  dots3[7] = new Dot(339, 421);
  dots3[8] = new Dot(420, 560);
  dots3[9] = new Dot(584, 560);
  dots3[10] = new Dot(664, 421);
  dots3[11] = new Dot(583, 278);
  println("created dots3");

  //level3 lines
  lines3[0] = new Line(0, 6);
  lines3[1] = new Line(0, 11);
  lines3[2] = new Line(1, 6);
  lines3[3] = new Line(1, 7);
  lines3[4] = new Line(2, 7);
  lines3[5] = new Line(2, 8);
  lines3[6] = new Line(3, 8);
  lines3[7] = new Line(3, 9);
  lines3[8] = new Line(4, 9);
  lines3[9] = new Line(4, 10);
  lines3[10] = new Line(5, 10);
  lines3[11] = new Line(5, 11);
  lines3[12] = new Line(6, 7);
  lines3[13] = new Line(7, 8);
  lines3[14] = new Line(8, 9);
  lines3[15] = new Line(9, 10);
  lines3[16] = new Line(10, 11);
  lines3[17] = new Line(6, 11);
  println("created lines3");

  currentDots = dots4;
  currentLines = lines4;
  //level4 docts
  dots4[0] = new Dot(262, 318);
  dots4[1] = new Dot(383, 407);
  dots4[2] = new Dot(338, 549);
  dots4[3] = new Dot(187, 550);
  dots4[4] = new Dot(144, 407);
  dots4[5] = new Dot(503, 319);
  dots4[6] = new Dot(624, 407);
  dots4[7] = new Dot(578, 549);
  dots4[8] = new Dot(429, 550);
  dots4[9] = new Dot(744, 317);
  dots4[10] = new Dot(865, 408);
  dots4[11] = new Dot(819, 548);
  dots4[12] = new Dot(669, 550);
  println("created dots4");

  //level4 lines
  lines4[0] = new Line(0, 2);
  lines4[1] = new Line(0, 3);
  lines4[2] = new Line(1, 3);
  lines4[3] = new Line(1, 4);
  lines4[4] = new Line(2, 4);
  lines4[5] = new Line(5, 7);
  lines4[6] = new Line(5, 8);
  lines4[7] = new Line(6, 8);
  lines4[8] = new Line(1, 6);
  lines4[9] = new Line(1, 7);
  lines4[10] = new Line(9, 11);
  lines4[11] = new Line(9, 12);
  lines4[12] = new Line(10, 12);
  lines4[13] = new Line(6, 10);
  lines4[14] = new Line(6, 11);
  println("created lines4");

  currentDots = dots1;
  currentLines = lines1;
}

void setupNightSky() {
  for (int i=0; i<numofCircles; i++) {
    x[i] = random(width);
    y[i] = random(height);
    size[i] = random(.1, 3);
  }
}

void drawNightSky() {
  background(#00005b);
  for (int i = 0; i < size.length; i++) {
    fill(random(235, 255), random(235, 255), 255, random(100, 255));
    noStroke();
    float distance = dist(mouseX, mouseY, x[i], y[i]);
    if (distance < 10) {
      ellipse(x[i], y[i], size[i]+3, size[i]+3);
    } else { 
      ellipse(x[i], y[i], size[i], size[i]);
    }
  }
}

class Dot {
  int x; // dot x position
  int y; // dot y position
  int s; // dot size
  color c; // dot color
  boolean selected;

  Dot(int localX, int localY) {
    x = localX;
    y = localY;
    s = 20;
    c = color(192);
  }

  void display() {
    noStroke();
    fill(c);
    ellipse(x, y, s, s);
  }

  boolean selected() {
    float disX = x - mouseX;
    float disY = y - mouseY;

    if (sqrt(sq(disX) + sq(disY)) < s/2 ) {
      //println("selected dot with x: " + x);

      return true;
    } else {
      return false;
    }
  }
}

class Line {
  int startDotIndex, endDotIndex;
  int startX, startY;
  int endX, endY;
  boolean visible;

  Line(int startDot, int endDot) {
    startDotIndex = startDot;
    endDotIndex = endDot;
    startX=currentDots[startDot].x;
    startY=currentDots[startDot].y;
    endX=currentDots[endDot].x;
    endY=currentDots[endDot].y;
    visible=false;
  }
  void display() {
    stroke(#FFD700, 250);
    line(startX, startY, endX, endY);
  }
}

void drawLine(int startDot, int endDot) {

  int a = 20;
  int b = 20;
  if (startDot < endDot) {
    a = startDot;
    b = endDot;
  } else if (startDot > endDot) {
    a = endDot;
    b = startDot;
  } else {
    drawing=false;
  }
  if (drawing==true) {
    for (int i = 0; i < currentLines.length; i++) {
      if (a == currentLines[i].startDotIndex && b == currentLines[i].endDotIndex) {
        currentLines[i].visible = true;
        println("line " + i  + " is visible");
      }
    }
  }
}

void checkResult() {
  int countLines=0;
  for (int i=0; i<currentLines.length; i++) {
    if (currentLines[i].visible==true) {
      countLines++;
    }
  }

  println("countLine: " + countLines);
  println("currentLines.length:"+currentLines.length);

  for (int i=0; i<currentLines.length; i++) {
    if ((countLines==totalLines-1)&&(countLines==currentLines.length)) {
      drawing=false;
      solved=true;
      mode++;
      switch(mode) {
      case 2:
        currentDots = dots2;
        currentLines = lines2;
        break;
      case 3:
        currentDots = dots3;
        currentLines = lines3;
        break;
      case 4:
        println("case 3");
        currentDots = dots4;
        currentLines = lines4;
        //println("currentDots length: " + currentDots.length);
        break;
      case 5:
        currentDots = dots1;
        currentLines = lines1;
        break;
      }

      println("mode: " + mode);

      selectedDot=99;
      startDot = 99;
      endDot = 99;
      totalLines = 0;
      solved=false;
    } else if (countLines!=totalLines-1)
    {
      selectedDot=99;
      startDot = 99;
      endDot = 99;
      totalLines = 0;
      currentLines[i].visible = false;
    }
  }
}