Final Project Documentation (Ellen)

Treenstrument & Treemotion  树琴与树情

By Jiannan Shi & Ellen Ying


Coming up with the Idea

When Nan and I first started to think about the the idea for our final project, we decided to play with music and sound, and to combine them with the concept pf nature. The original idea was an interactive board displaying four seasons (as shown in the picture). We planned to create trees, birds and a river, and when users interact through some kinds of sensors, these elements in nature will jointly show the characteristics of different seasons. The processing is used to playing Vivaldi’s Four Seasons. However, we found that interactivity was hard to be integrated well in this idea, and the physical part, the visual part and the audio part were not connected closely. Finally, we decide to change this idea a little bit.

We decided to stick to the larger concept of combining sound and music with nature. Inspired by the final project of Candy Bi last semester, we decided to create a tree and let users to interact with it. Since music was the core of our project, an instrument became our interactive medium.

But what kind of instrument was the most suitable candidate here? Harp. First, the interaction with a harp only contains one motion. When you touch the harp, it will make sounds. The simple one way of interaction can be directly link to the visuals, thus helping us better define our project theme. More importantly, tree is pronounced as shu in Chinese and harp is pronounced as shuqin. The same pronunciation connects the two closely. Additionally, emotion is pronounced as qing in Chinese, which is similar to qin. The concept of harp thus became the bridge connecting the concept of tree and emotion. This gave us a great inspiration and we finally defined our project concept in the following way: users pass their emotions to the tree by playing on the harp.


Creating: The interaction part

In the interaction part, we decided to create a harp which looked like a tree. Since users were going to interact by touching the string, the best sensor to sense this motion was 3-axis accelerometer sensor. We used 4 sensors to separately sense the motion of the four strings. Nan was responsible for making the harp and connecting the circuit (See Nan’s documentation).

Here are the pictures showing how he made it step by step:

(This is really beautiful)


Creating: The visual part

I was responsible for coding in this project. At first, I didn’t know how to draw a tree in processing, so I searched it online. I found that the basic idea of drawing a tree is recursion, or calling the function inside itself. It was not hard to draw a tree with simple lines after I understood this principle, but it is hard to make it beautiful. I search for some examples of trees on google, and I found a website called Tree Generation, which posts a lot of works of trees and flowers. I was attracted by a work called Magical Tree by Bárbara Almeida. The way her drew the tree and the leaves inspired me a lot. The tree looked very much realistic and the leaves consisting of large and small circles added a sense of mystery to the picture. I finally directly used her code for the brunch in my own project and started to think of drawing circles as leaves instead of drawing real leaves.

Since Nan and I made the Arduino part and the Processing part separately, I used the mouse and the keyboard to test my coding. The first step I wanted to do was to draw the leaves one by one when the mouse was pressed. However, I could only draw all the leaves at once (https://youtu.be/Vwuon2vLluM).

Then, I asked Nick for help. He changed the code a little and managed to draw the leaves one by one (https://youtu.be/BJ4ooUKKJV0). However, the problem occurred that the leaves would only appear from the left to right, and that the program would definitely go wrong after a while. As a result, I finally decided to draw the leaves randomly (https://youtu.be/eiVIgvYDbPE).

I originally planned to draw the leaves when the mouse was pressed and let them fade out when the mouse was released. However, I ran out of time so I was only able to let the leaves directly disappear when the mouse was released. This could be the thing I could fix when I further improve the project.

Since we planned to control the growth of leaves through the strings on the harp, the next step I did was to control the growth rate of the leaves by pressing different keys on the keyboard (https://youtu.be/Z0IAKLl_9oY). After I tried the code, I suddenly found that the changes were not obvious, and that when the leaves were drawn slowly, one even could not tell that these were leaves being drawn. So, I finally decided to change the color of the leaves when I pressed different keys.

After that I started to add sound to the drawing process. Nan had created four chords of different pitches played by a harp, and I let them separately play when different keys were pressed. The problem I met at this step was that the sound would be played again and again when I held the key, thus creating a weird echo. I turned to Jiwon for help and the problem was quickly fixed.


Creating: Combining the interaction part and the visual part

Now here came the combination of the interaction part and the visual part. I changed the mousePressed condition into the values of the four sensors. When they changed the value, the leaves would grow and the sound would be played. At first, we, met a problem that only the sound would play but the leaves were not drawn when we touched the string. We asked Louis for help but he was also unable to fix this problem. Since this problem occurred right before the due date, we considered plan B, which was using a separate button to control the tree. After we using the button, the problem was finally solved. This was also another point I could deal with when I further improve my project.

Here is our final outcome:


Conclusion

In order to make this project as good as we can, Nan and I went through a long and painful brainstorming process. It was very hard to closely combine the interaction with the visual as well as define the project topic and aim clearly. When creating the project, we also met with a lot of problems and finally had no choice but to change our original idea again and again. This problem-solving process was extremely intensive especially because it happened in the context of final week, but it also taught me a lot on how to properly deal with sudden crisis. 

Before we created the project, Nan and I had both agreed on the same project theme. Then, the task was separated clearly between Nan and me. I think this is why we cooperated so effectively on this project. Although we were working on different things separately, since we had the same orientation and the goal, we both contributed a major part to the final outcome.

In terms of the project itself, although it was completely different from what we thought at the very beginning, in my opinion, it was a really successful project. It demonstrated the learning outcome of the whole semester and also had a certain level of aesthetic values.

Finally, thanks to my really great and awesome partner Nan for his efforts all along the process of making this project. Also, special thanks to Nick, Jiwon, Louis and Sean for their help, and to Candy for the inspirations she gave me. 

 

//Processing - by Ellen
import processing.sound.*;
import processing.serial.*;

String myString = null;
Serial myPort;

int NUM_OF_VALUES = 5;   
int[] sensorValues;  


float n = 0; // noise input
PGraphics bg; // background
PGraphics tree;
ArrayList <PVector> leafs;
float minHue, maxHue;
int leafCounter = 0;
LEAF[] lf;
long framestuff = 300;
int valuefromArduino = 0;
PImage bgTree, bgTreeAlpha;
int sen1, sen2, sen3, sen4, leafgrow;

SoundFile sound1;
SoundFile sound2;
SoundFile sound3;
SoundFile sound4;



/*---------------*/


void setup() {
  size(900, 700); 
  frameRate(60); 
  smooth();
  fill(0, 4);
  //colorMode(HSB);
  
  setupSerial();
  
  sound1 = new SoundFile(this, "Sensor1.wav");
  sound2 = new SoundFile(this, "Sensor2.wav");
  sound3 = new SoundFile(this, "Sensor3.wav");
  sound4 = new SoundFile(this, "Sensor4.wav");


  tree = createGraphics(width, height); 
  leafs = new ArrayList<PVector>(); 

  createBackground(); 
  image(bg, 0, 0); // display background  
  createTree();
  image(tree, 0, 0);  //display tree

  save("tree.jpg");
  
  lf = new LEAF[leafs.size()];
  
  
  bgTree = loadImage("tree.jpg");
  

}


/*---------------*/


void draw() {
    updateSerial();
    printArray(sensorValues);
    
    for (int i = 0; i < lf.length; i++) {
    lf[i] = new LEAF(leafs.get(i).x, leafs.get(i).y, i, sensorValues[4]);
  }

    // add your code
    sen1 = sensorValues[0];
    sen2 = sensorValues[1];
    sen3 = sensorValues[2];
    sen4 = sensorValues[3];
    leafgrow = sensorValues[4];
    
    if (leafgrow == 1023) {
      
    if ((sen4 < 271 || sen4 > 279) || (sen3 < 200 || sen3 > 207) || (sen2 < 280 || sen2 > 290) || (sen1 < 270 || sen1 > 275)) {
      if (sen4 < 273 || sen4 > 279) {
          valuefromArduino = 1;
          sound4.play();
      } else if (sen3 < 200 || sen3 > 207) {
          valuefromArduino = 2;
          sound3.play();
      } else if (sen2 < 280 || sen2 > 290) {
          valuefromArduino = 3;
          sound2.play();
      } else if (sen1 < 270 || sen1 > 277) {
          valuefromArduino = 4;
          sound1.play();
      }
    }
    
    println(valuefromArduino);
    
    if (valuefromArduino == 4) {
      fill(random(50, 255), 0, random(0, 200), random(40, 100));
    } else if (valuefromArduino == 3) {
      fill(random(50, 255), random(150, 200), 0, random(40, 100));
    } else if (valuefromArduino == 2) {
      fill(0, random(100, 200), random(200, 255), random(40, 100));
    } else if (valuefromArduino == 1){
      fill(random(50, 140), random(50, 80), random(100, 255), random(40, 100));
    }
    
    lf[int(random(lf.length))].display();

  } else if (leafgrow < 1023){
    image(bgTree, 0, 0);
  }
}

/*---------------*/


void createTree() {
  tree.beginDraw();
  tree.noStroke();
  tree.background(0, 0);  // clear PGraphics (the tree drawn last time)
  for (int i = 0; i < 3; i++) {
    tree.fill(map(i, 0, 2, 60, 20));
    branch(width/2, height, 70, -HALF_PI, 150, 0); // draw the branches
  }
  tree.endDraw();
}


/*---------------*/


void branch(float x, float y, float bSize, float theta, float bLength, float pos) {  // (start pos X, start pos Y, Branch size, Angle, Branch length, 
  n += 0.01;  // increment noise input
  float diam = lerp(bSize, 0.7*bSize, pos/bLength);  // gradually reduce the diameter
  diam *= map(noise(n), 0, 1, 0.4, 1.6);  // multiply by noise to add variation

  tree.ellipse(x, y, diam, diam);
  if (bSize > 0.6) {
    if (pos < bLength) {
      x += cos(theta + random(-PI/10, PI/10));
      y += sin(theta + random(-PI/10, PI/10));
      branch(x, y, bSize, theta, bLength, pos+1);
    } else {
      leafs.add(new PVector(x, y));  // add a leaf at the intersection
      boolean drawleftBranch = random(1) > 0.1;
      boolean drawrightBranch = random(1) > 0.1;
      if (drawleftBranch) branch(x, y, random(0.5, 0.7)*bSize, theta - random(PI/15, PI/5), random(0.6, 0.8)*bLength, 0);
      if (drawrightBranch) branch(x, y, random(0.5, 0.7)*bSize, theta + random(PI/15, PI/5), random(0.6, 0.8)*bLength, 0);

      if (!drawleftBranch && !drawrightBranch) {  // if none of the branchs are drawn, draw a tip
        tree.pushMatrix();
        tree.translate(x, y);
        tree.rotate(theta);
        tree.quad(0, -diam/2, 2*diam, -diam/6, 2*diam, diam/6, 0, diam/2);
        tree.popMatrix();
      }
    }
  }
}


/*---------------*/


void createBackground() {
  bg = createGraphics(width, height);
  bg.beginDraw();
  bg.noStroke();
  for (float diam = 1.5*width; diam > 0.5*width; diam -= 20) {
    bg.fill(map(diam, 0.5*width, 1.5*width, 255, 210));
    bg.ellipse(width/2, height/2, diam, diam);
  }
  bg.endDraw();
}


  

void setupSerial() {
  printArray(Serial.list());
  myPort = new Serial(this, Serial.list()[ 3 ], 9600);

  myPort.clear();
  // Throw out the first reading,
  // in case we started reading in the middle of a string from the sender.
  myString = myPort.readStringUntil( 10 );  // 10 = 'n'  Linefeed in ASCII
  myString = null;

  sensorValues = new int[NUM_OF_VALUES];
}

void updateSerial() {
  while (myPort.available() > 0) {
    myString = myPort.readStringUntil( 10 ); // 10 = 'n'  Linefeed in ASCII
    if (myString != null) {
      String[] serialInArray = split(trim(myString), ",");
      if (serialInArray.length == NUM_OF_VALUES) {
        for (int i=0; i<serialInArray.length; i++) {
          sensorValues[i] = int(serialInArray[i]);
        }
      }
    }
  }
}



//LEAF
class LEAF {
  float x, y;
  float size;
  color c;
  float jitterX = random(-40, 40);
  float jitterY = random(-40, 40);
  float alpha = 255; //random(10, 40);
  float h;
  float minHue, maxHue;
  float leafgrow;
  
  LEAF(float posx, float posy, float col, int button) {
    x = posx;
    y = posy;
    size = random(0, 20);
    leafgrow = button;
    
    float rdn0 = random(255);
    float rdn1 = random(255);
    minHue = min(rdn0, rdn1);
    maxHue = max(rdn0, rdn1);
  
    //c = color(0, random(50, 255), random(0, 200));
    h = map(col, 0, leafs.size(), minHue, maxHue);
    
  }
  
  void display() {
    //fill(h, 255, 255, alpha);
    noStroke();
    if(leafgrow == 1023) {
    ellipse(x + jitterX, y + jitterY, size, size);
    }
  }
  
  
}



//Arduino - by Nan
void setup() {
  Serial.begin(9600);
}

void loop() {
  int sensor1 = analogRead(A1);
  int sensor2 = analogRead(A2);
  int sensor3 = analogRead(A3);
  int sensor4 = analogRead(A4);
  const int buttonPin = analogRead(A0);

  Serial.print(sensor1);
  Serial.print(",");
  Serial.print(sensor2);
  Serial.print(",");
  Serial.print(sensor3);
  Serial.print(",");
  Serial.print(sensor4);
  Serial.print(",");
  Serial.print(buttonPin);
  Serial.println();

  delay(10);
}

One thought on “Final Project Documentation (Ellen)

Leave a Reply