Final Project: the visible invisible

Here is an introductory video of my final project:

I was inspired by an artwork Clinamen by Celeste Boursier-Mougenot and would like to create a new means for the viewer to experience their selves. Initially I wanted the user to appear as one “plate ” among a group of plates and get a sense of self through the clinking sound they make when colliding with the other. However, later I realized this gives the user too little agency and they might lose their uniqueness along the way. After a lot more thoughts on possible ways to discover people’s ecological selves, I referred to the codes we learned from week 6 about the text rain and managed to generate waves of falling “balls” that could stay on viewer’s bodies and thereby reveal their existence in front of the dark canvas. I name it “the visible invisible” because I think sometimes when people feel invisible and a low self-esteem, they might feel better if they think about their interactions with the surroundings. No man is an island.

One of the problems I didn’t expect to encounter was actually the color choices. Initially I set the background white and falling ellipses blue because I’ve always found it very beautiful and romantic when the snow falls on someone’s shoulders on a crispy winter day. However, after user testing, I realized the color choice reduced the immersion of my project and it didn’t fit well with the volatility of the ellipses and the chilling background music I chose to match the main theme. After a few trials, I settled on this set of colors, pink, light turkish blue/green, and dark turkish blue/green so the display wouldn’t be that eerie. As it’s now more casual and vibrant, I also noticed that users are more willing to free themselves, move around, and play with my project.

I think this coincides with what I stated in my manifesto: “We generate the simplest things. The complexity lies in our choices. The complexity lies in interpretations. The complexity like in chance.” Technically I generate literally the simplest things, waves of ellipses, but every user would get different experiences depending on their performances and interpretations.

Another difficulty I had to deal with was the brightness detection. Because the lighting in this academic building isn’t ideal, and I wasn’t able to borrow a big enough screen, at first the project didn’t work stably because I used Cici’s methods to detect the brightness/darkness. To solve that problem, I’ve tried to offer users black coats so they could be seen by the computer vision. I changed it to color detection later so now as long as the users are colorful against a white-ish wall, which could mean dark hair or even bright color clothing, they could be detected by the computer.

I would like to continue working on sounds. So far I only have one single background music, but in the future I hope it could be manipulated by users’ movement. I’ve thought about making this project as an instrument so everytime the user moves to a new location, a new tone would be generated. However, I also thought it deviated from my initial idea of giving users a chance to experience “self”. I am still working on possible ways to bring out one’s existence through music.

Capture webcam;

import processing.sound.*;
SoundFile file;

ArrayList balls = new ArrayList();

int startTime = 0;
int timeIntervalFlag = 4000;
int lastTimeCheck;

int bMax = 300;

boolean fall = false;

void setup() {

  webcam = new Capture(this, 640, 480);
  lastTimeCheck = millis(); 

  file = new SoundFile(this, "sound1.mp3");

void draw() {
  if (webcam.available() == true) {;

  scale(2.25, 1.875);


  for (int i = 0; i < balls.size(); i++) {
    Ball b = (Ball) balls.get(i); 
    if (b.pos.y<=0 ||b.pos.y>=height) {
  if (millis() > lastTimeCheck + timeIntervalFlag) {
    fall = true;

  if (fall == true) {

    for (int j = 10; j <= width; j = j+10) {
      if (balls.size()<bMax) { 
        balls.add( new Ball(j, 0));

    lastTimeCheck = millis();
    fall = false;

class Ball {
  PVector pos;
  PVector vel;
  float rad;

  Ball(int x0, int y0) {
    pos = new PVector(x0, y0);
    vel = new PVector(0, 1);
    rad = 10;

  //boolean detectCollision(PImage cam) {
  //  color slice = cam.get((int)(pos.x), (int)(pos.y));
  //  return(brightness(slice)<=128);

  boolean detectCollision(PImage cam) {
    color slice = cam.get((int)(pos.x), (int)(pos.y));

    float r = red(slice);
    float g = green(slice);
    float b = blue(slice);


  void display() {
    if (pos.y<height/7) {
      fill(230, 100, 150);
    } else if (pos.y<height/4 && pos.y>height/7) {
      fill(50, 170, 170);
    } else if ( pos.y>height/4) {
      fill(30, 70, 70);

    ellipse(pos.x, pos.y, rad, rad);

  void move() {
    if (detectCollision(webcam)) {

Week 7: 3 Mosts

  1. My most hair-raising moment in vr was when I first started playing superhot and someone (in the game) approached me from behind. Because it was virtual reality and our perceptions depend most of the time only on visual and audio inputs, I couldn’t feel the faint breeze or hear anything (if the game didn’t want me to) when someone’s getting closer, it really got my hair raised when I turned around and “ran” into someone. I think vr has a lot of potentials in the horror industry.
  2. One of the most novel innovations was the usage of vr in court, which I read about in news of the week ( Vr technology allegedly helps to “present evidence in a clearer, more visual way” than traditional media such as powepoint slides. On one hand I am astonished and confident about the help vr could bring to the courtroom, but on the other hand I am also worried that vr might be a more powerful tool to mislead and manipulate the court.
  3. Another interesting moment I had in vr was when playing long echo and for the first time I looked down and saw my “body” in a vr game. I was used to “overlooking” the presence of my body and only seeing parts I had a need for, most of the time hands. It felt a little odd to see myself in a spacesuit with the whole body and yet didn’t have any control over it. I wonder when vr becomes more prevalent, will it change our perceptions of our selves?


On Generative Art

  1. Generative arts are complicated.
  2. We generate the simplest things.
  3. The complexity lies in our choices.
  4. The complexity lies in the scale.
  5. The complexity lies in interpretations.
  6. The complexity lies in subtlety and unconsciousness.
  7. The complexity lies in the culture.
  8. The complexity lies in shared humanities.
  9. The complexity lies in time.
  10. The complexity lies in chance.
  11. A generative art piece could be massive, but not redundant.
  12. A generative art piece could be concise, but not simplified.

On Generative Systems

  1. Are they representations?
  2. Are they merely tools?
  3. Are they maths?
  4. Are they autonomous?
  5. Are they independent?
  6. Are they in order?
  7. Are they manually randomized?

On the Notion of “Us”

  1. “Us” is where an artist comes from, but not where their emotions reside.
  2. “Us” is what an artist feels both familiar and unfamiliar with.
  3. “Us” is what an artist relate to, but also what they fear they cannot run far away enough from.

On Artist

  1. An artist is both understandable and strange at times.
  2. An artist is full of sentiments yet insensitive.
  3. An artist is smart yet so stupid.

On Artworks

  1. They want to be appreciated by everyone.
  2. They don’t want to be appreciated by everyone.
  3. They all think they are the artist’s no.1 baby.
  4. And yes they are.
  5. Or they might not be.
  6. They are nothing more than means for an artist to open up, get emotions out of the chest in single drops, sometimes in a profusion, and exchange for a closure.
  7. Maybe they become the closure along the way.

HW 6: Design for Physical prototyping

After a few discussions, Mari and I decided to focus on the exterior shape first and work on the interior mechanism later. Therefore, we referred to a shoe bag product we found on Taobao ( and added straps to it. At the end of the straps, there are fasteners so the bags could be attached to bags or stand as an independent handbag/shoulder bag themselves. We also allow the length of the straps to be adjustable so if they are added to a bag they can hold fast, or if they are to become shoulder bags they can be comfortable to carry. We kept the original design of the zipper.

Then I used Tinkercad to fabricate it. Here is what I got:

It’s too complicated to recreate the fasteners and length adjusters here, so I focused primarily on the shape of the bag once it’s loaded. I think this model can help us to further consider the aesthetics and comfort aspects of how this bag could connect to other bags.

Week 6: Reviews of VR Titles (old + new)

4 Old


Superhot VR – Oculus Rift – FPS – 9

Superhot is super fun! I really like it that the people are made of red glass and there are multiple ways to defeat them, such as hitting, shooting, or throwing darts or other objects on them. It was also quite interesting that the time slows down when a bullet is shot and the users can get a chance to dodge it. The game is very immersive and intense; I found myself sweating so much after playing it for 20 minutes. #slomo #action #superhot #adrenaline #glassmen

Long Echo – Oculus Rift – Game – 8

Instead of clicking on certain points or using a joystick, users move forward by holding onto “hard” object and pulling or starting the impetus devices on their wrists. It is a new (for me) experience of moving, and I appreciate this innovative idea. Also, I like it that the instruction is very clear and precise. It’s definitely been fun to experience zero gravity and working in a space station. #space #zerogravity #ai #training

Lost – Oculus Rift – Short Video – 8

I was amazed to discover the beast in the bush was a giant steel hand, and I definitely didn’t see its owner coming. It felt quite real when the gigantic robot looked down on me, and I was glad that the story ended there. #giantsteelhand #giantrobot #jungle #realistic

Transition – Oculus Rift – Experience – 7

I really love the graphic design of it. It’s so beautiful and almost epic. But it’s also too repetitive and slow in terms of the storyline if there’s one. I wish there wasn’t a dog that jumped to my boat and the boat could be smaller that only me could fit in, so I could have a more isolated space to experience and meditate. #boating #peaceful #whyadog?

Accounting – HTC Vive – Game – 7

#likethevoice #difficulttoplay #needmoreclue

In the Eyes of the Animals – Samsung Gear – Experience – 8

#amazing #immersive #beautifulsounds

Notes on Blindness – Samsung Gear – Experience – 7

#calming #slightlyconfusing #neonparticles #wheresfootprint?

4 New

The Night Cafe – Oculus Rift – Experience – 7

Who doesn’t love Van Gogh? It was so nice to be placed in Van Gogh’s world and see the objects of his masterpieces. It’s a shame that users can’t go outside of the room to see the starry night, and it got a little creepy when I had eye contact with Van Gogh and when I accidentally walked into him, but other than these, amazing experiences! #tributetovangogh #restricted #walkintoppl #delightful

Rick and Morty – HTC Vive – Game – 8

This one is more interesting and interactive than Accounting in that almost everything you touch or play with provides some feedback. The tasks “I” was instructed to were senseless, which actually might bring joy to Rick and Morty fans. Still, the cues were not enough and it’s difficult to play. #pranks #darkhumor #difficult #everythingisinteractive

Oz Chicken Slayer – Google Daydream – FPS – 7

Not sure if it’s a problem with the game or if it’s just the controller happened to be a little off when I played the game, I found it difficult to aim precisely at the chicken no matter how many times I tried to adjust the controller. This definitely influenced my experience, but apart from that, I think the green and chubby chicken is pretty cute. #greenchicken #difficulttoaim #slowmo

The Path to Luma – Google Daydream – Game – 7

Ugh the instructor talked too much! This game is easy and simple, so I think after some basic tutorials, the users could totally carry on without this many hints. I felt slightly uncomfortable from time to time because “I” was placed too low as if “I” was a dwarf, or the surroundings were too close to “me” that I felt suffocated. #cartoon #puzzle #toomuchinstruction #repetitive

Assignment 5: Techno-lization

Over the weekend I went with the class to the Museum of Contemporary Arts, where I spotted a bag design that caught my eye.

I really like the design of it, especially the combination of concrete and see-through rubber. However, I also found it a little dull and heavy looking, so I used Processing to make it a little more dynamic. Tint and Copy were the two primary effects I applied, and here it is:

(Short video: amym5)

I think compared to the original picture I took, this new version is less static and much more vibrant with a techno vibe. In the future if possible, I would like to add sound effects to it.

PImage img;

void setup() {
  size(1200, 600);
  img = loadImage("img5.jpg");

void draw() {
  image(img, 0, 0);
  tint(210, 30, 30, random(50, 100));
  int i = round(random(0, 560));
  copy(i-20, 0, i, 600, i, 0, i+20, 600);

  image(img, 600, 0);
  tint(30, 30, 210, random(50, 100));
  int j = round(random(560, 1200));
  copy(j, 0, j+20, 600, j-20, 0, j, 600);

Week 5: 2 new vr titles

Rick and Morty: Virtual Rick-ality

This game would be super fun to try for rick and morty fans. Like job simulator, you would experience being a clone of morty and interacting with 3d rick. I had positive experiences with accounting before so I am really looking forward to another puzzle solving game but in the context of rick and morty.


SUPERHOT appeared to be an ordinary shooting game, but after watching the trailer, I was really fascinated by the special effect that time slows down when you attack. I have never used a gun, but in many of the movies and tv shows I have watched, there’s often some slow-motions when a bullet is shot, so funny enough, I find this slow-mo shooting closer to my “reality”. Would like to play.


Assignment 4: A collection of receipts and tickets

I used to collect receipts and tickets when traveling because I believed they marked all the highlights of my journeys, and they also helped me calculate the expenses. From time to time I looked back on those pieces of papers and recalling the sights I visited and the meals I had with family and friends. I don’t do that often when I am at school, but just to try it out I restarted at the weekend. Luckily it had been a quite eventful weekend and I had something to show.

Here I would like to present my data as if it were in a bullet journal. I listed events happened during the weekend with the exact/estimated time, items, and how much they cost. It’s not clearly shown in the scanned picture below, but all the free events were marked in blue. I glued what I collected under each item, but there are cases when I got an e-receipt or no receipt. 

And I folded most of the receipts/tickets because it would be more fun for my readers. I think it’s human nature to be “nosy” around other people’s life, and I personally love watching other people’s vlogs on youtube. I hope through unfolding the receipts/tickets, people can gain a sense of “peeking” into my life. Here’s another picture I took:

After processing and organizing my collection, I got two final remarks:

  1. My days start from noon on weekends…
  2. An eventful weekend might get expensive, but not all the good time cost money. For example, one of the highlights of this weekend was actually discovering a mysterious institute under the Pearl Tower. It was so coincidental and the view was so beautiful in the area. I also really enjoyed the late night walk with my friend along the Huangpu River and we talked so much that night! Both of these lovely events cost nothing and if there were no this class assignment, I probably would forget about them because they left no “trace”. It got me rethink my usual approaches to document the trips I had been on. I used to only consider the events that cost money and come with a paper of proof significant, but sometimes, the best time might come from activities that are totally free!

Reading Response 4: You Say Data, I Say System

Through this article, the author encourages us to not only look at the data, but also the system the data is in. It reminds me of the significance of standing on a higher ground to interpret the information we are given. One quote that really stood out to me in the article was “the true medium of data visualization is not color or shape; it’s the decision”. Without conscious choices, data is cold and meaningless, nothing more than a gathering of random pieces of information. We present data in the way we design because we take into consideration our audience characteristics, the message we try to convey on top of the contents of the data, and the effects our decisions could make. The data then has life and becomes unique and almost personal to the people who represent it.

Take this reading response as an example: all these words as data are shared among English users, but because I want to share my thoughts after reading the article, I try to articulate my thoughts through carefully picking these words and putting them in a certain order and hope my readers would understand my views. I could have changed the font size or style of some words if there were things I would like to emphasize, and those who agree with what I said above don’t resonate with the words (aka the data), but with me as the system that generates this data.

HW 3: Presentation Reflection

Here is my project proposal:

Reflecting back on my presentation, I realized there’re two things I should keep in mind in future project proposals:

Firstly, I came up with my project idea, a shoe bag for workout shoes during school days, because I have personally encountered the inconvenience and am looking for solutions. However, not every one of my audience has the same need, at least not as bad as me. Therefore, although this time I am only targeting a small group of people for a specific purpose, it might help if I present other needs, such as people carrying other kinds of shoes or other items, so a broader range of people can relate and become willing to involve in the problem-solving process.

Also, even though the prompt was not to give any solution yet, I found it quite difficult to implement. I understand we should do so to keep our minds open, but if there isn’t a greater design in mind, how can we identify the shortcomings of existing products? With that being said, we could work on presenting the problems in a more neutral and general way so we are not leading our audience to the solution we’ve come up with.