Final Project: Mouth Shooting Game/Wenyi Liu(Jenny)/Dan

Project: Mouth Shooting Game

Student: Wenyi (Jenny) Liu

Partner:Ruoxiao(Lily) Yu

Materials: Processing, FaceOSC, Arduino, in-built camera, joystick, wires


What it is?                                                                       2016-12-11-18_29_36-microsoft-edge

A shooting game

which is controlled by

the movement  of mouth and hand


It is  a classic shooting game, having key elements like left and movement, fire, hit and score.

But we use FaceOSC to detect and quantify the face, and then convert it into Processing so players can use mouth  open and close use movement  to control fire and start and restart.

And we use joystick (and the communication between Arduino and Processing )to replace mouth or keyboard to control the left and right movement.

2016-12-11-18_28_28-microsoft-edge img_20161211_182301

We intend to refresh classic games with more interesting interaction Continue reading

//use mouse to control the cannon  

import oscP5.*;
OscP5 oscP5;
import processing.serial.*;
import processing.sound.*;
Serial serialPort;
// Define variables for storing x and y angle
float angleX, angleY = 0;
// num faces found
int found;
int b=1;
int k=1;
SoundFile file;
SoundFile file2;
PImage img;
// pose
float poseScale;
PVector posePosition = new PVector();
PVector poseOrientation = new PVector();

// gesture
float mouthHeight;
float mouthWidth;
float eyeLeft;
float eyeRight;
float eyebrowLeft;
float eyebrowRight;
float jaw;
float nostrils;

int remain=15;  
int credit=0;  
int hit=0;  
PFont font;  

int m=330;
cannon a=new cannon((int)random(20,500));  
cannonball[] c=new cannonball[15];  
plane p=new plane(0,(int)random(20,200));  
void setup(){  
  file=new SoundFile(this, "file.mp3");
  file2=new SoundFile(this,"file2.mp3");;
  oscP5 = new OscP5(this, 8338);
  oscP5.plug(this, "found", "/found");
  oscP5.plug(this, "poseScale", "/pose/scale");
  oscP5.plug(this, "posePosition", "/pose/position");
  oscP5.plug(this, "poseOrientation", "/pose/orientation");
  oscP5.plug(this, "mouthWidthReceived", "/gesture/mouth/width");
  oscP5.plug(this, "mouthHeightReceived", "/gesture/mouth/height");
  oscP5.plug(this, "eyeLeftReceived", "/gesture/eye/left");
  oscP5.plug(this, "eyeRightReceived", "/gesture/eye/right");
  oscP5.plug(this, "eyebrowLeftReceived", "/gesture/eyebrow/left");
  oscP5.plug(this, "eyebrowRightReceived", "/gesture/eyebrow/right");
  oscP5.plug(this, "jawReceived", "/gesture/jaw");
  oscP5.plug(this, "nostrilsReceived", "/gesture/nostrils");
  serialPort = new Serial(this, Serial.list()[1], 9600);
  //font=createFont("font1.ttf", 50);  
 // textFont(font);  
  for(int i=0;i<15;i++)  
     c[i]=new cannonball();  
void serialEvent(Serial serialPort) 
  // Reads from the port into a buffer of bytes up to and including a particular character.
  // If the character isn't in the buffer, 'null' is returned.
  String inputString = serialPort.readStringUntil('n');

  if (inputString != null)
    // Removes whitespace characters from the beginning and end of a String.
    // In addition to standard whitespace characters such as space, carriage return,
    // and tab, this function also removes the Unicode "nbsp" character.
    inputString = trim(inputString);

    // The split() function breaks a string into pieces using a character or string as the divider.
    // filling substrings into a float array
    float[] values = float(split(inputString, ","));

    // we are waiting for two elements
    // put the numbers in the values array-variable
    if (values.length >= 2)
      // Re-maps a number from one range to another.
      angleX = map(values[0], 0, 1023, 0, 1);
      angleY = map(values[1], 0, 1023, 0, 1);
    if (angleY>0.69&&m<570){
  if (angleY<0.51&&m>30){

// Called directly after setup() and continuously executes the lines of code contained inside
// its block until the program is stopped or noLoop() is called.
// The draw() function is called automatically and should never be called explicitly. 

//void mousePressed()
  //int i=0;  
   // for(i=0;i<10;i++)  
    //  if(c[i].flag==1) break;  
   // }  
   // if(i==10) return;  
   // remain--;  

void draw()  { 
  text("Open Your Mouth" ,80,200);
  text("To Start",200,250);
{  k=k+1;
  if (mouthHeight<5&&k==2){
  if (k==3){
{  b=b+1;
  if (mouthHeight<5&&b==2){
  if (b==3){
  int i=0;  
      if(c[i].flag==1) break;  
    if(i==15) return;[i]);  

 // println(angleY);
  for(int i=0;i<15;i++)  
  for(int i=0;i<15;i++)  
 // if(hit==1)  
 // {  
  //  hit=0;  
  //  int j=5;  
  //  for(int i=0;i<10;i++)  
   // {  
    //  if(c[i].flag==-1)   
    //  {  
     //   c[i]=new cannonball();  
     //   j--;  
     //   remain++;  
     //   if(j==0) break;  
     // }  
  text("Remain: "+remain,450,50);  
  text("Credit: "+credit,450,100);  
    text("Run Out Of Ammo",90,150);  
    text("Final Score: "+credit,110,200); 
    text("Try Again?",170,250);
    text("Open Your Mouth ",100,300);   
    text("To Restart",170,350); 
     for(int i=0;i<15;i++)  
       c[i]=new cannonball();  


public void found(int i) {
//println("found: " + i);
  found = i;

public void poseScale(float s) {
 // println("scale: " + s);
  poseScale = s;

public void posePosition(float x, float y) {
 // println("pose positiontX: " + x + " Y: " + y );
  posePosition.set(x, y, 0);

public void poseOrientation(float x, float y, float z) {
 // println("pose orientationtX: " + x + " Y: " + y + " Z: " + z);
  poseOrientation.set(x, y, z);

public void mouthWidthReceived(float w) {
  //println("mouth Width: " + w);
  mouthWidth = w;

public void mouthHeightReceived(float h) {
 println("mouth height: " + h);
  mouthHeight = h;

public void eyeLeftReceived(float f) {
 // println("eye left: " + f);
  eyeLeft = f;

public void eyeRightReceived(float f) {
 // println("eye right: " + f);
  eyeRight = f;

public void eyebrowLeftReceived(float f) {
  //println("eyebrow left: " + f);
  eyebrowLeft = f;

public void eyebrowRightReceived(float f) {
 // println("eyebrow right: " + f);
  eyebrowRight = f;

public void jawReceived(float f) {
  //println("jaw: " + f);
  jaw = f;

public void nostrilsReceived(float f) {
  //println("nostrils: " + f);
  nostrils = f;

// all other OSC messages end up here
void oscEvent(OscMessage m) {
  if(m.isPlugged() == false) {
    println("UNPLUGGED: " + m);
class cannon  
  int x;  
  cannon(int x)  
  void display()  
  void fire(cannonball c)  
 class cannonball  
  int x;  
  int y;  
  int flag;  
    flag=1;//can be fired  
  void fire(int x,int y)  
  void fly()  
    if(flag==-1) y-=5;  
  void display()  
class plane  
  int x;  
  int y;  
  int flag;  
  int speed;  
  plane(int x,int y)  
  void fly()  
  int t=0;
  void crash(cannonball c)  
    {  t=t+1;
  void display()  

Final essay about Interaction and project/Jenny/ Dan

Final Essay

about my understanding of Interaction and my plan abot the final project

Part one

I think interaction is a process and also a result of effective communication and practical application among several objects or people. To communicate to understand and connect and then to apply to make cool things happen. People nowadays are also trying to make it more interesting by  making good use of multiple senses, big data, UE, computer vison and so on, and make it  more accessible to ordinary people by using computer language, simple kits and so on. I think this is exactly what our course is about. All the learning about Processing, Arduino, sensors, actuators, UE, computer vision and digital fabrication are intended for us to do interesting and practical designs to get involved and enjoy the interaction among humans, outer world and computer. And in many cases,  computers help quantize our senses, kits and techniques help create corresponsive reactions, designs help make interesting and useful effects. Besides,  audience-oriented designs and improvements make interaction nowadays even more effective and joyful.


Part two

For the final project, Lilly and I plan to make a simple interaction game.  We will be using Face OSC which we found very interesting in the class because it’s real-time and simplified depiction of faces and can be made fun of if properly used. We did some research and found that it can talk to Processing and the position information in the code of building the between the communication of these two was data which we could make good use of. We plan to use Face OSC to detect the position apparatus (now we are mostly thinking about eyebrows), then transmit the data to Processing, make the images of apparatus shown in the window and let the data to go in the code of the game. The game will be catching people jumping off the roof which is on fire. To be specific, we are thinking about people moving the eyebrow to change its position (mostly up and down) which means the position of air cushion in the game and we also plan to add a joystick to control the other direction of eyebrow(only left and right).Altogether, it would be a game of people dancing with their eyebrows and turning the joystick to control the position of air cushion in the game to catch people and then get credits. To achieve this, we will be using Arduino, Processing and Face OSC; we will try to make it look like a game which means adding image, audio effects and so on; we will also try to do some research of UE and take the audience’s opinions into account.

Face OSC and Procesing

2016-11-28-14_41_42-interaction-lab-22-computer-visionn-pdf-%e2%80%8e-microsoft-edge  (From in-class presentation)

Brainstorm                                                        Draft for prototype

img_20161128_143351                                      img_20161128_143345                        

Good luck with my final project.

Lab 8/Jenny/Dan

Project: moving an image using a face that the computer detects

Date: 18th. Nov. 2016

My code for one face:

1 2 3

Before going to the lab, I downloaded opencv library and got my in-built computer camera working. In the class I used the sample code of face detection to detect face first. Then I loaded image and displayed in the draw loop. The size of picture first didn’t change with the size of face in the camera, the I defined gloabal variable to show its position and scale and made in the image() function and it worked. Then I also tried to dectect two faces and use the image to cover both faces. I wrote the x and y postitions into a 2 length array and if I draw rectangles, they it has two unlike the blinking  two with the previous code. However when I dislay picture, there was an erro and I am still working on it.


My imcomplete code

1 2016-11-18-16_10_39-list-_-processing-3-2-1 2016-11-18-16_11_05-list-_-processing-3-2-1 2016-11-18-16_11_50-list-_-processing-3-2-1


Lab 7/ Pace/Jenny/Dan

Project: Pace

Materials: Arduino, board, wires, potentiometer, gifs and music from the Internet

Description: By turning the potentiometer, it goes from walking to riding horse, to cycling and then to driving, the background mucic goes faster and faster.


My  Arduino Code                                        My Processing Code


I first paried up with Penny to do the project with the pics and mucis I collected before the class. She did the circuit and used the code offered to tranmit data from Arduino to Processing. However we ran into a problem that the numbers seemed fine in Arduino, but in Processing, they are never stable. Luisa helped up debug this for a very long time. She improved it by interpreting data into number to be used in my written conditionals and used trim and string. I got part of it. Then I got what was left and did the others.


I still wanted to look for a simpler version that I can understand better. Then I found that dividing the data from the sensor in Arduino in the original code could just work. I guess that’s what we neglected although I Learned it befor. Then, I imprted sound library, gifanimation library to import music and gifs. I had some trouble finding the zip file of gifanimation library copatible with 3.0 and then found it just on Github. Then I used the functions like play(); rate(); image(); and so one to continue my project. Then it took some time but went quite smoothly. In the end when I was almost done I found that after running some time there is an erro in my sketch, then I realised I should add loop() to go on palying the music.


Lab 6/Jenny/Dan

Date: 4th Oct 2016

Partner: Sumner

Projects: using Arduino to control the moving of motors to creat drawing machines


  • 1 stepper motor
  • 1 SH75440NE ic chip
  • 1 5 volt power supply
  • 1 power plug
  • 1 potentiometer
  • 1 Arduino and USB cable
  • Laser-cut mechanisms
  • Pens that fit the laser-cut mechanisms

The Circuit to control  one steeper motor

It went quite smoothly, since it was just making a curcuit according to the map below.

But I didn’t know what the motor supply was for and when I unplugged it, it still worked anyway. And Bily saild its was to make the motor move more stably and kind of answed my problem. And there was some problem with our motor, when we moved it to another palce to make two work together, it stoped rotated, only wildly viberated. We got help from Antonious, the first time was beacuse one wire was loose, the second time was because the power was not connected on one side of the board but we needed it to be since we added the potentiometer with power coming in from the other side.


Then, there was another part happening in Arduino code. Without potentiometer first, we just made it move with a preset speed which was 60 with code stepper_oneRevolution borrowed fron the library, since it went overly wild,  I also changed it into 30 for further use. We also borrowed the code stepper_speedControl to use potentiometer to control the speed of its rotation with the curcuit in which potentiometer was added.

There was also another part which was to connect the laser-cut mechanism and make it hold the pen stably. To do this, my partner did some slight adjustments like the connection of them and the other group working with us also changed the height of the paper where to draw. In the end, we found a relatively nice speed and make them work to draw.

So, the third vedio was  what our one motor works like.

The second was the wild one of two working together.

The first one was them with a adjusted speed.

first is the wild one of two working together.

The third is the one with potentiometer set with a speed.

/Midterm Project/Jenny(Wenyi) Liu/Dan

Project: Lantern event in an old-time

by Jenny and Lily

This is a simplified interactive installation that imitates the scene of lighting Chinese lanterns in an old town. It also demonstrates the environment of the town and the status of the lanterns and guides people to do the lantern event.


Inspiration: The tradition of lighting lanterns to send wishes in China and its adoption by old-town tourism make us want to recreate the scene  and tell when to light lanterns and how they are like. In addition, it serves to make people engage to the event.


Materials: Processing, Arduino, leds, wires, button, potentiometer


Demo: img_0869

Inspiration: The tradition of lighting lanterns to send wishes in China and its adoption by old-town tourism make us want to recreate the scene  and tell when to light lanterns and how they are like. In addition, it serves to make people engage to the event.

How it works:

❖Turning the potentiometer changes the time.

❖When it comes to night, the green light is on and it’s a “go” sign to light the lanterns.

❖Now if  you press the button, the lanterns will float in the sky.

❖Meanwhile you can change the night darker still by turning the potentiometer  and the red light shows the relative brightness of lanterns.

❖Besides, the lines down in the picture instruct tourists to get involved in the lantern event.

Its possible use and its possible updates:



A: A  game about draging lanterns by mouse and get credicts

In the processing ,we found the moverment is rather rigid and the thing is rather boring and lacks interaction and lanterns can be replaced by any other thing here.

B: Pretty much the same, but if we push the button once, we get one more lantern by add arraylist.

But the data button reads is more than 1 when we press it once, so what shows in processing is not exactly what we want.


1:Latterns: We start our project with the very element, Chinese lanterns, so I drew a draft first, and then draw it in processing with several adjustments. It first came with line function, but I had to fill color, then I had to chang it into vertex function. What’s more, to make it look nicer, I borrowed the oppacity data frlanternsclassom the processing forum. In order to later make the lattern combine with its animation, I rewrite it in a class under another tab and also add movement to it, which was learned right after I drew this out.


2. Background:I found a very nice picture of an old town online. In order to tell the time difference, Lily editted the picture with diffrent colore contrasts. I learned the instruction of inserting gifs into processing online and then wrote the code in own processing.oldtown1oldtown2oldtown3oldtown4 oldtown5

3.The circuit on Arduino

We wired Two leds, a button and a potentiometer together, which went quite smoothly.

And because of the UE feedback from our friend: it was terrible to both see things and control two things in a messed up board, I tried to organise the wires but it was not ideal enough, then we put them onto two boards with leds and sensors seperated. The very right one is one we use in the end, although the pic was shot from a unwise angle.


img_20161024_210813 img_20161024_211856img_0873

4.The code in Processing and Ardunio



I borroed the code of  SimpelCallResponse to make it first work to send data. But due to delay, unstability of Arduino board, it was not ideal enough, this problem was solevd by adding more codes like string by Lily in the lab with the help of fellows.

We together wrote all the conditions for the potentiometer to control the background and the button to control the appearance of the lanterns. Since I drew the lanterns and so set the limits of their positions and made them conbined with the class where I used arrays and also tried to use arraylist for prototype 2 but failed in the end because of the effects. Lily added the code in Arduino for the potentiometer to change the status of leds and inserted fonts in Processing.


Possible updates:

❖The animation of lanterns

❖More user involvements

❖The effects of lights and the use of buttons

Lessons learnt:

  1. It is wise to break down the project, but when we try to combine them together, we  will easily run into problems like the problem we met in Prototype 2 and loops under loops. This takes patience and really requires us to know the structure of what we are doing
  2. Using tabs and comments can really simply the code and help debugging later. The most used way of debugging is to print the value seperately and go step by step.
  3. We really can find a lot of instructions online and get a lot of help from the fellows and faculty. So don’t  hesitate to ask.                                                                                                                           AND SPECIAL THANKS to: Dan, Antonious, XY, Louis, Eric






Field trip one: Maker Carnival/Wenyi(Jenny) Liu/Dan

Today we went to the maker carnival and saw many interesting projects.

There are several  projects connected to 3D printing and it’s good to see how they simplift those sketching and printing process to make it in toy production and so on.


There are also several things connected with carpentry. They basically work on two things, one is to clear the dust as much as possible, one is to make it as simple as possible for people to do small things. There is also one which I am very intereted in. It especially designs tools to make make holes and gives instructions to make wooden project with traditional Chinese tenon-and-mortise technic.

Then, it comes to Arduino.

This time I at least have some idea what kind of sensors they are probably using and how the Processing talks to Arduino to do all the interesting stuff.



Lab 5/Communication between Arduino and Processing/Wenyi(Jenny) Liu/Daniel

Partner: Zihan Feng

First we made a circuit on the breadborad. One part is to send the digitalRead HIGH or Low of switch to Processing, the other part is to have a led and a resistor to protect into which can be controlled  by the processing. And they do not affect each other when working. So this circut is used in both projects.img_20161014_135516img_20161014_140648

Project 1: use the switch on Arduino to control the color of the spuare in processing

First, we used the code simpleRead from library in processing and moved the last part of the code to Arduino.

We then changed the declaration of pinmode number to 13 which was connected with our switch and then uploaded the sketch.

Afterwards, we ran the processing and it worked. Once we press the button, the square turned grey and when we relessed it, it turned blak.

Project 2: use mouse postion to control the led on Arduino

We used the code simpleWrite in Procssing and moved the last part of the code to Arduino.

Then, we changed the pinmode to 3 which was connected with our led and then uploaded it.

Afterwards we ran the sketch and saw that when mouse is over the square in Processing, the led is on, otherwise, it’s off.


At last, we considered the combination of Processing and Arduino and found it complicated things sometimes. Then, we thought about how to transfer multiple data in betwwen instead of simple digital one. XY showed me the code SerialCallRespons in which Arrays were used to store data and we were also told that we should change the code according to our needs and the use of sensors.

Lab 4/Distance sensor and Led/ Wenyi(Jenny) Liu/Daniel

Name:Wenyi(Jenny) Liu


Date:30th Sept

Project: using the data from the distance sensor to control the led

Materials: Led, Wires, Resistor, Breadboard, Arduino

Fist I borrowed the sensor and connected to my Arduino with the guidence online and then copied its sample code making the Arduino read its number. Then I pluged it in and found it working.

Then I moved on to the actuator part in which the distence controls the led whether is on or not.

I added a if conditionals into the loop. Since recently I’ve been learning stuff about Processing, it really took me some time to realise that its format is quite is slightly different and I had to declare its pin and digital or analog read and write. Then I made the cuicurt on the board. I first shortcircuited the led and corrected it right after Richard pointing it out and then it worked but not as I expected. I mean my code was to let the led on when I was near enough. However it went the opposite way.

void setup () {

Serial.begin (9600);

pinMode (A0, INPUT);


void loop () {

uint16_t value = analogRead (A0);

// uint16_t range = get_gp2d12 (value);

Serial.println (value);

//Serial.print (range);

Serial.println (” mm”);

Serial.println ();

delay (30);

int led = 9;//led is attached to 9

pinMode (led, OUTPUT);

if (value<=40) {

digitalWrite(led, HIGH);

} else {digitalWrite (led, LOW);


uint16_t get_gp2d12 (uint16_t value) {

if (value < 10) value = 10;

return ((67870.0 / (value – 3.0)) – 40.0);


To solve this problem in the easiest way which was also the IMA way as Richard said, I just change value<=40 into>=40, and it went perfectly fine. Yeah, I got what I wanted.