# Documentation for Cat’s Restaurant from Jingyi in Antonius’ class

Team member: me and me and me and me

My original idea was to create a Photo Booth with which people can take selfies. I brainstormed a lot of situations in which people can take interesting photos. For example, wedding or zoo. In order to get more inspiration, I downloaded almost all the Photo Booth I could find in Apple Store and tried to see if I could get any inspiration from them. It turned to be nothing. Then one day when I was browsing random pictures, I saw a cat and immediately I got this inspiration. I originally planed to use face detection and put a cat face on it. Then I wanted to use my hand to catch whatever falls down from the screen. But at that moment I had no idea how to realise what was in my mind so I talked to Antonius about it. Then he gave me some instruction that I could use colorTracking in opencv and a library called box2d. That was all I know before I started.

So here it comes, this is the first step to do this. I firstly used faceDetection and regularly things move down with the change of y.

IMG_6331

As you can see in the video, at this point colour tracking is already used. But it is not working very well because I haven’t got those special gloves yet. Also, I only used colour tracking for one colour. Later I will work on two colours.

Then I moved to the next stage.

I tried to combines things I had with box2d which is the most difficult part for me. I read through the chapter 5 of Code of Nature and then get throughly what how box2d works. As I saw in the sample code and also the textbook, there should be boundary which is stainable and particles which can bounce up.

So I just did it as the sample code did it.

problem 1

As you can see in the video, there are two problems at this point. The first one is there is a black line on the screen which is the boundary in the sample code. I planed to make it move with my hand later. The other problem is more obvious– things are going crazy. This is because the (0,0) point in box2d is is the middle of the screen, which is different from processing itself. And also, opencv has scaled the scene into a narrower vision.

After solving all these problems, I started to think about boundary. The first thing came into my mind was find the boundary of the colour pixels of my gloves and then make that boundary into the boundary that is stainable. However, this is too much work and also I didn’t now how to realise it. So another idea came up. I could use the colour to actually control something and then make that object stainable. The way to do this is to set the mode into STATICS instead of KINETIC.

Also, in the process, some other problems bumped up. For example, this one:

I spent a long time working on this because it seems to make no sense. Then Aven helped me with this. It turned to be because that only when the body I created which is banana, strawberry and cupcakes are killed, the next one would come out. So I couldn’t kill the body twice. After I deleted the killBody in the main tab, the problem was solved.

Then it was the problem of colour tracking. I didn’t  take video of that because that made no sense. Basically the situation is although the numbers are all correct, the claw wouldn’t follow my hand. I spent a long time figure out why. Firstly Professor Shiffman told me that the colortracking is not that sensitive so actually I couldn’t do too much to solve this problem. However, he told me that I could use LERP to make things smoother. This worked very well. But the object was still not following my hand. Then with drawing the points of the pixels of my gloves, I suddenly saw the problem. The reason why the claw was not following my hand was because it was only moving in a quarter of the screen ( the up and left corner). So now the problem became easier. It was also because of the size and scale.

After solved all these problems, the code run very well. So I though I should make it more interactive. Then I designed that when the claw touched something, for example a bomb, the game should be over. I tested it with banana.

IMG_6311

As you could see in the video, when the claw touches the banana, the program should turn to next page. But as is shown in the video, it is not sensitive. I tried to enlarge within which the two X meet, but it didn’t work very well.

Also you can see, now all the objects would fall and not stay in the screen. However, I think this is not fully showing the magic of box2d so again I created the boundary at the bottom of the screen which prevent the things from falling out of the screen from y direction. (The can still be swept out of the screen in x direction. Use your hand!)

This means that I need to change the condition which let the next object appears. (originally there should only be one element in every Arraylist.) So I changed it into once the Y is greater than certain value, the next element appears. In this way, there can be more than one banana or cupcake or strawberry on the screen.

The last thing I changed was the objects that could follow colour. I replace one of the claw with a plate, and then change the restitution value into 0 which means that the thing collide with this object would move in the opposite direction. Instead, it stops. But the restitution value of the other claw is still positive which means it will bounce things away. And then I added a cook hat on cat’s face.

Here is the final show:

IMG_6328

And here is the code:

import gab.opencv.*;
import processing.video.*;
import java.awt.*;
import shiffman.box2d.*;
import org.jbox2d.collision.shapes.*;
import org.jbox2d.collision.shapes.Shape;
import org.jbox2d.common.*;
import org.jbox2d.dynamics.*;
import org.jbox2d.dynamics.contacts.*;
import processing.sound.*;
SoundFile file;
// A reference to our box2d world
Box2DProcessing box2d;
// A list for all of our rectangles
ArrayList<Cupcake> cupcakes;
ArrayList<Banana> bananas;
ArrayList<Fish> fishes;
ArrayList<Leftclaw> leftclaws;
ArrayList<Rightclaw> rightclaws;

Surface surface;
float handX1 = 0;
float handY1 = 0;
float handX2 = 0;
float handY2 = 0;
Capture video;
OpenCV opencv;

color trackColor1;
color trackColor2;

PImage cupcake;
PImage cat;
PImage banana;
PImage fish;
//PImage strawberry;
//PImage bomb;
PImage leftclaw;
PImage rightclaw;
PImage hat;
PImage go;
PImage logo;

float y;
float d1;
float d2;
float d;
float s;
float t;
float o;
float p;
float q;
boolean state = true;

void setup() {
size(640, 480);
file = new SoundFile(this, “music.mp3”);
file.loop();

video = new Capture(this, width/2, height/2);
opencv = new OpenCV(this, width/2, height/2);
trackColor1 = color(255, 87, 55);
trackColor2 = color(112, 229, 129);
smooth();

// Initialize box2d physics and create the world
box2d = new Box2DProcessing(this);
box2d.createWorld();

// Turn on collision listening!
box2d.listenForCollisions();

// We are setting a custom gravity
box2d.setGravity(0, -100);
// Create ArrayLists
cupcakes = new ArrayList<Cupcake>();
bananas = new ArrayList<Banana>();
leftclaws = new ArrayList<Leftclaw>();
rightclaws = new ArrayList<Rightclaw>();
fishes = new ArrayList<Fish>();

surface = new Surface();

video.start();
//trackColor1 = color(255, 87, 55);
//trackColor2 = color(112,229,129);

frameRate(30);
}
void draw() {
if (state) begin();
if (!state) game();
}

void begin(){
background (255, 204, 0);
image(cat,231,180,192,236);
image(logo,82,47,1155,112);
imageMode(CORNER);
image(go,494,363,110,79);
if ( 490< mouseX && mouseX < 600 && 360 < mouseY && mouseY < 440) {
println(mouseX, mouseY);
state = false;
}

}
void game() {
pushMatrix();

scale(2);
imageMode(CORNER);
image(video, 0, 0 );
//filter(GRAY);
//filter(INVERT);

Rectangle[] faces = opencv.detect();
// println(faces.length);
for (int i = 0; i < faces.length; i++) {
//println(faces[i].x + “,” + faces[i].y);
//imageMode(CERTER);
image(cat, faces[i].x, faces[i].y-0.33*faces[i].height, faces[i].width, 1.5*faces[i].height);
image(hat, faces[i].x-0.17*faces[i].width, faces[i].y-0.8*faces[i].height, 1.2*faces[i].width, faces[i].height);
//imageMode(CORNER);
}
//scale(0.5);
popMatrix();

int[] colors = colorDetection();
// println(colors);

// We must always step through time!
box2d.step();
// Display all the bananas
for (Banana b : bananas) {
b.display();
}

// Display all the cupcakes
for (Cupcake c : cupcakes) {
c.display();
}

// Display all the fishes
for (Fish f : fishes) {
f.display();
}

handX1 = lerp(handX1, colors[0], 0.2);
handY1 = lerp(handY1, colors[1], 0.2);
handX2 = lerp(handX2, colors[2], 0.2);
handY2 = lerp(handY2, colors[3], 0.2);
// remove old body
leftclaws.get(leftclaws.size()-1).killBody();
//remove from arraylist
leftclaws.remove(leftclaws.size()-1);

// remove old body
rightclaws.get(rightclaws.size()-1).killBody();
//remove from arraylist
rightclaws.remove(rightclaws.size()-1);
//println(handX1, handY1);
//println(handX2, handY2);

// Display the current leftclaw
int currentClaw1 = leftclaws.size()-1;
Leftclaw l = leftclaws.get(currentClaw1);
l.display();

// Display the current rightclaw
int currentClaw2 = rightclaws.size()-1;
Rightclaw r = rightclaws.get(currentClaw2);
r.display();
// bananas that leave the screen, we delete them
Banana b = bananas.get(bananas.size()-1);
if (b.done()) {
}

/*
if (bananas.size() > 0&& !bananas.get(bananas.size()-1).done()) {
println(bananas.get(bananas.size()-1).pos.x);
float bx = bananas.get(bananas.size()-1).pos.x;
float by = bananas.get(bananas.size()-1).pos.y;
println(bx + ” ” + by);
if ((bx <= colors[0]+50 && bx >= colors[0]-50 && by <= colors[1]+50 && by >= colors[1]-50)||(bx <= colors[2]+50 && bx >= colors[2]-50 && by <= colors[3]+50 && by >= colors[3]-50)) {
//if((bx-colors[0])<30 && (bx-colors[0])>-30)&& by <= colors[1]+30 && by >= colors[1]-30) {
state = false;
ellipse(bx, by, 200, 200);
}
}

*/

// cupcakes that leave the screen, we delete them
Cupcake c = cupcakes.get(cupcakes.size()-1);
if (c.done()) {
}

// fishes that leave the screen, we delete them
Fish f = fishes.get(fishes.size()-1);
if (f.done()) {
}
}

void captureEvent(Capture c) {
}

void mousePressed() {
if (mouseButton == LEFT) {
trackColor2 = get(mouseX, mouseY);
}
if (mouseButton == RIGHT) {
trackColor1 = get(mouseX, mouseY);
}
}
int[] colorDetection() {

int closestX1 = 0;
int closestY1 = 0;
int closestX2 = 0;
int closestY2 = 0;

for (int x = 0; x < video.width; x ++ ) {
for (int y = 0; y < video.height; y ++ ) {
int loc = x + y*video.width;
// What is current color
color currentColor1 = video.pixels[loc];
float r1 = red(currentColor1);
float g1 = green(currentColor1);
float b1 = blue(currentColor1);
float r2 = red(trackColor1);
float g2 = green(trackColor1);
float b2 = blue(trackColor1);
float d = abs(r1 – r2);
float s = abs(g1 – g2);
float t = abs(b1 – b2);
if (d<20 && s<20 && t<20) {
closestX1 = x;
closestY1 = y;
//point(x, y);
}
}
}

for (int x = 0; x < video.width; x ++ ) {
for (int y = 0; y < video.height; y ++ ) {
int loc = x + y*video.width;
// What is current color
color currentColor2 = video.pixels[loc];
float r3 = red(currentColor2);
float g3 = green(currentColor2);
float b3 = blue(currentColor2);
float r4 = red(trackColor2);
float g4 = green(trackColor2);
float b4 = blue(trackColor2);
float o = abs(r3 – r4);
float p = abs(g3 – g4);
float q = abs(b3 – b4);
if (o<10 && p<10 && q<10) {
closestX2 = x;
closestY2 = y;
}
}
}

int [] fourcolor = {closestX1*2, closestY1*2, closestX2*2, closestY2*2};
//println(fourcolor);
return (fourcolor);
}

class Banana {

// We need to keep track of a Body and a radius
Body body;
float r;
Vec2 pos;
Banana(float x, float y, float r_) {
r = r_;
// This function puts the particle in the Box2d world
makeBody(x,y,r);
}

// This function removes the particle from the box2d world
void killBody() {
box2d.destroyBody(body);
}

// Is the particle ready for deletion?
boolean done() {
// Let’s find the screen position of the particle
pos = box2d.getBodyPixelCoord(body);
// Is it off the bottom of the screen?
if (pos.y > height+r*2-300) {
//killBody();
return true;
}
return false;
}

//
void display() {
// We look at each body and get its screen position
Vec2 pos = box2d.getBodyPixelCoord(body);
// Get its angle of rotation
float a = body.getAngle();
pushMatrix();
translate(pos.x,pos.y);
rotate(-0.5*a);
imageMode(CENTER);
image(banana,0,0,r*2,r*2);
popMatrix();
}

// Here’s our function that adds the particle to the Box2D world
void makeBody(float x, float y, float r) {
// Define a body
BodyDef bd = new BodyDef();
// Set its position
bd.position = box2d.coordPixelsToWorld(x,y);
bd.type = BodyType.DYNAMIC;
body = box2d.world.createBody(bd);

// Make the body’s shape a circle
CircleShape cs = new CircleShape();

FixtureDef fd = new FixtureDef();
fd.shape = cs;
// Parameters that affect physics
fd.density = 4;
fd.friction = 0.01;
fd.restitution = 0.2;

// Attach fixture to body
body.createFixture(fd);

// Give it a random initial velocity (and angular velocity)
body.setLinearVelocity(new Vec2(random(-10f,10f),random(5f,10f)));
body.setAngularVelocity(random(-10,10));
}

}

class Surface {
// We’ll keep track of all of the surface points
ArrayList<Vec2> surface;
Surface() {
surface = new ArrayList<Vec2>();

// This is what box2d uses to put the surface in its world
ChainShape chain = new ChainShape();

// Perlin noise argument
float xoff = 0.0;

// This has to go backwards so that the objects bounce off the top of the surface
// This “edgechain” will only work in one direction!
for (float x = width+10; x > -10; x -= 5) {

// Doing some stuff with perlin noise to calculate a surface that points down on one side
// and up on the other
float y;
y=height;

// Store the vertex in screen coordinates

// Move through perlin noise
xoff += 0.1;

}

// Build an array of vertices in Box2D coordinates
// from the ArrayList we made
Vec2[] vertices = new Vec2[surface.size()];
for (int i = 0; i < vertices.length; i++) {
Vec2 edge = box2d.coordPixelsToWorld(surface.get(i));
vertices[i] = edge;
}

// Create the chain!
chain.createChain(vertices,vertices.length);

// The edge chain is now attached to a body via a fixture
BodyDef bd = new BodyDef();
bd.position.set(0.0f,0.0f);
Body body = box2d.createBody(bd);
// Shortcut, we could define a fixture if we
// want to specify frictions, restitution, etc.
body.createFixture(chain,1);

}

// A simple function to just draw the edge chain as a series of vertex points
void display() {
strokeWeight(2);
stroke(0);
noFill();
beginShape();
for (Vec2 v: surface) {
vertex(v.x,v.y);

}
endShape();
}
}

class Cupcake {

// We need to keep track of a Body and a radius
Body body;
float r;
Vec2 pos;
Cupcake(float x, float y, float r_) {
r = r_;
// This function puts the particle in the Box2d world
makeBody(x,y,r);
}

// This function removes the particle from the box2d world
void killBody() {
box2d.destroyBody(body);
}

// Is the particle ready for deletion?
boolean done() {
// Let’s find the screen position of the particle
pos= box2d.getBodyPixelCoord(body);
// Is it off the bottom of the screen?
if (pos.y > height+r*2) {
// killBody();
return true;
}
return false;
}

//
void display() {
// We look at each body and get its screen position
Vec2 pos = box2d.getBodyPixelCoord(body);
// Get its angle of rotation
float a = body.getAngle();
pushMatrix();
translate(pos.x,pos.y);
rotate(-a);
imageMode(CENTER);
image(cupcake,0,0,r*2,r*2);
popMatrix();
}

// Here’s our function that adds the particle to the Box2D world
void makeBody(float x, float y, float r) {
// Define a body
BodyDef bd = new BodyDef();
// Set its position
bd.position = box2d.coordPixelsToWorld(x,y);
bd.type = BodyType.DYNAMIC;
body = box2d.world.createBody(bd);

// Make the body’s shape a circle
CircleShape cs = new CircleShape();

FixtureDef fd = new FixtureDef();
fd.shape = cs;
// Parameters that affect physics
fd.density = 3;
fd.friction = 0.01;
fd.restitution = 0.3;

// Attach fixture to body
body.createFixture(fd);

// Give it a random initial velocity (and angular velocity)
body.setLinearVelocity(new Vec2(random(-10f,10f),random(5f,10f)));
body.setAngularVelocity(random(-10,10));
}

}

class Fish {

// We need to keep track of a Body and a radius
Body body;
float r;

Fish(float x, float y, float r_) {
r = r_;
// This function puts the particle in the Box2d world
makeBody(x,y,r);
}

// This function removes the particle from the box2d world
void killBody() {
box2d.destroyBody(body);
}

// Is the particle ready for deletion?
boolean done() {
// Let’s find the screen position of the particle
Vec2 pos = box2d.getBodyPixelCoord(body);
// Is it off the bottom of the screen?
if (pos.y > height+r*2) {
// killBody();
return true;
}
return false;
}

//
void display() {
// We look at each body and get its screen position
Vec2 pos = box2d.getBodyPixelCoord(body);
// Get its angle of rotation
float a = body.getAngle();
pushMatrix();
translate(pos.x,pos.y);
rotate(-a);
imageMode(CENTER);
image(fish,0,0,r*2,r*2);
popMatrix();
}

// Here’s our function that adds the particle to the Box2D world
void makeBody(float x, float y, float r) {
// Define a body
BodyDef bd = new BodyDef();
// Set its position
bd.position = box2d.coordPixelsToWorld(x,y);
bd.type = BodyType.DYNAMIC;
body = box2d.world.createBody(bd);

// Make the body’s shape a circle
CircleShape cs = new CircleShape();

FixtureDef fd = new FixtureDef();
fd.shape = cs;
// Parameters that affect physics
fd.density = 6;
fd.friction = 0.01;
fd.restitution = 0.5;

// Attach fixture to body
body.createFixture(fd);

// Give it a random initial velocity (and angular velocity)
body.setLinearVelocity(new Vec2(random(-10f,10f),random(5f,10f)));
body.setAngularVelocity(random(-10,10));
}

}

class Leftclaw {

// We need to keep track of a Body and a radius
Body body;
float r;

Leftclaw(float x, float y, float r_) {
r = r_;
// This function puts the particle in the Box2d world
makeBody(x,y,r);
}

// This function removes the particle from the box2d world
void killBody() {
box2d.destroyBody(body);
}

// Is the particle ready for deletion?
boolean done() {
// Let’s find the screen position of the particle
Vec2 pos = box2d.getBodyPixelCoord(body);
// Is it off the bottom of the screen?
if (pos.y > height+r*2) {
killBody();
return true;
}
return false;
}

//
void display() {
// We look at each body and get its screen position
Vec2 pos = box2d.getBodyPixelCoord(body);

pushMatrix();
translate(pos.x,pos.y);
fill(175);
stroke(0);
strokeWeight(1);
imageMode(CENTER);
image(leftclaw,0,0,r*2,r*2);
popMatrix();
}

// Here’s our function that adds the particle to the Box2D world
void makeBody(float x, float y, float r) {
// Define a body
BodyDef bd = new BodyDef();

// Set its position
println(x,y);
bd.position = box2d.coordPixelsToWorld(x,y);
bd.type = BodyType.STATIC;
body = box2d.world.createBody(bd);

// Make the body’s shape a circle
CircleShape cs = new CircleShape();

FixtureDef fd = new FixtureDef();
fd.shape = cs;
// Parameters that affect physics
fd.density = 1;
fd.friction = 0.01;
fd.restitution = 1;

// Attach fixture to body
body.createFixture(fd);

// Give it a random initial velocity (and angular velocity)
body.setLinearVelocity(new Vec2(random(-10f,10f),random(5f,10f)));
body.setAngularVelocity(random(-10,10));
}

}

class Rightclaw {

// We need to keep track of a Body and a radius
Body body;
float r;

Rightclaw(float x, float y, float r_) {
r = r_;
// This function puts the particle in the Box2d world
makeBody(x,y,r);
}

// This function removes the particle from the box2d world
void killBody() {
box2d.destroyBody(body);
}

// Is the particle ready for deletion?
boolean done() {
// Let’s find the screen position of the particle
Vec2 pos = box2d.getBodyPixelCoord(body);
// Is it off the bottom of the screen?
if (pos.y > height+r*2) {
killBody();
return true;
}
return false;
}

//
void display() {
// We look at each body and get its screen position
Vec2 pos = box2d.getBodyPixelCoord(body);

pushMatrix();
translate(pos.x,pos.y);
fill(175);
stroke(0);
strokeWeight(1);
imageMode(CENTER);
image(rightclaw,0,0,r*2,r*2);
popMatrix();
}

// Here’s our function that adds the particle to the Box2D world
void makeBody(float x, float y, float r) {
// Define a body
BodyDef bd = new BodyDef();

// Set its position
println(x,y);
bd.position = box2d.coordPixelsToWorld(x,y);
bd.type = BodyType.STATIC;
body = box2d.world.createBody(bd);

// Make the body’s shape a circle
CircleShape cs = new CircleShape();

FixtureDef fd = new FixtureDef();
fd.shape = cs;
// Parameters that affect physics
fd.density = 1;
fd.friction = 0.01;
fd.restitution = 1;

// Attach fixture to body
body.createFixture(fd);

// Give it a random initial velocity (and angular velocity)
body.setLinearVelocity(new Vec2(random(-10f,10f),random(5f,10f)));
body.setAngularVelocity(random(-10,10));
}

}