# Kinetic Interface – Midterm Project (Emerald)

Course Name: Kinetic interface

Instructor: Moon

Partner: Ribirth

• Description

Basically our project is an educational hotpot game. It shows players different time that different food need to be cooked in a hotpot. We use leap motion to enable the grab gesture of the player, so it gives a more physical sense of use chopsticks to get the food.

We have two stages, the first has a start button, and player should grab to start the game (also a “grab to start” instruction shown above the button). The second stage is the main gaming part, which contains all those food, hotpot, timer, and so on. We use “switch-case” to function the change of the stage. When you run the project, the default stage is stage 0 (the button part), and when it detects your grab gesture, it will switch to stage 1, which is the game part.

As you can see in the demo video at the bottom of this documentation, there is a timer on the right-top corner. If you grab different food to put into the pot, you will see the timer runs in different speed (because the time bar goes in same speed), which thus can show how long you have put certain food in the hotpot. The timer, as well as the time bar, will not start counting time until you put a food into the pot. Also, once it starts cooking, there will be words showing conditions of food under the time bar. For instance, at first it shows “NOT YET!”, some time later it shows “Ready!”, and if you still don’t take out the food, it will show “OVERCOOKED!”.

• Process & Difficulties

The Process of this game is: grab the food – put the food into the pot – wait to take out the food from the pot – eat it/throw it away. I did meet many problems when coding this whole process. Fortunately we finally found the way to solve these problems.

In terms of grabing the food, the biggest problem at first is the food you grab can only be displayed within the certain range. I find the reason is because I have wrote the code “if (dist1 < 50 && HG > 0.6)” (dist1 stands for the distance between chopsticks and the plate of meat; HG stands for hand grab strength). To display the food you grab, we choose to stick every single image of different food to the chopsticks’ position, and tint the opacity to 0, so that the food images are transparent at first, and when you grab, the opacity will be tint again to 300. In this case, I find that there should be a bigger if statement outside “if (dist1 < 50 && HG > 0.6)”, which is “if (opacity1 == 0)”. So once you grab the food, the opacity changes to 300, and the “dist1 < 50” will not be the constrain any more.

We also think a lot about how to start cooking another food when you finish the previous one. For example, the pot range is 100 pixel from the center point, and you move the food to 100 – 200 pixel means you take out the food, more than 200 means you throw the food away. But this may cause a mess when playing the game. We also think of swipe to throw away, but finally we decide to stick to the grab gesture. Not only because we already have the HG variable, which can be much easier to code, but also loosening hand to let go the food makes more sense. So if HG is less than 0.6, the opacity I mentioned above will be tint again to 0, as well as the timer and time bar.

• Other

We also add some effects to make the game more interesting.

1. We add background music and “game start” sound effect.
2. If you take out the food prematurely, you will see a sad face appear on the screen, and the color of food remains unchanged. If you take out the food on time, the color will slightly change and a smiling face will appear. If you take out it overcooked, sad face appear again and the color will turn to black.
• Future Improvments

As the feedbacks during presentation, it will be better if we adjust the perspective of the pot. We can also add more sounf effects when putting the food into the pot. Maybe also add bubbles on the pot to create a sense of boiling.

• Demo

``````import processing.sound.*;
SoundFile file;
SoundFile no;
SoundFile yes;

//import
//  ddf.minim.*;
//Minim minim;
//AudioPlayer player;

int stage = 0;
int time = 0;
int foodtime = 0;

float opacity1 = 0;
float opacity2 = 0;
float opacity3 = 0;
float opacity4 = 0;
float opacity5 = 0;
float opacity6 = 0;

float handX, handY;
float HG;

PFont myFont;
PImage background;
PImage clock;
PImage star;
PImage timebar;
PImage start;
PImage table;

PImage meat;
PImage cabbage;
PImage shiitake;
PImage crab;
PImage goat;
PImage meatball;

PImage meat1;
PImage cabbage1;
PImage shiitake1;
PImage crab1;
PImage goat1;
PImage meatball1;

PImage pot;
PImage chopstick1;
PImage chopstick2;

PImage success;
PImage fail;

void setup() {

size(600, 600);
background(255);

leapMotion_setup();
myFont = createFont("Skia-Regular_Bold", 32);
textFont(myFont);

file = new SoundFile(this, "bgm.mp3");
file.loop();
//minim = new
//  Minim(this);
no = new SoundFile(this, "no.mp3");
yes = new SoundFile(this, "yes.mp3");

}

void draw() {

leapMotion_draw();

switch( stage ) {
case 0:
stage_opening();
break;
case 1:
stage_play();

break;
case 2:
break;
}
}

void stage_opening() {

pushMatrix();
background(background);
imageMode(CENTER);
translate(width/2, height/2);
scale(0.5);
image(start, 0, 0);
popMatrix();

stroke(10);
fill(252, 98, 80);
text("Crab to Strat!",200,200);

if (HG>0.6) {
pushMatrix();
background(background);
//background(255);
imageMode(CENTER);
translate(width/2, height/2);
scale(0.4);
image(start, 0, 0);
//player.play();
popMatrix();
time ++;
println(time);
if (time>10) {
stage = 1;
}
}
}

//if (mousePressed) {
//  stage = 1;
//}
//background(255, 255, 0);
//fill(255);
//text("StartPage", 10, 20);

void stage_play() {
background(255);
image(table, 0, 0, width*2, height*2);
image(meat, 70, 200, 100, 100);
image(cabbage, 70, 350, 100, 100);
image(shiitake, 540, 200, 100, 100);
image(crab, 540, 350, 100, 100);
image(goat, 210, 470, 100, 100);
image(meatball, 400, 470, 100, 100);
image(timebar, 250, 170, 700, 500);

pushStyle();
imageMode(CENTER);
image(pot, width/2, height/2, 369, 232);
popStyle();

pushMatrix();
pushStyle();
translate(handX, handY);
rotate(PI/1.2);
imageMode(CENTER);
image(chopstick1, 0, 15, 150, 15);
popStyle();
popMatrix();

pushMatrix();
pushStyle();
translate(handX, handY);
rotate(PI/1.1);
imageMode(CENTER);
image(chopstick2, 0, -15, 150, 15);
popStyle();
popMatrix();

pushStyle();
tint(255, 255, 255, opacity1);
image(meat1, handX-60, handY+20, 40, 50);
popStyle();
pushStyle();
tint(255, 255, 255, opacity2);
image(cabbage1, handX-60, handY+20, 40, 80);
popStyle();
pushStyle();
tint(255, 255, 255, opacity3);
image(shiitake1, handX-60, handY+20, 30, 30);
popStyle();
pushStyle();
tint(255, 255, 255, opacity4);
image(crab1, handX-60, handY+20, 40, 50);
popStyle();
pushStyle();
tint(255, 255, 255, opacity5);
image(goat1, handX-60, handY+20, 50, 50);
popStyle();
pushStyle();
tint(255, 255, 255, opacity6);
image(meatball1, handX-60, handY+20, 30, 30);
popStyle();

float dist1 = dist(handX, handY, 120, 200 );
float dist2 = dist(handX, handY, 120, 350 );
float dist3 = dist(handX, handY, 590, 220 );
float dist4 = dist(handX, handY, 590, 370 );
float dist5 = dist(handX, handY, 290, 490 );
float dist6 = dist(handX, handY, 480, 480 );
float dist_drop = dist(handX-30, handY+30, 300, 300);

image(star, 50+foodtime*3, 115, 400, 400);
image(clock, 520, 60, 140, 120);

// ====  meat ==== //

if (opacity1 == 0) {
if (dist1 < 50 && HG > 0.6) {
opacity1 = 300;
}
}
if (opacity1 == 300 && dist_drop < 100) {
fill(0, 0, 0);
text(foodtime/6, 510, 70);
foodtime ++;
println(foodtime);
stroke(10);
fill(255);
if (foodtime>90) {
fill (232, 42, 6);
text ("OVER COOKED!", 220, 100);
} else if (foodtime>60 & foodtime<90) {
fill(232, 118, 80);
} else if (foodtime<60) {
text ("NOT YET!", 80, 100);
}
} else if (opacity1 == 300 && foodtime < 90 && foodtime > 60 && dist_drop > 100 ) {
pushStyle();
tint(255, 255, 255, opacity1);
image(success, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity1-280);
image(meat1, handX-60, handY+20, 40, 50);
popStyle();
} else if (opacity1 == 300 && foodtime > 90 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity1);
image(fail, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity1);
image(meat1, handX-60, handY+20, 40, 50);
popStyle();
} else if (opacity1 == 300 && foodtime > 1 && foodtime < 60 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity1);
image(fail, 300, 300, 200, 200);
popStyle();
}
if (opacity1 == 300 && HG < 0.5) {
foodtime = 0;
opacity1 = 0;
}

// ====  cabbage ==== //

if (opacity2 == 0) {
if (dist2 < 50 && HG > 0.6) {
opacity2 = 300;
}
}
if (opacity2 == 300 && dist_drop < 100) {
fill(0, 0, 0);
text(foodtime/2, 510, 70);
foodtime ++;
println(foodtime);
stroke(10);
fill(255);
if (foodtime>90) {
fill (232, 42, 6);
text ("OVER COOKED!", 220, 100);
} else if (foodtime>60 & foodtime<90) {
fill(232, 118, 80);
} else if (foodtime<60) {
text ("NOT YET!", 80, 100);
}
} else if (opacity2 == 300 && foodtime < 90 && foodtime > 60 && dist_drop > 100 ) {
pushStyle();
tint(255, 255, 255, opacity2);
image(success, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity2-280);
image(cabbage1, handX-60, handY+20, 40, 80);
popStyle();
} else if (opacity2 == 300 && foodtime > 90 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity2);
image(fail, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity2);
image(cabbage1, handX-60, handY+20, 40, 80);
popStyle();
} else if (opacity2 == 300 && foodtime > 1 && foodtime < 60 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity2);
image(fail, 300, 300, 200, 200);
popStyle();
}
if (opacity2 == 300 && HG < 0.5) {
foodtime = 0;
opacity2 = 0;
}

// ====  shiitake ==== //
if (opacity3 == 0) {
if (dist3 < 50 && HG > 0.6) {
opacity3 = 300;
}
}
if (opacity3 == 300 && dist_drop < 100) {
fill(0, 0, 0);
text(foodtime/2, 510, 70);
foodtime ++;
println(foodtime);
stroke(10);
fill(255);
if (foodtime>90) {
fill (232, 42, 6);
text ("OVER COOKED!", 220, 100);
} else if (foodtime>60 & foodtime<90) {
fill(232, 118, 80);
} else if (foodtime<60) {
text ("NOT YET!", 80, 100);
}
} else if (opacity3 == 300 && foodtime < 90 && foodtime > 60 && dist_drop > 100 ) {
pushStyle();
tint(255, 255, 255, opacity3);
image(success, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity3-280);
image(shiitake1, handX-60, handY+20, 30, 30);
popStyle();
} else if (opacity3 == 300 && foodtime > 90 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity3);
image(fail, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity3);
image(shiitake1, handX-60, handY+20, 30, 30);
popStyle();
} else if (opacity3 == 300 && foodtime > 1 && foodtime < 60 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity3);
image(fail, 300, 300, 200, 200);
popStyle();
}
if (opacity3 == 300 && HG < 0.5) {
foodtime = 0;
opacity3 = 0;
}

// ====  crab ==== //
if (opacity4 == 0) {
if (dist4 < 50 && HG > 0.6) {
opacity4 = 300;
}
}
if (opacity4 == 300 && dist_drop < 100) {
fill(0, 0, 0);
text(foodtime/3, 510, 70);
foodtime ++;
println(foodtime);
stroke(10);
fill(255);
if (foodtime>90) {
fill (232, 42, 6);
text ("OVER COOKED!", 220, 100);
} else if (foodtime>60 & foodtime<90) {
fill(232, 118, 80);
} else if (foodtime<60) {
text ("NOT YET!", 80, 100);
}
} else if (opacity4 == 300 && foodtime < 90 && foodtime > 60 && dist_drop > 100 ) {
pushStyle();
tint(255, 255, 255, opacity4);
image(success, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity4-280);
image(crab1, handX-60, handY+20, 40, 50);
popStyle();
} else if (opacity4 == 300 && foodtime > 90 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity4);
image(fail, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity4);
image(crab1, handX-60, handY+20, 40, 50);
popStyle();
} else if (opacity4 == 300 && foodtime > 1 && foodtime < 60 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity4);
image(fail, 300, 300, 200, 200);
popStyle();
}
if (opacity4 == 300 && HG < 0.5) {
foodtime = 0;
opacity4 = 0;
}

// ====  goat ==== //
if (opacity5 == 0) {
if (dist5 < 50 && HG > 0.6) {
opacity5 = 300;
}
}
if (opacity5 == 300 && dist_drop < 100) {
fill(0, 0, 0);
text(foodtime/7, 510, 70);
foodtime ++;
println(foodtime);
stroke(10);
fill(255);
if (foodtime>90) {
fill (232, 42, 6);
text ("OVER COOKED!", 220, 100);
} else if (foodtime>60 & foodtime<90) {
fill(232, 118, 80);
} else if (foodtime<60) {
text ("NOT YET!", 80, 100);
}
} else if (opacity5 == 300 && foodtime < 90 && foodtime > 60 && dist_drop > 100 ) {
pushStyle();
tint(255, 255, 255, opacity5);
image(success, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity5-280);
image(goat1, handX-60, handY+20, 50, 50);
popStyle();
} else if (opacity5 == 300 && foodtime > 90 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity5);
image(fail, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity5);
image(goat1, handX-60, handY+20, 50, 50);
popStyle();
} else if (opacity5 == 300 && foodtime > 1 && foodtime < 60 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity5);
image(fail, 300, 300, 200, 200);
popStyle();
}
if (opacity5 == 300 && HG < 0.5) {
foodtime = 0;
opacity5 = 0;
}

// ==== meatball ==== //
if (opacity6 == 0) {
if (dist6 < 50 && HG > 0.6) {
opacity6 = 300;
}
}
if (opacity6 == 300 && dist_drop < 100) {
fill(0, 0, 0);
text(foodtime/4, 510, 70);
foodtime ++;
println(foodtime);
stroke(10);
fill(255);
if (foodtime>90) {
fill (232, 42, 6);
text ("OVER COOKED!", 220, 100);
} else if (foodtime>60 & foodtime<90) {
fill(232, 118, 80);
} else if (foodtime<60) {
text ("NOT YET!", 80, 100);
}
} else if (opacity6 == 300 && foodtime < 90 && foodtime > 60 && dist_drop > 100 ) {
pushStyle();
tint(255, 255, 255, opacity6);
image(success, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity6-280);
image(meatball1, handX-60, handY+20, 30, 30);
popStyle();
} else if (opacity6 == 300 && foodtime > 90 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity6);
image(fail, 300, 300, 200, 200);
popStyle();
pushStyle();
tint(0, 0, 0, opacity6);
image(meatball1, handX-60, handY+20, 30, 30);
popStyle();
} else if (opacity6 == 300 && foodtime > 1 && foodtime < 60 && dist_drop > 100) {
pushStyle();
tint(255, 255, 255, opacity6);
image(fail, 300, 300, 200, 200);
popStyle();
}
if (opacity6 == 300 && HG < 0.5) {
foodtime = 0;
opacity6 = 0;
}
}

import de.voidplus.leapmotion.*;

LeapMotion leap;

void leapMotion_setup() {

leap = new LeapMotion(this);

}

// ======================================================
// 1. Callbacks

void leapOnInit() {
// println("Leap Motion Init");
}
void leapOnConnect() {
// println("Leap Motion Connect");
}
void leapOnFrame() {
// println("Leap Motion Frame");
}
void leapOnDisconnect() {
// println("Leap Motion Disconnect");
}
void leapOnExit() {
// println("Leap Motion Exit");
}

void leapMotion_draw() {

int fps = leap.getFrameRate();
for (Hand hand : leap.getHands ()) {

// ==================================================
// 2. Hand

int     handId             = hand.getId();
PVector handPosition       = hand.getPosition();
PVector handStabilized     = hand.getStabilizedPosition();
PVector handDirection      = hand.getDirection();
PVector handDynamics       = hand.getDynamics();
float   handRoll           = hand.getRoll();
float   handPitch          = hand.getPitch();
float   handYaw            = hand.getYaw();
boolean handIsLeft         = hand.isLeft();
boolean handIsRight        = hand.isRight();
float   handGrab           = hand.getGrabStrength();
float   handPinch          = hand.getPinchStrength();
float   handTime           = hand.getTimeVisible();
PVector spherePosition     = hand.getSpherePosition();

// --------------------------------------------------
// Drawing
//hand.draw();

handX = hand.getPosition().x;
handY = hand.getPosition().y;

HG = handGrab;

// ==================================================
// 3. Arm

if (hand.hasArm()) {
Arm     arm              = hand.getArm();
float   armWidth         = arm.getWidth();
PVector armWristPos      = arm.getWristPosition();
PVector armElbowPos      = arm.getElbowPosition();
}

// ==================================================
// 4. Finger

Finger  fingerThumb        = hand.getThumb();
// or                        hand.getFinger("thumb");
// or                        hand.getFinger(0);

Finger  fingerIndex        = hand.getIndexFinger();
// or                        hand.getFinger("index");
// or                        hand.getFinger(1);

Finger  fingerMiddle       = hand.getMiddleFinger();
// or                        hand.getFinger("middle");
// or                        hand.getFinger(2);

Finger  fingerRing         = hand.getRingFinger();
// or                        hand.getFinger("ring");
// or                        hand.getFinger(3);

Finger  fingerPink         = hand.getPinkyFinger();
// or                        hand.getFinger("pinky");
// or                        hand.getFinger(4);

for (Finger finger : hand.getFingers()) {
// or              hand.getOutstretchedFingers();
// or              hand.getOutstretchedFingersByAngle();

int     fingerId         = finger.getId();
PVector fingerPosition   = finger.getPosition();
PVector fingerStabilized = finger.getStabilizedPosition();
PVector fingerVelocity   = finger.getVelocity();
PVector fingerDirection  = finger.getDirection();
float   fingerTime       = finger.getTimeVisible();

// ------------------------------------------------
// Drawing

// Drawing:
// finger.draw();  // Executes drawBones() and drawJoints()
//finger.drawBones();
//finger.drawJoints();

// ------------------------------------------------
// Selection

// ================================================
// 5. Bones
// --------
// https://developer.leapmotion.com/documentation/java/devguide/Leap_Overview.html#Layer_1

Bone    boneDistal       = finger.getDistalBone();
// or                      finger.get("distal");
// or                      finger.getBone(0);

Bone    boneIntermediate = finger.getIntermediateBone();
// or                      finger.get("intermediate");
// or                      finger.getBone(1);

Bone    boneProximal     = finger.getProximalBone();
// or                      finger.get("proximal");
// or                      finger.getBone(2);

Bone    boneMetacarpal   = finger.getMetacarpalBone();
// or                      finger.get("metacarpal");
// or                      finger.getBone(3);

// ------------------------------------------------
// Touch emulation

int     touchZone        = finger.getTouchZone();
float   touchDistance    = finger.getTouchDistance();

switch(touchZone) {
case -1: // None
break;
case 0: // Hovering
// println("Hovering (#" + fingerId + "): " + touchDistance);
break;
case 1: // Touching
// println("Touching (#" + fingerId + ")");
break;
}
}

// ==================================================
// 6. Tools

for (Tool tool : hand.getTools()) {
int     toolId           = tool.getId();
PVector toolPosition     = tool.getPosition();
PVector toolStabilized   = tool.getStabilizedPosition();
PVector toolVelocity     = tool.getVelocity();
PVector toolDirection    = tool.getDirection();
float   toolTime         = tool.getTimeVisible();

// ------------------------------------------------
// Drawing:
tool.draw();

// ------------------------------------------------
// Touch emulation

int     touchZone        = tool.getTouchZone();
float   touchDistance    = tool.getTouchDistance();

switch(touchZone) {
case -1: // None
break;
case 0: // Hovering
// println("Hovering (#" + toolId + "): " + touchDistance);
break;
case 1: // Touching
// println("Touching (#" + toolId + ")");
break;
}
}
}

// ====================================================
// 7. Devices

for (Device device : leap.getDevices()) {
float deviceHorizontalViewAngle = device.getHorizontalViewAngle();
float deviceVericalViewAngle = device.getVerticalViewAngle();
float deviceRange = device.getRange();
}
}
``````