Capstone Final Documentation | Kate Thoma-Hilliard

Recalibrating Accents : Synecdoche

After your physical appearance the next thing someone usually notices about you is your accent. A person’s speech especially their accent or seemingly lack of one can affect how they interact with you. If you live anywhere with a public-school system, chances are you are close to someone who speaks differently than you. But who has an accent. Everyone does, that includes you too.

Synecdoche is a website that allows people to see their accent as well as explore how other accents filter the way people pronounce English words. The user will be able to see visualizations of their speech and be provided with tools to analyze the visualizations. Utilizing the International Phonetic Alphabet users will be able to experience accents in different ways. The IPA was selected because each symbol has a unique sound which allows it to convey slight pronunciation differences. They will also gain a basic understanding of the IPA as a language tool.

Another integral part of the project is to expose users to other accents because it has been found that this improves people’s abilities to unpack others’ accents. Users will be able to input words and see approximations of how other speakers might pronounce them.

Understanding accents is even more important as English has become the default language when two speakers’ native languages are mutually unintelligible.

The aim of Synecdoche is not to advocate for people to change their accents but gain more knowledge about accents and understand that accents do not make someone fundamentally different. Synecdoche aims to be a conversation starter to address how accents affect people and why they shouldn’t.

Final | Kinetic Interfaces| Kate

I called this project “Lightanenome” just because I liked the sound of the name. It is a two player game with lights scattered around the screen. Players must physically move around the space to move their characters and turn all the lights on or off depending on their character.

Here is the early concept art.

The idea is that people have fun running around in a space playing a more classical style game. There is a little bit of a built in obstacle with the layout of the different levels, but the idea is that it is mainly a competition against your opponent. And unlike games like wii, you are moving your player in the space with an opponent in the same space not as a controller but more like the live chess game in the first Harry Potter.

I kept it animated to make it easier for me to keep user test while building it, but would have to handle scale and size of environment if I were to make it webcam display. It is also going from creating a 2D environment to playing it in 3D so I would want to make better use of this.

The game ended up having 4 levels with the layout and number of lights being the variable. I tried to make it so players had to stand at a certain place each time before it began to add some level of fairness. Another idea that I liked that someone suggested was to make the number of lights that were on and off proportional to who won the last level.

The biggest problem when I switched to the Kinect was that the skeletons indices changed when people became undetected by the Kinect. And the characters would float up for no reason, this might have been because of how I triggered the jumping because some people would end up floating away. I should have created an upper bound. But I thought it would force the user to jump more.  But I think I would need to add color tracking or something to make it have better calibration. I also did not put the text in very stylistically.

For now I added a rock-paper-scissor element to create an interaction when they bumped into each other and give one a possible advantage. I got feedback that it might seem a little weird but I think I didn’t develop the design enough, so it could still work.

The biggest change I would make would be to improve the look and the calibration. But I would stick with the idea and mechanics. I would also add sound.

 

 

Final Project Final Concept | Kate

This is the beginning of what my game looks like.

The yellow character will be turning on the lights while the blue one turns them off. The lights will all start on, but will be more spread out than they currently are.  Each character will spawn on either side. I will mark where players should stand to start, or make them return their characters to a certain place to begin. Then the game will be timed and the characters must try to turn all the lights on or off depending on who they are. The yellow player wins if all the lights are off at any point in the given time. The other player wins if there is at least one light on by the end of the game.

The platforms are there because there is gravity and so while you can continuously jump and reach them with the Kinect controlling the movement this is probably not advisable. In order to make it fair I will play around with how the lights get turned on/off whether it be a delay or having to use a hand signal. The characters will also be able to affect each other by hitting or blocking the other one.

Week 12 | Kinetic Interfaces | Kate

I was  originally going to create some simple drawing thing where multiple people could send info in and draw, but was not quite sure how to send different types of info. I guess another solution for how I built mine would have been to restrict the coordinates each person could draw in. I ended up making an extremely simple drawing thing, the idea was that people don’t know the full picture when they are playing around they just see their square moving. Another fun thing  may have been to add a third dimension. I also made it rotate without the sender knowing to add an element of randomness and to again show that they don’t know what is happening. It may be more fun if the sender was playing snake so they would be forced to move. The code is simply developed from what we used in class and is not that complicated.

// IMA NYU Shanghai
// Kinetic Interfaces
// MOQN
// Apr 19 2018

/**
 * based on oscP5sendreceive by andreas schlegel
 * oscP5 website at http://www.sojamo.de/oscP5
 */
 

import oscP5.*;
import netP5.*;


OscP5 oscP5;
NetAddress myRemoteLocation;
float x,y,r;
void setup() {
  size(400,400);
  
  oscP5 = new OscP5(this,12000);
  myRemoteLocation = new NetAddress("127.0.0.1",12000);
  rectMode(CENTER);
  x = width/2;
  y = height/2;
}


void draw() {
  background(0);
  rect(x,y,10,10);
  r = mouseX;
}

/*
void mousePressed() {
  OscMessage msg = new OscMessage("/test");
   
  // let's send mouseX&Y positions
  float x = map(mouseX, 0, width, 0, 1);
  float y = map(mouseY, 0, height, 0, 1);
  msg.add( x );
  msg.add( y );
  oscP5.send( msg, myRemoteLocation );
  
  ellipse(mouseX, mouseY, 10, 10);
}
*/
void keyPressed(){
  
  OscMessage msg = new OscMessage("/test");
  if(keyCode == LEFT){
    x -= 1;
    if(x<0){
      x= width;
    }
    msg.add(x);
    msg.add(y);
  }
  if(keyCode == RIGHT){
    x += 1;
    if(y>width){
      x = 0;
    }
    msg.add(x);
    msg.add(y);
  }
  if(keyCode == UP){
    y -= 1;
    if(y<0){
      y = height;
    }
    msg.add(x);
    msg.add(y);
  }
  if(keyCode == DOWN){
    y += 1;
    if(y>height){
      y = 0;
    }
    msg.add(x);
    msg.add(y);
  }
  msg.add(r);
  oscP5.send( msg, myRemoteLocation );
}
// IMA NYU Shanghai
// Kinetic Interfaces
// MOQN
// Apr 19 2018

/**
 * based on oscP5sendreceive by andreas schlegel
 * oscP5 website at http://www.sojamo.de/oscP5
 */
 

import oscP5.*;
import netP5.*;
PImage reflect;

OscP5 oscP5;

float x, y,x2,r;
float xValue = 0;
float yValue = 0;
float rValue = 0;
float d1,d2,d3,d4;

void setup() {
  size(800,800);
  
  oscP5 = new OscP5(this,12000);
  rectMode(CENTER);
  
  fill(255,0,0);
  rect(width/4,height/4,width/2,height/2);
  fill(0,255,255);
  rect(width/4,3*height/4,width/2,height/2);
  fill(255,255,0);
  rect(3*width/4,height/4,width/2,height/2);
  fill(0,0,255);
  rect(3*width/4,3*height/4,width/2,height/2);
}


void draw() {
  //background(0);
  r = map(rValue,0,width,0,TWO_PI);
  
  x = xValue;
  x2 = map(xValue,0,400,800,400);
  y = map(yValue,0,400,800,400);
  fill(255);
  pushMatrix();
  rotate(r);
  rect(xValue,yValue,10,10);
  rect(x,y,10,10);
  rect(x2,yValue,10,10);
  rect(x2,y,10,10);
  if(dist(xValue,yValue,width/2,height/2)<50){
  beginShape();
  d1 = dist(xValue,yValue,x2,yValue);
  for(int i = 0;i<d1-1;i++){
    float x1 = lerp(xValue,x2,i/d1);
    float y1 = lerp(yValue,yValue,i/d1);
    float n = noise(x*frameCount,y*frameCount);
    //y1 = map(y1*n,0,400,y1-50,y1+50);
    vertex(x1,y1);
  }
  endShape();
  
  beginShape();
  d2 = dist(x2,yValue,x2,y);
  for(int i = 0;i<d2-1;i+=3){
    float x3 = lerp(x2,x2,i/d2);
    float y3 = lerp(yValue,y,i/d2);
    vertex(x3,y3);
  }
  endShape();
  
  beginShape();
  d3 = dist(x2,y,x,y);
  for(int i = 0;i<d3-1;i+=3){
    float x4 = lerp(x2,x,i/d3);
    float y4 = lerp(y,y,i/d3);
    vertex(x4,y4);
  }
  endShape();
  
  beginShape();
  d4 = dist(x,y,xValue,yValue);
  for(int i = 0;i<d4-1;i+=3){
    float x5 = lerp(x,xValue,i/d4);
    float y5 = lerp(y,yValue,i/d4);
    vertex(x5,y5);
  }
  endShape();
  }
  popMatrix();
  /*
  noFill();
  if(dist(xValue,yValue,width/2,height/2)<50){
    beginShape();
    vertex(xValue,yValue);
    vertex(x2,yValue);
    vertex(x2,y);
    vertex(x,y);
    vertex(xValue,yValue);
    endShape();
  }
  */
}


void oscEvent(OscMessage msg) {
  println("___");
  println("Pattern: " + msg.addrPattern() );
  println("Typetag: " + msg.typetag() );
  
  xValue = msg.get(0).floatValue();
  yValue = msg.get(1).floatValue();
  rValue = msg.get(2).floatValue();
  println("x: " + rValue );
  //println("y: " + yValue );
  println();
}

Week 11 Kinect | Kinetic Interfaces | Kate

For this project I had originally wanted to put the depth image and skeleton together, but didn’t want to figure out how to map them onto each other since the sizing is a little off.  So I decided to sort of reuse an earlier homework and make it for Kinect. I decided to also give it a sort of electric theme for fun. It was lagging when I tried to add the buzzing orb, but I think that is because it is computing too many things at once. I started by just using the hand joints and then working with noise and lerp to make it more interesting looking. I played around with the class code until I figured out which parts I wanted to keep. Next I tried to add more to make it more interesting than the first version to make it so that if you cut through 10 balls you could have a super charged orb for a little. I was originally going to make it more timed, but found this easier to control. The video is a little off because I fixed the bug at the beginning that made the giant orb appear until something was detected.

import processing.sound.*;

// IMA NYU Shanghai
// Kinetic Interfaces
// MOQN
// Apr 8 2018

/** 
 * Based on Kinect for Windows v2 library for processing example code
 * by Thomas Sanchez Lengeling (http://codigogenerativo.com/)
 
 Sound File from Sharesynth found on Freesound.org
 **/

SoundFile buzz;
PVector handL, handR;
float d,x,y,pX,pY;
ArrayList<Balls> ball = new ArrayList<Balls>();
int count = 0;
int H = 20;
boolean first= true;
void setup(){
  size(1920, 1080, P3D);
  ellipseMode(CENTER);
  rectMode(CENTER);
  handL = new PVector();
  handR = new PVector();
  for(int i = 0;i<5;i++){
    ball.add(new Balls(random(width),random(height)));
  }
  buzz = new SoundFile(this,"sharesynth.wav");
  setupKinect();
}


void draw() {
  background(0);
 
  updateKinect();
    for(int j = 0;j<ball.size();j++){
    ball.get(j).display();
    ball.get(j).move();
    ball.get(j).check();
  } 
  d = dist(handL.x,handL.y,handR.x,handR.y); 
  noFill();
  beginShape();
  for(int i = 0;i<d-1;i+=8){
      y = lerp(handL.y,handR.y,i/d);
      x = lerp(handL.x,handR.x,i/d);
      pY = lerp(handL.y,handR.y,i+1/d);
      pX = lerp(handL.x,handR.x,i+1/d);
      stroke(255,255,255);
      float n = noise(x*frameCount,y*frameCount);
      y = map(y*n,0,height/2,y-H,y+H);
      vertex(x,y);
      
      for(int j = 0;j<ball.size();j++){
        if(dist(ball.get(j).x,ball.get(j).y,x,y)<30){
          first = false;
          count ++;
          ball.remove(j);
          ball.add(new Balls(random(width),random(height)));
        }
      }
  }
  endShape();
   beginShape();
  for(int i = 0;i<d-1;i+=5){
      y = lerp(handL.y,handR.y,i/d);
      x = lerp(handL.x,handR.x,i/d);
      pY = lerp(handL.y,handR.y,i+1/d);
      pX = lerp(handL.x,handR.x,i+1/d);
      //bolt.vertex(x,y);
      stroke(255,255,0);
      float n = noise(x*frameCount-5,y*frameCount-5);
      y = map(y*n,0,height/2,y-H,y+H);
      //point(x,y);
      vertex(x,y);
      
      for(int j = 0;j<ball.size();j++){
        if(dist(ball.get(j).x,ball.get(j).y,x,y)<30){
          first = false;
          count ++;
          ball.remove(j);
          ball.add(new Balls(random(width),random(height)));
        }
      }
  }
  endShape();
  beginShape();
  for(int i = 0;i<d-1;i+=3){
      y = lerp(handL.y,handR.y,i/d);
      x = lerp(handL.x,handR.x,i/d);
      pY = lerp(handL.y,handR.y,i+1/d);
      pX = lerp(handL.x,handR.x,i+1/d);
      //bolt.vertex(x,y);
      stroke(0,0,255);
      float n = noise(x*frameCount+5,y*frameCount+5);
      y = map(y*n,0,height/2,y-H,y+H);
      //point(x,y);
      vertex(x,y);
      
      for(int j = 0;j<ball.size();j++){
        if(dist(ball.get(j).x,ball.get(j).y,x,y)<30){
          first = false;
          count ++;
          ball.remove(j);
          ball.add(new Balls(random(width),random(height)));
        }
      }
  }
  endShape();
  noStroke();
  fill(105);
  rect(handL.x, handL.y, 10, 50);
  rect(handR.x, handR.y, 10, 50);
  textSize(40);
  if(count%10 <2 && dist(handL.x,handL.y,handR.x,handR.y)<400 && first == false){
    buzz.play();
    text("POWERED UP",50,50);
    stroke(255);
    fill(0);
     beginShape();

    float avgX;
    float avgY;
    avgX = (handL.x + handR.x)/2;
    avgY = (handL.y + handR.y)/2;
    for(int i = 0;i<360;i+= 8){
     float ang = map(i,0,360,0,2*PI);
     float c;
     float s;
     c = 350*cos(ang);
     s = 350*sin(ang);
     float cX = c + avgX;
     float cY = s+avgY;
     float n = noise(cX*frameCount+12,cY*frameCount+12);
     cX = map(cX * n,0,width/2,cX-H,cX+H);
     cY = map(cY * n,0,height/2,cY-H,cY+H);
     vertex(cX,cY);
     endShape();
    }
    stroke(0,0,255);
    beginShape();

    float avgX1;
    float avgY1;
    avgX1 = (handL.x + handR.x)/2;
    avgY1 = (handL.y + handR.y)/2;
    for(int i = 0;i<360;i+= 2){
     float ang = map(i,0,360,0,2*PI);
     float c;
     float s;
     c = 350*cos(ang);
     s = 350*sin(ang);
     float cX = c + avgX1;
     float cY = s+avgY1;
     float n = noise(cX*frameCount,cY*frameCount);
     cX = map(cX * n,0,width/2,cX-H,cX+H);
     cY = map(cY * n,0,height/2,cY-H,cY+H);
     vertex(cX,cY);
     endShape();
    }
    stroke(255,255,0);
    beginShape();

    float avgX2;
    float avgY2;
    avgX2 = (handL.x + handR.x)/2;
    avgY2 = (handL.y + handR.y)/2;
    for(int i = 0;i<360;i+= 6){
     float ang = map(i,0,360,0,2*PI);
     float c;
     float s;
     c = 350*cos(ang);
     s = 350*sin(ang);
     float cX = c + avgX2;
     float cY = s+avgY2;
     float n = noise(cX*frameCount+10,cY*frameCount+10);
     cX = map(cX * n,0,width/2,cX-H,cX+H);
     cY = map(cY * n,0,height/2,cY-H,cY+H);
     vertex(cX,cY);
     endShape();
    }
  
    for(int j = 0;j<ball.size();j++){
        if(dist(ball.get(j).x,ball.get(j).y,avgX,avgY)<550){
          count ++;
          ball.remove(j);
          ball.add(new Balls(random(width),random(height)));
        }
    }
  }
  fill(255);
  text(count,width-50,50);
}
import KinectPV2.KJoint;
import KinectPV2.*;


KinectPV2 kinect;


void setupKinect() {
  kinect = new KinectPV2(this);

  kinect.enableSkeletonColorMap(true);
  kinect.enableColorImg(false);

  kinect.init();
}


void updateKinect() {
  image(kinect.getColorImage(), 0, 0, width, height);

  ArrayList<KSkeleton> skeletonArray =  kinect.getSkeletonColorMap();

  //individual JOINTS
  for (int i = 0; i < skeletonArray.size(); i++) {
    KSkeleton skeleton = (KSkeleton) skeletonArray.get(i);
    if (skeleton.isTracked()) {
      KJoint[] joints = skeleton.getJoints();

      // update the PVector variables that we already declared
      handL.x = joints[KinectPV2.JointType_HandLeft].getX();
      handL.y = joints[KinectPV2.JointType_HandLeft].getY();
      
      handR.x = joints[KinectPV2.JointType_HandRight].getX();
      handR.y = joints[KinectPV2.JointType_HandRight].getY();
    }
  }
}
class Balls{
  float x,y, sX, sY,a;
  boolean done;
  color c;
  Balls(float posX, float posY){
    x = posX;
    y = posY;
    sX = random(5);
    sY = random(5);
    c = color(random(255),random(255),random(255),80);
    a = random(30,60);
  }
  float display(){
    strokeWeight(1);
    fill(c);
    ellipse(x,y,a,a);
    return x;
  }
  float move(){
    x += sX;
    y += sY;
    return y;
    
  }
  void check(){
    if(x<0 || x>width){
      sX = -sX;
    }
    if(y<0 || y> height){
      sY = -sY;
    }
  }
  void finished(boolean done){
    if(done){
    }
  }
}

Week 9+10 | Kinetic Interfaces | Kate

For this project I decided to try a few different knobs working off what we started in class. I kept the threshold slider because I thought it was very useful. Then I changed it so that instead of just swiveling randomly the user can only face a wall and move around the “room”. I wanted to add a central figure as almost a character which is the blue thing in the middle I made it a little proportional to how close one gets to things in the “room”. I decided to try a few different elements the 1 value sliders, the range slider, the knob, and toggles. The toggles have less to do about where things are so I separated them from the rest. I wanted a sort of wave look to the point clouds but for it made the whole point cloud image smaller.

import controlP5.*;

// IMA NYU Shanghai
// Kinetic Interfaces
// MOQN
// Mar 28 2018


import KinectPV2.*;

KinectPV2 kinect2;
PImage depthImg;

int flash = 5;
int thresholdMin = 1;
int thresholdMax = 4499;
//GUI
int guiX =10;
int guiY = 150;
int sliderW = 100;
int sliderH = 20;
int sliderGap = sliderH + 10;
int pointSize = 1;
float posX = 0.3;
float posY = 0.3;
float ang=0;
int ba = color(0);
int pt = color(255);
boolean tog = false;
boolean distort = false;
ControlP5 cp5;
Slider2D position;
Range depthRange;
Knob lookAround;
void setup() {
  size(800, 600, P3D);
  smooth();
  kinect2 = new KinectPV2(this);
  kinect2.enableDepthImg(true);
  kinect2.init();

  // Allocate a blank image
  depthImg = new PImage(KinectPV2.WIDTHDepth, KinectPV2.HEIGHTDepth, ARGB);
  cp5 = new ControlP5(this);
  //name doesnt matter
  depthRange = cp5.addRange("DepthRange")
    .setBroadcast(false)
    .setPosition(guiX,guiY+sliderGap*0)
    .setSize(sliderW*2,sliderH)
    .setRange(1,4499)
    .setRangeValues(1,4499)//sets initial values
    .setHandleSize(15)
    ;
  //for sliders names matter for what variable they are linked to
  /*cp5.addSlider("thresholdMin")
     .setPosition(guiX,guiY+sliderGap*0)
     .setRange(1,4499)
     .setSize(sliderW,sliderH)
     .setSliderMode(Slider.FLEXIBLE)
     .setValue(1)
     ;
  cp5.addSlider("thresholdMax")
     .setPosition(guiX,guiY+sliderGap)
     .setRange(1,4499)
     .setSize(sliderW,sliderH)
     .setValue(4499)
     ;
  */
  cp5.addSlider("flash")
     .setPosition(guiX,guiY+sliderGap*2)
     .setRange(1,10)
     .setSize(sliderW,sliderH)
     .setNumberOfTickMarks(8)
     .setValue(5)
     .setSliderMode(Slider.FLEXIBLE);
     ;
  
  //this ones doesnt
  position = cp5.addSlider2D("Position")
    .setSize(200,200)
    .setPosition(10,height-200-30)
    .setValue(0,0)
    //xmin,ymin,xmax,ymax
    .setMinMax(-width/4,-height/4,width/4,height/4)
    ;
  lookAround = cp5.addKnob("ang")
    .setRange(-PI,PI)
    .setValue(0)
    .setPosition(guiX,guiY+sliderGap*4)
    .setRadius(40)
    .setDragDirection(Knob.VERTICAL)
    .setNumberOfTickMarks(4)
    .setTickMarkLength(3)
    .snapToTickMarks(true)
    ;
  cp5.addToggle("light")
    .setPosition(width-75,25)
    .setSize(50,20)
    .setValue(true)
    .setMode(ControlP5.SWITCH)
    ;
  cp5.addToggle("distort")
    .setPosition(width-75,100)
    .setSize(50,20)
    .setValue(false)
    ;
    
}


void draw() {
  background(ba);
  
  thresholdMin = int(depthRange.getArrayValue(0));
  thresholdMax = int(depthRange.getArrayValue(1));

  // let's make a 3D space!
  // the origin(0,0,0) should be the center of the canvas
  pushMatrix();
  translate(width/2,height/2);
  translate(position.getArrayValue(0), 0,position.getArrayValue(1));
  //for rotation we should go mouseX to y rotation, mouseY to x rotation, flip y values
  rotateY(ang);
  noFill();
  stroke(pt);
  box(512,424,512);

  int[] rawDepth = kinect2.getRawDepthData();
  int w = KinectPV2.WIDTHDepth;
  int h = KinectPV2.HEIGHTDepth;
  depthImg.loadPixels();
  for (int i=0; i < rawDepth.length; i++) {
    int x = i % w;
    int y = floor(i / w);
    int depth = rawDepth[i]; // z

    if ( depth >= thresholdMin
      && depth <= thresholdMax
      && depth != 0) {

      float r = map(depth, thresholdMin, thresholdMax, 255, 0);
      float b = map(depth, thresholdMin, thresholdMax, 0, 255);

      depthImg.pixels[i] = color(r, 0, b);

      if (x % 3 == 0 && y % 3 == 0) { 
        float pX = map(x, 0, w, -w/2, w/2); 
        float pY = map(y, 0, h, -h/2, h/2);
        float pZ = map(depth, 1, 4499, 500, -500);
        stroke(pt);
        strokeWeight(1);
        float periodX = frameCount%pX;
        float periodY = frameCount%pY;
        float periodZ = frameCount%pZ;
        if(distort== true){
          if(pX%flash == 0){
             point(periodX*2, periodY*2, periodZ*2);
          }
        }else{
          point(pX, pY, pZ);
        }
      }
    } else {
      depthImg.pixels[i] = color(ba);
    }
  }
  depthImg.updatePixels();


  popMatrix();
  pushMatrix();
  stroke(0,0,255);
  fill(0,0,255);
  float scaler = map(position.getArrayValue(1),-height/4,height/4,1,2);

  translate(width/2,height-150*scaler);
  sphere(60*scaler);
  popMatrix();
  pushMatrix();
  translate(width/2,height-50*scaler);
  box(50*scaler,100,50*scaler);
  popMatrix();

  image(kinect2.getDepthImage(), 0, 0, w * 0.3, h * 0.3);
  image(depthImg, 0, 0, w * 0.3, h * 0.3);

  fill(255);
  text(frameRate, 10, 20);
}

void light(boolean flag){
  if(flag == true){
    ba = color(0);
    pt = color(255);
  }else{
    ba = color(255);
    pt = color(0);
  }
}

User Testing 2 (Kate)

Since the last user test I have been trying to make it more intuitive and make better user flow. The user interface is still a little confusing, but users have been able to navigate through it easily without help. In addition to these I added many more functions to the website. It is now capable of turning Chinese characters into pinyin and IPA. I have also added the data for the different accents and found a way to adjust the IPA spelling accordingly. I found a module to read the IPA aloud, but it doesn’t sound very great.

For this round I had 2 people from the last round and one new person to see if they could navigate it without knowing anything ahead of time. All people were from different countries so they potentially had different accents which helped with testing the speech recognition.

Only one of my users asked what to do, but it was not the user, but I think it was more because they thought there was too much text. Most users did not find the other pages on their own, so I think I need to increase the font size to make it more visible.

The interactive part of my website made all of my users very excited, but they all liked different parts. The user that doesn’t like to use the speaking part was excited that they could finally use Chinese on the site. But upon finding that feature didn’t understand why the IPA was helpful if we have pinyin, so I need to do better at explaining the IPA’s potential. Another user really enjoyed the speech recognition and trying to get the computer to detect the right words. The last liked the text to speech part the most. I think it is a good indicator that the interactions are fun enough for people to want to try a few times, but users are still missing the meaning behind them.

While testing it there were a couple technical difficulties and some concern over clarity, but they did not deter the users from trying things out.

Moving ahead I need to focus on user design, I want to make the buttons into icons to cut down on text. I also need to make the educational text more meaningful, because it is very dry right now. I will continue to add a couple more functions, but it is pretty complete right now as far as that goes. I have been looking at a similar types of websites, but they tend to rely images, but I think it would be too busy if I do.

 

Kinetic Interfaces | Midterm (Kate)

For this project I decided to make a sort of hybrid between Pacman and Snake. I decided to call it Light Runner. The player runs around and is always moving unless you use the keys to stop it. It draws inspiration from these two old arcade games mentioned above. The idea of making it a flashlight type tag game came from watching someone else’s project in a creative coding class. I never saw their code, but liked the aesthetic. There are monsters that chase you and spawn every minute. Part of the code was altered from code written by Daniel Shiffman.  The music is from Nightwatcher by Vegasis. The Leapmotion code is altered from class.

I started with the idea of basic flashlight tag and built in game features from there. I had a strong idea of what kind of functions I would need and just researched how to accomplish these. After researching I decided on the code that would be most useful to alter I went through it to find the parts I need and to understand how to apply them to my game. I used pixels so I could have the gradation effect. The light works by changing the pixels opacity of the black mask I made based on the distance from the center which was usually a character. The following happened by calculating distance between the two characters by using vectors and changing the speed based on the magnitude of the vectors. Most of the game features like the map and score were things that I added based on how I remembered older more retro games looking. I got feedback that the map made no sense to add, but I also added a time limit and sometimes the Leapmotion isn’t as responsive so I think it is helpful to keep the user from being too frustrated. Also it helped when I was checking to see if it worked. If I were to expand this game I would make it two player and add more tools. The ones I came up with were a little limited and based on things I knew I would be able to find a coding solution for. I think the idea that using the tools would have a negative effect is interesting as well. I was going to make tools with stronger negative effects than just not being able to control movement by using them, but didn’t come up with any good solution.

I started by making the functions and controlling it with keys and then adding in basic Leapmotion controls. The style is sort of retro so the controls are simple. I drew the art myself so it has a more uniform style. The 5 different tools you can have are a flashlight, lightning, food, tracker, and cloak. The flashlight allows you to shoot a beams from the center to the monsters, but you can’t see yourself at the same time. The lightning flashes when you close your hand, but by doing this you can’t move. The food boosts your speed but as a function of time. The tracker allows you to see where the monsters are. The cloak repels the monsters. All of these have lifespans and will spawn a new object after being used. You win when you have collected at least 10 items and there are at least 3 monsters. You die if any of the monsters touch you, but only 5 can spawn. The spawning happens every time your computers clock hits 1 min.

It can be a bit difficult to control, but does not read change the characters direction if your hand is closed so that smooths it out a little. You can switch hands, I originally was going to have one hand for the character and one for using objects, but I think this turned out easier for the user to get the hang of. The monsters seem a little slow when using the keyboard, but when using the Leapmotion it is an appropriate speed since the sensitivity of the Leapmotion can be a little unpredictable.

There is a little map in the top left to help you find the next item and the item you picked up will show int he top right, like your tool box. I chose to have no instructions just because I don’t like having them and I think the game is easy to catch on to.

For the actual process it started with me prototyping a light moving around in a controlled way. Then I added one monster and the player. Then I slowly expanded it I was afraid with the for loops and calculating distances between pixels constantly would slow it down, and sometimes it did, but it seems to run smoothly now. That is part of the reason the game is not at full screen. If I were to improve on this I would also make a type of background environment that would take some navigating with obstacles. More tools and playing options would also be added. I would also like to use the Leapmotion to more of its full potential to make it more interactive. The only real challenge I had was having the lights spawn properly and dealing with simple state machines. I am also not used to using a Leapmotion so I think I didn’t control the game as well as some people would be able to.

import processing.sound.*;

import de.voidplus.leapmotion.*;

//Some of these are not used, but I am a little anxious to delete them incase I actually used them
Me me;
Objects ob;
int fc,s;
boolean shoot = false;
boolean light = false;
boolean all = false;
boolean hide = false;
boolean boost = false;
boolean reached = false;
boolean flee = false;
boolean win = false;
boolean l,r;
int finCount,lastFin;
float grab;
ArrayList <Monster> Monsta = new ArrayList<Monster>();
ArrayList <Light> Lights = new ArrayList<Light>();
float obX,obY;
PImage mask;
boolean spawn, last;
PImage winner, loser;
boolean gameOver =false;
PFont pix;
int flashLight = 4;
int score = 0;
int sc;
PVector location;
int now;
LeapMotion leap2;
float pFingerX, pFingerY;
float fingerX, fingerY;
SoundFile back;
int threshold = 30;

void setup(){
  noCursor();
  size(1200,800);
  smooth();
  imageMode(CENTER);
  me = new Me(int(random(4)));
  location = new PVector(random(width),random(height));
  Monsta.add(new Monster(new PVector (random(width), random(height)),1));
  Lights.add(new Light(new PVector(me.pos.x,me.pos.y),new PVector(me.pos.x,me.pos.y)));
  leap2 = new LeapMotion(this);
  mask = createImage(1200,800,ARGB);
  ob = new Objects(location);
  LeapMotion_setup();
  pix = createFont("font/slkscr.ttf",32);
  winner = loadImage("winner.png");
  loser = loadImage("loser.png");
  frameRate = 24;
  s = int(random(5));
  spawn = false;
  last = false;
  finCount = 0;
  lastFin = finCount;
  back = new SoundFile(this,"Vegasis_-_Nightwatcher.mp3");
  back.loop();
}

void draw(){
  background(255);
  
  mask.loadPixels();
  LeapMotion_run();
  fc = frameCount/20;
  
  //gesture mostly for checking if you are there and if your hand is closed
  for (Hand hand : leap2.getHands()) {

    boolean handIsLeft         = hand.isLeft();
    boolean handIsRight        = hand.isRight();
    float   handGrab           = hand.getGrabStrength();
    float   handPinch          = hand.getPinchStrength();
    PVector spherePosition     = hand.getSpherePosition();
    float   sphereRadius       = hand.getSphereRadius();
    finCount = hand.getOutstretchedFingers().size();
    l = hand.isLeft();
    r = hand.isRight();
    grab = hand.getGrabStrength();
  }
  
  //Hand Part this is to move the character
  float accel = dist(pFingerX,pFingerY,fingerX,fingerY);
  if (accel > threshold) {
    float difX = abs(pFingerX - fingerX);
    float difY = abs(pFingerY - fingerY);
    if(grab != 1.0){
      if(finCount > 2){
        if(difX > difY && difX >50){
          if((pFingerX-fingerX)<0){
            me.right(boost,fc);
          }else{
            me.left(boost,fc);
        }
        }else if(difY > 50){
          if((pFingerY-fingerY)<0){
            me.down(boost,fc);
          }else{
            me.up(boost,fc);
          }
        }
      }
    }
  }
  //if(lastFin == 0 && finCount >= 3){
    if(s == 0 && ob.reached && grab == 1.0){
      shoot = true;
    }
  //}
  
  // save the current finger position
  pFingerX = fingerX;
  pFingerY = fingerY;
  
  /*this is what creates the flashlight affect
  the code is inspired from a Shiffman code I found for using a flashlight
  to reveal a picture but it worked a differently
  */
  for(float x = 0; x<width;x++){
    for(float y = 0;y<height;y++){
      int loc = int(x + y*width);
      //This shows the object you picked up
      mask.pixels[loc] = color(0);
      if(dist(width-50,50,x,y)<40){
        float dC = dist(width-50,50,x,y)*6;
        mask.pixels[loc] = color(0,0,0,dC);
      }
      //this is the orb around the user
      if(dist(me.pos.x,me.pos.y,x,y)<120){
        float d = dist(me.pos.x,me.pos.y,x,y)*2;
        if(!shoot){
          mask.pixels[loc] = color(0,0,0,d);
        }
      }
      //This is for when the light shoots out
      for(int i = 0;i<Lights.size();i++){
        if(dist(Lights.get(i).pos.x,Lights.get(i).pos.y,x,y)<40){
            float dLi = dist(Lights.get(i).pos.x,Lights.get(i).pos.y,x,y)*6;
            if(shoot){
              mask.pixels[loc] = color(0,0,0,dLi);
            }
        }
      }
      //This is for the tracking
      for(int i = 0;i<Monsta.size();i++){
        float mX,mY;
        mX = Monsta.get(i).pos.x;
        mY = Monsta.get(i).pos.y;
        if(dist(mX,mY,x,y)<40){
          float dMo = dist(mX,mY,x,y)*6;
          if(s == 2 && ob.reached == true){
            mask.pixels[loc] = color(0,0,0,dMo);
          }
        }
      }
    }
  }
  //This updates the monsters movement
  for(int i = 0;i<Monsta.size();i++){
    float mX,mY;
    mX = Monsta.get(i).pos.x;
    mY = Monsta.get(i).pos.y;

    Monsta.get(i).display(fc, flee);
    Monsta.get(i).steer(me.location(),false);
    Monsta.get(i).update();
    Monsta.get(i).arrive(me.location());
    //This checks if you died
    if(dist(me.pos.x,me.pos.y,mX,mY)<25){
      gameOver = true;
    }
  }
  //This is how the monsters spawn
  if(second() %60 == 0){
    spawn = true;
  }
  //This controls the spawning since it's in the draw loop and spawns a light orb each time as well
  if(spawn != last){
    if(Monsta.size()<5){
      float x,y;
      x = random(width);
      y = random(height);
      Monsta.add(new Monster(new PVector (x,y),int(random(1,4))));
      Lights.add(new Light(new PVector(x,y),new PVector(width/2,height/2)));
    }
  }
  //These are to check which tool functions should be activated
  if(s == 4 && ob.reached){
    flee = true;
  }else{
    flee = false;
  }
  if(s == 3 && ob.reached){
    boost = true;
  }else{
    boost = false;
  }
  //This updates the users movement
  me.display(fc);
  me.update();
  //These are the updates for if you use the flashlight
  if(shoot){
    for(int i = 0;i<Lights.size();i++){
      float mX,mY,lX,lY;
      mX = Monsta.get(i).pos.x;
      mY = Monsta.get(i).pos.y;
      lX = Lights.get(i).pos.x;
      lY = Lights.get(i).pos.y;
      if(dist(lX,lY,mX,mY)<10){
        shoot = false;
      }
      if(shoot){
        for(int j = 0;j<Monsta.size();j++){
          Lights.get(i).display();
          Lights.get(i).steer(Monsta.get(i).location(),false);
          Lights.get(i).update();
          Lights.get(i).arrive(Monsta.get(i).location());
        }
      }
    }
  }
  //This is how the light orb effect works with a mask of a black image playing with the opacity as a function of distance
  mask.updatePixels();
  //This updates the images shown whether the object is still out or being used or picked up
  if(!ob.reached){
    ob.hidden(s);
  }
  //These are the updates for the using the object and starting the countdown
  ob.pickedUp(me.location());
  ob.tool();
  
  //This spawns a new object
  if(ob.used()){
    s = int(random(5));
    location = new PVector(random(width),random(height));
    ob = new Objects(location);
  }
  //More update on using the objects at the right time
  if(s != 1 || ob.reached == false){
    image(mask,width/2,height/2);
  }else{
    if(grab != 1  && ob.reached == true){
      image(mask,width/2,height/2);
    }
  }
  
  //Using the object
  if(ob.reached){
    ob.display(s);
  }
  //This keeps score of how many items you used
  score += ob.score;
  if(ob.used()){
    score++; 
  }
  //This makes the object show up on the map since its dark
  noFill();
  stroke(255,0,0);
  rect(50,50,120,80);
  obX = (ob.pos.x/ 10)+50;
  obY = (ob.pos.y/ 10)+50;
  strokeWeight(5);
  if(!ob.reached){
    point(obX,obY);
  }
  strokeWeight(1);
  textSize(24);
  textFont(pix);
  //This is handles the text
  text("SCORE: "+score,50,height-50);
  text("LVL: " + Monsta.size(),width-160,height-50);
  //This checks if you won
  if(Monsta.size()>3 && score >=10){
    win = true;
  }
  textSize(72);
  //More text
  if(gameOver){
    fill(0);
    noStroke();
    rect(0,0,width,height);
    fill(0,0,255);
    stroke(0,0,255);
    text("GAMEOVER",3*width/4-textWidth("gameover"),height/2+32);
    image(loser,width/4,height/2);
  }
  if(win){
    text("YOU WIN",width/4-textWidth("you win"),height/2+32);
    image(winner,3*width/4,height/2);
  }
  //Restarts the game
  if(win || gameOver){
    if(grab == 1.0){
      Monsta.clear();
      Lights.clear();
      Monsta.add(new Monster(new PVector (random(width), random(height)),1));
      Lights.add(new Light(new PVector(Monsta.get(0).pos.x,Monsta.get(0).pos.y),new PVector(me.pos.x,me.pos.y)));
      me = new Me(int(random(4)));
      win = false;
      gameOver = false;
      score = 0;
    }
  }
  //Checks if things have happened to keep things from getting stuck in an infinity loop
  last = spawn;
  lastFin = finCount;
  //This checks if you're there
  if(!l && !r){
    fill(0,255,0);
    textSize(24);
    text("NO MOTION DETECTED",width/2-textWidth("NO MOTION DETECTED")/2,50);
  }
}
//This is the keyboard option I left it since it is easier to test with
void keyPressed(){
  if(keyCode == UP){
    me.up(boost, ob.timer());
  }
  if(keyCode == DOWN){
    me.down(boost, ob.timer());
  }
  if(keyCode == LEFT){
    me.left(boost, ob.timer());
  }
  if(keyCode == RIGHT){
    me.right(boost, ob.timer());
  }
  if(keyCode == ' '){
    me.stop();
  }
  if(keyCode == ENTER){
    if(s == 0 && ob.reached){
      shoot = true;
    }
  }
  if(key == 'q'){
    if(win || gameOver){
      Monsta.clear();
      Lights.clear();
      Monsta.add(new Monster(new PVector (random(width), random(height)),1));
      Lights.add(new Light(new PVector(Monsta.get(0).pos.x,Monsta.get(0).pos.y),new PVector(me.pos.x,me.pos.y)));
      me = new Me(int(random(4)));
      win = false;
      gameOver = false;
      score = 0;
      s = 0;//int(random(5));
      location = new PVector(random(width),random(height));
      ob = new Objects(location);
    }
  }
}

import de.voidplus.leapmotion.*;
//This is from class
// ======================================================
// Table of Contents:
// ├─ 1. Callbacks
// ├─ 2. Hand
// ├─ 3. Arms
// ├─ 4. Fingers
// ├─ 5. Bones
// ├─ 6. Tools
// └─ 7. Devices
// ======================================================


LeapMotion leap;


void LeapMotion_setup() {
  leap = new LeapMotion(this);
}


void LeapMotion_run() {

  int fps = leap.getFrameRate();
  for (Hand hand : leap.getHands ()) {


    // ==================================================
    // 2. Hand

    int     handId             = hand.getId();
    PVector handPosition       = hand.getPosition();
    PVector handStabilized     = hand.getStabilizedPosition();
    PVector handDirection      = hand.getDirection();
    PVector handDynamics       = hand.getDynamics();
    float   handRoll           = hand.getRoll();
    float   handPitch          = hand.getPitch();
    float   handYaw            = hand.getYaw();
    boolean handIsLeft         = hand.isLeft();
    boolean handIsRight        = hand.isRight();
    float   handGrab           = hand.getGrabStrength();
    float   handPinch          = hand.getPinchStrength();
    float   handTime           = hand.getTimeVisible();
    PVector spherePosition     = hand.getSpherePosition();
    float   sphereRadius       = hand.getSphereRadius();

    // --------------------------------------------------
    // Drawing
    // hand.draw();

    // ==================================================
    // 3. Arm

    if (hand.hasArm()) {
      Arm     arm              = hand.getArm();
      float   armWidth         = arm.getWidth();
      PVector armWristPos      = arm.getWristPosition();
      PVector armElbowPos      = arm.getElbowPosition();
    }


    // ==================================================
    // 4. Finger

    Finger  fingerThumb        = hand.getThumb();
    // or                        hand.getFinger("thumb");
    // or                        hand.getFinger(0);

    Finger  fingerIndex        = hand.getIndexFinger();
    // or                        hand.getFinger("index");
    // or                        hand.getFinger(1);

    Finger  fingerMiddle       = hand.getMiddleFinger();
    // or                        hand.getFinger("middle");
    // or                        hand.getFinger(2);

    Finger  fingerRing         = hand.getRingFinger();
    // or                        hand.getFinger("ring");
    // or                        hand.getFinger(3);

    Finger  fingerPink         = hand.getPinkyFinger();
    // or                        hand.getFinger("pinky");
    // or                        hand.getFinger(4);
    
    
    // fingerX = fingerIndex.getPosition().x; // ***
    // fingerY = fingerIndex.getPosition().y; // ***


    for (Finger finger : hand.getFingers()) {
      // or              hand.getOutstretchedFingers();
      // or              hand.getOutstretchedFingersByAngle();

      int     fingerId         = finger.getId();
      PVector fingerPosition   = finger.getPosition();
      PVector fingerStabilized = finger.getStabilizedPosition();
      PVector fingerVelocity   = finger.getVelocity();
      PVector fingerDirection  = finger.getDirection();
      float   fingerTime       = finger.getTimeVisible();

      // ------------------------------------------------
      // Drawing

      // Drawing:
      // finger.draw();  // Executes drawBones() and drawJoints()
      // finger.drawBones();
      // finger.drawJoints();

      // ------------------------------------------------
      // Selection

      switch(finger.getType()) {
      case 0:
        // System.out.println("thumb");
        break;
      case 1:
        // System.out.println("index");
        
        fingerX = fingerPosition.x;  // ***
        fingerY = fingerPosition.y;  // ***
        
        break;
      case 2:
        // System.out.println("middle");
        break;
      case 3:
        // System.out.println("ring");
        break;
      case 4:
        // System.out.println("pinky");
        break;
      }


      // ================================================
      // 5. Bones
      // --------
      // https://developer.leapmotion.com/documentation/java/devguide/Leap_Overview.html#Layer_1

      Bone    boneDistal       = finger.getDistalBone();
      // or                      finger.get("distal");
      // or                      finger.getBone(0);

      Bone    boneIntermediate = finger.getIntermediateBone();
      // or                      finger.get("intermediate");
      // or                      finger.getBone(1);

      Bone    boneProximal     = finger.getProximalBone();
      // or                      finger.get("proximal");
      // or                      finger.getBone(2);

      Bone    boneMetacarpal   = finger.getMetacarpalBone();
      // or                      finger.get("metacarpal");
      // or                      finger.getBone(3);

      // ------------------------------------------------
      // Touch emulation

      int     touchZone        = finger.getTouchZone();
      float   touchDistance    = finger.getTouchDistance();

      switch(touchZone) {
      case -1: // None
        break;
      case 0: // Hovering
        // println("Hovering (#" + fingerId + "): " + touchDistance);
        break;
      case 1: // Touching
        // println("Touching (#" + fingerId + ")");
        break;
      }
    }


    // ==================================================
    // 6. Tools

    for (Tool tool : hand.getTools()) {
      int     toolId           = tool.getId();
      PVector toolPosition     = tool.getPosition();
      PVector toolStabilized   = tool.getStabilizedPosition();
      PVector toolVelocity     = tool.getVelocity();
      PVector toolDirection    = tool.getDirection();
      float   toolTime         = tool.getTimeVisible();

      // ------------------------------------------------
      // Drawing:
      // tool.draw();

      // ------------------------------------------------
      // Touch emulation

      int     touchZone        = tool.getTouchZone();
      float   touchDistance    = tool.getTouchDistance();

      switch(touchZone) {
      case -1: // None
        break;
      case 0: // Hovering
        // println("Hovering (#" + toolId + "): " + touchDistance);
        break;
      case 1: // Touching
        // println("Touching (#" + toolId + ")");
        break;
      }
    }
  }


  // ====================================================
  // 7. Devices

  for (Device device : leap.getDevices()) {
    float deviceHorizontalViewAngle = device.getHorizontalViewAngle();
    float deviceVericalViewAngle = device.getVerticalViewAngle();
    float deviceRange = device.getRange();
  }
}




// ======================================================
// 1. Callbacks

void leapOnInit() {
  // println("Leap Motion Init");
}
void leapOnConnect() {
  // println("Leap Motion Connect");
}
void leapOnFrame() {
  // println("Leap Motion Frame");
}
void leapOnDisconnect() {
  // println("Leap Motion Disconnect");
}
void leapOnExit() {
  // println("Leap Motion Exit");
}

class Light{
  PVector pos, tar, vel, acc, shot;
  float maxForce, maxSpeed, a;
  /*the spawn is set at the middle it used to be at the location the
  player is at, but then I added more monsters and it doesn't update the users position well,
  so I just stuck with the center
  */
  Light(PVector target, PVector spawn){
    pos = spawn;
    tar = target;
    vel = new PVector(0,0);
    acc = new PVector(0,0);
    maxSpeed = 5;
    maxForce = 5;
  }
  void display(){
    update();
    check();
  }
  void update(){
    pos.add(vel);
    
    vel.add(acc);
    vel.limit(maxSpeed);
    acc.mult(0);
  }
  void check(){
    if(pos.x<0){
      pos.x = width;
    }
    if(pos.x>width){
      pos.x = 0;
    }
    if(pos.y<0){
      pos.y = height;
    }
    if(pos.y>height){
      pos.y = 0;
    }
  }
  /*This is similar to the monster code and was also altered from Shiffmans code
  it allows the light to spawn from the center and go to the monsters
  */
  void seek(PVector target){
    acc.add(steer(target,false));
  }
  void arrive(PVector target){
    acc.add(steer(target,true));
  }
  PVector steer(PVector target, boolean reached){
    PVector steer;
    PVector desired = PVector.sub(target,pos);
    float d = desired.mag();
    if(d > 0){
      desired.normalize();
      if((reached) && (d < 20.0f)){
        desired.mult(maxSpeed * (d/20.0f));
      }else{
        desired.mult(maxSpeed);
      }
      steer = PVector.sub(desired,vel);
      steer.limit(maxForce);
    }else{
      steer = new PVector(0,0);
    }
    return steer;
  }
}

class Me{
  float r;
  PVector pos,vel,acc, le,ri,up,dow;
  PImage meL,meR, meS;
  int fc,select, t;
  boolean still = false;
  boolean boost = false;
  boolean last = false;
  
  Me(int s){
    le = new PVector(0,-5);
    ri = new PVector(0,5);
    up = new PVector(-5,0);
    dow = new PVector(5,0);
    select = s;
    //This picks an initial direction
    switch(select){
      case 0:
        vel = new PVector(0,5);
        break;
      case 1:
        vel = new PVector(0,-5);
        break;
      case 2:
        vel = new PVector(5,0);
        break;
      case 3:
        vel = new PVector(-5,0);
        break;
    }
    pos = new PVector(random(width),random(height));
    acc = new PVector();
    //makes the animation
    meL = loadImage("personL.png");
    meR = loadImage("personR.png");
    meS = loadImage("personS.png");
  }
  void display(int f){
    //for some reason it won't face the exact correct direction
    r = vel.heading()+90;
    check();
    fc = f;
    fill(0,0,255);
    if(still){
      image(meS,pos.x,pos.y);
    }else{
      if(fc%2 == 0){
        pushMatrix();
        translate(pos.x,pos.y);
        rotate(r);
        image(meL,0,0);
        popMatrix();
      }else{
        pushMatrix();
        translate(pos.x,pos.y);
        rotate(r);
        image(meR,0,0);
        popMatrix();
      }
    }
  }
  void check(){
    //lets it wrap around this makes it easier for the player since the monster doesn't do this
    if(pos.x<0){
      pos.x = width;
    }
    if(pos.x>width){
      pos.x = 0;
    }
    if(pos.y<0){
      pos.y = height;
    }
    if(pos.y>height){
      pos.y = 0;
    }
  }
  void update(){
    pos.add(vel);
  }
  void up(boolean booster, int time){
    //this is the velocity vector and changes when we eat the food
    boost = booster;
    t = time;
    vel.y = -5;
    vel.x = 0;
    if(boost){
      vel.y = -t*0.1;
      vel.y=constrain(vel.y,-t,-6);
    }
    last = boost;
    
    still = false;
  }
  void down(boolean booster, int time){
    boost = booster;
    t = time;
    vel.y = 5;
    vel.x = 0;
    if(boost){
      vel.y = t*0.1;
      vel.y=constrain(vel.y,4,t);
    }
    last = boost;
    
    still = false;
  }
  void left(boolean booster, int time){
    boost = booster;
    t = time;
    vel.y = 0;
    vel.x = -5;
    if(boost){
      vel.x = -t*0.1;
      vel.x=constrain(vel.x,-t,-6);
    }
    last = boost;
    
    still = false;
  }
  void right(boolean booster, int time){
    boost = booster;
    t = time;
    vel.y = 0;
    vel.x = 5;
    if(boost){
      vel.x = t*0.1;
      vel.x=constrain(vel.x,4,t);
    }
    last = boost;
    
    still = false;
  }
  void stop(){
    vel.y = 0;
    vel.x = 0;
    still = true;
  }
  PVector location(){
    return pos;
  }
}

class Monster{
  PVector pos, tar, vel, acc;
  float maxForce, maxSpeed, a, fc, r;
  boolean still=false;
  boolean fl;
  PImage moL,moR, moPL, moPR, moBL, moBR;
  int selec;
  Monster(PVector target, int c){
    selec = c;
    pos = PVector.random2D();
    tar = target;
    fl = false;
    vel = new PVector(0,0);
    acc = new PVector(0,0);
    
    maxSpeed = c*0.9;
    maxForce = c*0.9;
    moL = loadImage("monsterL.png");
    moR = loadImage("monsterR.png");
    moPL = loadImage("moPL.png");
    moPR = loadImage("moPR.png");
    moBL = loadImage("moBL.png");
    moBR = loadImage("moBR.png");
  }
  /*This shows the monster there are 3 dif types the speed and color are related
  the rotation allows the monster to always be following the player, it was altered from Shiffmans code*/
  void display(int f, boolean fleeing){
    fl = fleeing;
    fc = f;
    update();
    r = vel.heading()+90;
    check();
    if(fc%2 == 0){
      pushMatrix();
      translate(pos.x,pos.y);
      rotate(r);
      if(selec == 0){
      image(moL,0,0);
      }else if(selec==1){
        image(moPL,0,0);
      }else{
        image(moBL,0,0);
      }
      popMatrix();
    }else{
      pushMatrix();
      translate(pos.x,pos.y);
      rotate(r);
      if(selec == 0){
        image(moR,0,0);
      }else if(selec==1){
        image(moPR,0,0);
      }else{
        image(moBR,0,0);
      }      
      popMatrix();
    }
  }
  void update(){
    vel.add(acc);
    vel.limit(maxSpeed);
    pos.add(vel);
    acc.mult(0);
  }
  //this only happens if the monster spawns at the corner because it moves by following a the users' vector
  void check(){
    if(pos.x<0){
      pos.x = width;
    }
    if(pos.x>width){
      pos.x = 0;
    }
    if(pos.y<0){
      pos.y = height;
    }
    if(pos.y>height){
      pos.y = 0;
    }
  }
  /*this code was altered from Shiffmans code and uses vectors to calculate the distance between the player
  and the desired target. The speed of pursuit is a function of the distance between the two. It does not travel
  in grid like movements like the user so it adds difficulty if the user keeps having to make extra moves
  */
  void seek(PVector target){
    acc.add(steer(target,false));
  }
  void arrive(PVector target){
    acc.add(steer(target,true));
  }
  
  PVector steer(PVector target, boolean reached){
    PVector steer;
    PVector desired = PVector.sub(target,pos);
    float d = desired.mag();
    if(d > 0){
      desired.normalize();
      if((reached) && (d < 20.0f)){
        desired.mult(maxSpeed * (d/50.0f));
      }else{
        desired.mult(maxSpeed);
      }
      if(fl){
        desired.mult(-1);
      }
      steer = PVector.sub(desired,vel);
      steer.limit(maxForce);
    }else{
      steer = new PVector(0,0);
    }
    return steer;
  }
  
  PVector location(){
    return pos;
  }
}

class Objects{
  PImage light, feet, flash, pizza, cape;
  PVector pos, loc;
  int sel,start,score;
  float x,y;
  boolean reached, done, last;

  Objects(PVector loc){
    score = 0;
    reached = false;
    flash = loadImage("flashLight.png");
    light = loadImage("lightning.png");
    feet = loadImage("feet.png");
    pizza = loadImage("pizza.png");
    cape = loadImage("cape.png");
    pos = loc;
    reached = false;
    done = false;
    last = false;
  }
  //switch case for spawning the different objects
  //this part shows the object you picked up
  void display(int s){
    sel = s;
    switch(sel){
      case 0:
        image(flash,width-50,50);
        break;
      case 1:
        image(light,width-50,50);
        break;
      case 2:
        image(feet,width-50,50);
        break;
      case 3:
        image(pizza,width-50,50);
        break;
      case 4:
        image(cape,width-50,50);
        break;
    }
  }
  //this is the object you can pick up
  void hidden(int s){
    sel = s;
    switch(sel){
      case 0:
        image(flash,pos.x,pos.y);
        break;
      case 1:
        image(light,pos.x,pos.y);
        break;
      case 2:
        image(feet,pos.x,pos.y);
        break;
      case 3:
        image(pizza,pos.x,pos.y);
        break;
      case 4:
        image(cape,pos.x,pos.y);
        break;
    }
  }
  boolean used(){
    return done;
  }
  //starts its lifespan
  void tool(){
    if(reached){
      start--;
      if(start==0){
        done = true;
      }
    }
    
  }
  //this checks if you picked up the object
  void pickedUp(PVector location){
    loc = location;
    if(dist(loc.x,loc.y,pos.x,pos.y)<20){
      reached = true;
      start = 200;
      if(reached != last){
        score = 1;
      }else{
        score = 0;
      }
      last = reached;
    }else{
      score = 0;
    }
  }
 
  int timer(){
    return start;
  }
}

Week 8 | Capstone Progress (Roopa)

Lately I have been working on making the website look better. I had originally tried to integrate some templates into the design, but it was too difficult to customize it. This picture below shows the landing page as it is now. The second picture shows what it looked like with the template, so I spent this past weekend trying to get it back to that look but by hard coding it. The third picture is to show I have usable links up as well. Another part of improving the look of the website has been trying to look at other more casual education type websites to see how they curate them and what kind of text they have. When doing the user testing I realized it was confusing for users to use the site because it had little instruction or explanation. This made sense as in the beginning I focused more on the functionality of the site and spent much of the time looking for Javascript modules that would suit my needs. Moving forward this week I plan to flush out the look of the pages and add more meaningful text, that was my original goal for this week, but I got caught up in the CSS. Currently it does support speech to text and translating the text written or spoken to the International Phonetic Alphabet (IPA).

Cur 

This project has an educational side as well so on top of designing and getting the website running I have been doing research to gather data for the site as well. Because of the nature of the data it was difficult to find a large database of the data in the desired form, but I was able to find one that provided data on a wide variety of people who speak English as a language other than their first. I was going to try HTML scraping, but the website was a little messy, so I went through and put it in a spreadsheet so I could easily use it. The next step for this part is writing equations for the different language generalizations and finding a way to implement them. I have found a module that can do text to speech for the IPA, but it sounds very robotic.

To be able to easily use the equations and show how things are shifting between accents I have just made the common IPA charts into spreadsheets so I can plot out how things shift and use them for the equations. I will have to make the decision on how much of the IPA I will be using as some phonemes exist when people speak English in an accent or dialect other than the neutral American one. But it would also make building the website and planning for how much of the IPA to let people explore more difficult.

For the hardware side I have begun trying to plot out how the IPA might look on a keyboard, I modeled it slightly after the Chinese keyboard for Zhuyin and Korean keyboards with vowels on one side and consonants on the other, but am still not 100% set on the design. For the actual keyboard I am building it will be in a square grid, but I would also like to make a keyboard for those trying the website without access to the special keyboard.

Below I inserted the p5.js code I had for visualizing the speech. I had originally had it show the entire wave, but I think I will try to make it into a plot of place of articulation for vowels. A version of something similar to the one below.

Image result for vowel place of articulation

Here is how someone can interact with it now. There are still some bugs, but after user testing I decided to make the text and spoken part more independent.

From user testing I think the most difficult part will actually be the curation of the website, which I didn’t actually account for, but is what I will need the most feedback on especially once I settle on a design.

var ch;
var mic;
var recording;
var clip;
var onOff = false;
var fft;

function setup(){
    createCanvas(400,400);
    background(0);
    mic = new p5.AudioIn();
    mic.start();
    recording = new p5.SoundRecorder();
    recording.setInput(mic);
    clip = new p5.SoundFile();
    fft = new p5.FFT();
    fft.setInput(clip);
}

function draw(){
    background(0);
    fill(0,255,140);
    text("tap to record",50,height/2);

    var waveForm = fft.analyze();
    if(onOff === true){
        fill(255,0,0);
        ellipse(width/2,height/2,20,20);
    }
    fill(200,30,180);
    for(var i =0;i<waveForm.length;i++){
        ellipse(map(i,0,1024,0,width),map(waveForm[i],0,255,height,0),5,5);
    }
}

function keyPressed(){
    if(key == 'o'){
        if(onOff === false && mic.enabled){
            recording.record(clip);        
            onOff = true;
            print('recording');
        }else{
            onOff = false;
            recording.stop();
            background(255);
        }
    }
    if(key == 'p'){
        clip.play();
    }
}

User Testing 1 | Kate

The goal for this round was for people to be able to input text or talk into the microphone and have the site recognize and translate it into the IPA. Going in to user testing I knew that I had not done much to curate the website and that really showed during testing. People were a little confused of what to do and what language things were being translated into. An unexpected coding error came up when the text wouldn’t change when switching from typing to using the microphone. It was still picking it up, but wasn’t displaying it.

I had one person who knew of the IPA test it and the rest did not. I tried it with people with different accents, but one of the users did not want to try the audio part. Which was okay since there is a typing option, but was not helpful in trying out the website. The biggest take away moving forward is how to convey what I am doing but also try to keep it light on the text. Most of the users did not move beyond the first page even though there were other tabs. I think part of it is because it is very clearly still in progress. But users were excited to see the translation and that it worked, so it is good that the interaction seems to intrigue users. However, I would like to try to make the controls more natural I am still getting a grasp of how to use the modules properly.