Midterm Project: Facial Recognition Lockbox

Partner: Daniela Oh

Instructor: Antonius

For this project coming up with an idea was pretty difficult. We weren’t sure exactly what we wanted to do at first, but eventually we decided on something to do with facial recognition as we had found some cool API’s to use with it. We decided to make a simple lockbox with arduino that would only open when the owner of the box showed their face in front of it.

My personal role was on getting the facial recognition part working while Daniela worked on the arduino part of it and the actual hardware components like making the box and putting all the devices on it accordingly. My first thought was to make a basic architecture chart to figure out exactly how to get this whole thing running.

Architecture 1

Simple, but architecture charts help me figure out my exact workflow and know what each part is supposed to independently accomplish. Broken down, there were a few major steps, take the picture of the person, send that picture to the API to compare, return the similarity value received, and open the servo or keep it closed depending on the similarity value. Once I started fleshing out the steps, a lot of problems emerged, primarily, the processing libraries I found for facial recognition were either too slow or too weak. Most of them also just used machine learning principles and therefore depended on the strength of your machine to calculate all the values from scratch locally. I knew it would be quicker and more reliable to make a call to an established API from a facial recognition service and get the results that way. I’ve used Kairos before in python projects so I knew they were dependable and relatively quick (as long as you aren’t an idiot and try to update the existing gallery on their database every time you run the code instead of just once and then wonder why your code has such an abysmal run time…because…I totally didn’t do that).

This choice, while better for receiving reliable results, opened a whole new can of worms. How do I get processing and python to play nice together?? I looked into how a processing sketch can run a python script, but then there was either an issue with supplying and argument to the script or taking the output. I tried figuring out a way for processing to execute a batch command and do it all itself, but that didn’t go anywhere (there were some tips on it, but they were way over my head). I then thought of using socket programming to get them to talk to each other. I knew a little bit about setting up basic servers in python, so I decided to try that and see if I could get decent results. I kept trying slightly different things for each individual part and eventually, my architecture scheme ended up looking like this.

Networking Architecture Schematic

From processing to python back to processing and then to arduino. Overall, the final project lived up to what I wanted. Here’s the box that Daniela ended up putting together.

17670389_10212700029716575_517963405_o 17668835_10212700029676574_1534524846_o

The servo acts as a lock to prevent someone from opening it until the processing sketch gets the verification from the python server. Here it is in action.

So…it works. But that’s not to say it works to my satisfaction. If I had more time with the project, there are a lot more kinks to work out. I’ll go through some of them.

I need a good way to check the user closes the box and is done with it to automatically lock the box. I also need a better way to take the picture than with mouseclick. I want to install an arduino button for both of these tasks, but the pushbutton in my kit didn’t work out with the current design. On another hardware note, man the box is ugly. We’d like to make it much more aesthetically pleasing if we were to move further with it.

The biggest issue though? The server. I got a bare bones server up and running and did basically no error handling throughout it. Client disconnects? Server goes down. Client sends and error message? Server goes down. API returns an error? Server goes down. It is very finicky and I need to put some failure tolerance into the whole system. Either that or work with another idea I had which was using pipes to make both processing and python just search a directory repeatedly until a txt file with the info normally sent over the server appears. While that may be more failure proof, I think an optimized server setup will be quicker and less memory intensive.

Overall, I’m very proud of this project. Getting this done in a week is something that I’m pretty surprised about. On the surface the project seemed pretty simple, but every turn led to more and more issues so I’m glad we were able to at least get a prototype up and running in the time we had. I learned a lot about networking and working with external processing libraries throughout this project and would like to either extend this project or work on a similar one in the future.

 

 

Arduino Servo Code:

#include <Servo.h>
Servo servo;
int servoValue;
int trueFalse; //if it's daniela, true, else false
int value;

void setup() {
  servo.attach(9);
  Serial.begin(9600);
}

void loop() {
  servoValue = analogRead(0); //read save analog value
  while (Serial.available()){
    trueFalse = Serial.read();
  }
  if(trueFalse=='t'){
    servo.write(90);
  }else if(trueFalse=='f'){
    servo.write(182);
  }
  
}

Processing Client and webcam code:

import processing.net.*; 
import processing.serial.*;
import processing.video.*;
//initiate the client, capture  and serial objects
Client myClient;
Serial myPort;
Capture cam;

String file = "C:\Users\Main Character\Desktop\faceRecognition\kairos-face-sdk-python-master\pictures\temp\pic.jpg";
int newMessage;
int pictureFlag;
//this string will be used as a sort of buffer for processing to designate
//where it stored the file to
int newFile = 0;
void setup() { 
  size(640, 480); 
  //initialize the client on home ip with same port as python server 
  //(the port is arbitrary, just has to be the same as the server)
  myClient = new Client(this, "127.0.0.1", 5555); 
  //setup communication to arduino
  myPort = new Serial(this, Serial.list()[0], 9600);
  //setup webcam to take frames from stream
  String[] cameras = Capture.list();
  if (cameras.length == 0) {
    println("There are no cameras available for capture.");
    exit();
  } else {
   cam = new Capture(this, cameras[1]);
   cam.start(); 
    }     
  
} 

void draw() { 
  //read the camera stream and display that in the window
  if (cam.available() == true) {
    cam.read();
    set(0,0,cam);
  }
  
  //check the client is active. Then check if the string value has been changed
  //if the string value has been changed, send that value to the python server
  //then reset the string value to empty
  
  //return server output after it has processed the file
  //*TO DO*: handle dropped client connections 
  if(myClient.active()){
    if(newFile == 1){
      myClient.write(file);
      newFile = 0;
    }  
    String cur = myClient.readString();
    if(cur != null){
      newMessage = 1;
      pictureFlag = int(cur);
      println(cur);
    }
  }
  
  if(newMessage == 1){
    if (pictureFlag == 1){
    myPort.write('t');
    println("Welcome Daniela");
    }else{
    println("You are not Daniela");
    myPort.write('f'); 
    }
    newMessage = 0;
    pictureFlag = 0;
  }
  


} 

void mouseClicked(){
  cam.stop();
  saveFrame(file);
  newFile = 1;
  cam.start();
}

Python server code:

import socket
from _thread import *
import kairos_face
import os

#store your api key and id below
kairos_face.settings.app_id = '50d0c8e2'
kairos_face.settings.app_key = '1517e15dc89de6b9a27de5bad83afe78'
#designate your host and port info
#the blank host info designates it will accept any host
host = ''
port = 5555

#determine if this is first connection for the client
firstConnect = True

#create socket object s
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#attempt to bind the socket object to the designated host and port
#if there is a failure, print the error
try:
    s.bind((host,port))
except socket.error as e:
    print(str(e))
#listen for the processing client to connect to the server
s.listen(1)
print("waiting for a connection...")
   
#use the kairos api to take the file designated by the client and 
#compare it to the existing gallery in place
#return the verification value to denote similarity of the faces

def verify(filename,subject,gallery):
    recognized = kairos_face.verify_face(file=filename, 
                                           subject_id=subject,
                                           gallery_name=gallery)
    #print(recognized)
    confidence = recognized['images'][0]['transaction']['confidence']
    print(confidence)
    if(confidence > .85):
        return("1")
    else:
        return("0")


#function to handle the client and server interactions
def threaded_client(conn):
    #constantly check for information from the client side
    while True:
        data = conn.recv(2048)
        message = (data.decode("utf-8"))
        #reply = ("Server: " + message)
        print(message)
        verificationVal = str(verify(message, "daniela", "gallery1"))
        if not data:
            break
        conn.sendall(str.encode(verificationVal))
        
    conn.close()
    
while True:    
    conn, addr = s.accept()
    #on initiation notify the client they connected
    if(firstConnect):
        conn.send(str.encode("Server and client connected"))
        firstConnect = False
    #notify server side of connection info
    print('connected to: ' + addr[0]+":"+str(addr[1]))
    #create a new thread for server client communication
    start_new_thread(threaded_client,(conn,))

Digital Farm Midterm Project Idea

For my midterm project, I drew inspiration from the garden in a jars you sometimes see in stores. They are fully enclosed structures that allow the plant to grow with little to no input from humans aside from topping up the water every now and then. I wanted to create these for the purpose of cleaning the air. I read in one of the assigned readings about how this company was working on making cleaner air spaces in their offices and figuring out exactly how many plants were necessary per human there and then planning accordingly. I thought that had to be a lot of work, it would be a lot easier to buy prepartitioned amounts of plants and just get more if there are more people as opposed to engineering and entire rig before hand. Here is a diagram of my initial blueprint.

17474422_1440064839379612_1191681095_o

The entire project will be encapsulated inside a large water jug. The bottom layer will be the growth solution for the plants. The next layer up will be the growth medium. I’m still not sure on whether I’d like for the roots to be in the air, or in some medium like rockwool or soil. In air would be cleaner and get the solution to the plants quicker, but having soil would give more of a failure buffer in case there is an issue with water the plants. These layer coverings will be painted a dark color to protect from things like algae or overexposure to light. The next layer up will be the plants themselves. On the outside of the jug, there will be a fan and two holes through the jug, this is for the sake of ventilating the air inside the jug, both to control temperature and also to get the new cleaner air into the outside space. There will be a tube at the top connecting to the reservoir that allows for the user to add new growth solution or water to the jug.

The water solution will be fed to the plants on a timed cycle. There will be a pump in the bottom reservoir connected to an arduino that will start every 15 minutes (number subject to change with testing). The rig will use an ebb and flow system, simply filling up the upper chamber with the roots everytime the pump turns on and then allowing it to drain afterwards. When the pump is turned on, the fan will also turn on to circulate the air within the jug.

As for plant choice, I’m still not sure. I’m leaning towards spider plants and peace lilies for the sake of hardiness while still cleaning the air. Like it was mentioned in class its hard to tell exactly how much air is being purified without precise testing, so for now I just want to make sure the plants don’t die.

Overall, my goal is to make a system that requires as little human interaction as possible. I want for the human to be able to stick the entire rig into a corner and only think about it once every week or two to fill up water and be done with it.  I also want simple scalability. Serving the needs of one versus many shouldn’t become more and more difficult to create a large enough rig to house all the plants, it should just simply be getting a few more water jugs.

 

Lab 6: Serial Communication

Date: 3/17/17

Professor: Antonius

For this lab we practiced converting serial communication from Arduino to Processing and vice versa. The actual prompt for this lab was pretty broad, allowing for a lot of creativity in exactly how to implement it.

My original thought was to use my pre-existing mario animation and instead of binding the movement of mario to keyboard keys, to map it to the input of a potentiometer. I did that, but I couldn’t figure out the speed of it well and ended up trying a different project as I didn’t feel like troubleshooting it.

Instead I ended up making a wave generator. The processing code took in a value from the connected potentiometer. It then set that value to the y-value of the draw function. The x-value was naturally accumulating each frame. Once the x made it all the way to the right, the screen would wipe. It doesn’t make too much sense in text, but the video below should explain it better.

Overall, this was a pretty simple lab based around putting what we learned in class into actual use. I feel like I don’t totally understand serial streams yet, but debugging this work has helped out a lot.

Stupid Pet Trick

For this project we had to make a stupid pet trick, basically any simple arduino circuit that elicited a simple output to a user’s input. There were two major problems I had throughout all this, remembering to document, and keeping stupid tricks stupid.

Starting out, I decided to make some sort of arcade game, similar to the “see if you can hit this the hardest” punch machines, but the exact opposite. I wanted to use a piezoelectric sensor and make people touch it the softest to “win”. I put win in quotation marks here because what exactly win meant changed quite a few times over the course of this project.

I started on the first iteration. I tested that I could receive input and correctly update a temporary value tracking the player’s score based on the input the sensor received. Once that was completed, I hooked up a buzzer to sound every time someone got a new high score. This wasn’t too difficult, but I ran into a pretty big problem: the piezoelectric sensor took input all the time. This meant that just sitting there, the sensor would take a 0 as input and all of a sudden that’s the new high score without the user doing anything. I changed it so that it required a threshold value of 3 to count as a user response, but this came with another issue: getting a 3 really wasn’t that hard. Often times people would get a 3 on accident, and all of a sudden since that scores is the floor, that high score can never be beaten.

To try to at least remedy that issue a little, I decided to make the trick into more of a game between two people. The two would compete on separate sensors to see who could get the lowest score. This did help as people would quickly reset the scoreboard and try again to see who could get that illusive 3 first. This is the eventual setup that I went with for the pet trick fair, and people did seem to enjoy it a lot. The best reactions however were when I challenged people to get under a 3, a literal impossible task that they didn’t know was futile.  Shown below is the final circuit.17349237_1433582960027800_2139946291_o

Now for the two major issues. First, at the end of all this, I realized I really hadn’t documented anything yet. I jotted down notes the whole way, but as for pictures I had nothing. For projects moving on I’ll have to be more careful about keeping up with the changes in my work. The bigger learning point however was the difficulty of keeping a project stupid, especially a game. Once you make a game you want to do everything in your power to improve the user experience. This makes making a “stupid” game very difficult as it just seems like a “bad” game instead. I constantly wanted to reiterate over this project and add new features, such as a casing for it to make a pseudo arcade cabinet, graphics to play, added sound effects, game modes, and more, but at the end of the day I had to remind myself the scope of the task. Getting carried away is really easy.

 

 

int player1 = 0;
int player2 = 1;
int led = 7;
int buzzer = 3;
int High = 10000;
int cur = 0;
bool flag = true;

void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  pinMode(player1, INPUT);
  pinMode(player2, INPUT);
  pinMode(led, OUTPUT);
  pinMode(buzzer, OUTPUT);
}

void loop() {
  // put your main code here, to run repeatedly:
  //Serial.println(analogRead(0)/4);
  if (flag == true) {
    Serial.print("Player 1's turn ");
    cur = 0;
    while (cur < 3) {
      cur = analogRead(0) / 4;
      if ((cur < High && cur > 2) || cur == High) {
        High = cur;
        Serial.print(" High Score: ");
        Serial.println(High);
        newHighScore();
      }
      else if (cur > 2) {
        Serial.println(cur);
      }
    }
    flag = false;
    delay(500);
  }
  else {
    Serial.print("Player 2's turn ");
    cur = 0;
    while (cur < 3) {
      cur = analogRead(1) / 4;
      if ((cur < High && cur > 2) || cur == High) {
        High = cur;
        Serial.print("High Score: ");
        Serial.println(High);
        newHighScore();
      }
      else if (cur > 2) {
        Serial.println(cur);
      }
    }
    flag = true;
    delay(500);
  }
}

void newHighScore() {
  digitalWrite(led, HIGH);
  tone(buzzer,200);
  delay(50);
  digitalWrite(led, LOW);
  tone(buzzer,100);
  delay(50);
  digitalWrite(led, HIGH);
  tone(buzzer,200);
  delay(50);
  digitalWrite(led, LOW);
  tone(buzzer,100);
  delay(50);
  digitalWrite(led, HIGH);
  tone(buzzer,200);
  delay(50);
  digitalWrite(led, LOW);
  noTone(buzzer);
  

}

Hydroponics Project

For this project, my partner Joseph and I had to create a setup that would hold multiple plants and be able to water them automatically every 15 minutes and know when to stop watering them as well. Our original idea ended up being very close to our final product. We decided to use a drip system to put water into multiple individual bottles which would each hold a plant. Those bottles would have a gutter underneath them which would catch the extra runoff and put it into a reservoir with a pump. This pump would then push the water from this main reservoir into a second reservoir above where it would flow back into the drip system. This second reservoir has a moisture sensor that can tell the arduino to cut the pump once the second reservoir fills up to a specific height.

Once we were happy with this idea, we moved on to working with the arduino. This was my first time using a relay and connecting the components on the breadboard to control an external device such as the pump. The most annoying part was definitely the wires, they were a bit frayed and hard to work with, but after a lot of troubleshooting we got the basic moisture sensor working.

Once this worked we swapped out the LED for a relay connected to a power adapter and a pump.

 

With our arduino components working fine, we had to create an actual rig for the water to flow through. Our original idea was to use upside down water bottle tops to hold the plants, perforated water bottles bottoms to drip water through, a pvc pipe for the gutter system which we would drill the bottles into to secure them, and then some tubing to connect the various reservoirs. The only major change was not using pvc. While cutting through the bottles to get them to the correct shapes and sizes, I realized they could rather easily be slotted into one another. After doing this, glue gunning the seams to prevent water leaks, and then cutting holes to insert the upside down water bottles in, we had a working pipe to channel water back into the main reservoir.

The entire setup was extremely close to our initial plan and I’m actually excited about how well it turned out. The code for the arduino can be found below.

int moistSensor = A0;
int curMoist = 0;
int pump = 3;


void setup() {
  // put your setup code here, to run once:
  Serial.begin(9600);
  pinMode(pump, OUTPUT);
}

void loop() {
  // put your main code here, to run repeatedly:
  curMoist = analogRead(moistSensor);
  Serial.println(curMoist);
  delay(100);
  if(curMoist > 600){
    digitalWrite(pump,0); 
    delay(900000);
  }
  else{
    digitalWrite(pump,1);
  }
  
}

Chapter 13

  • What factors are needed for seed germination?

First the seeds must be viable, the dormancy conditions must be overcome, and the environmental conditions must be favorable. These conditions differ from plant to plant. Once these factors are overcome, the seeds need favorable temperatures, access to water, and access to oxygen.

  • What factor is the most critical factor for germination?

The most critical factor is temperature. Different seeds have different required and tolerated temperature ranges, in addition to different optimal temperatures where they germinate the best.

  • What is germination media and what is it optimized to do?

Germination media is specifically made to ensure soil to seed contact while still offering high water retention, optimal drainage, and allowing the seed to breathe. Often times the optimal conditions for germination of a seed are different than those for growing the actual plant afterwards.

  • Is germinating media more likely to be native soil or soilless?

Germination media is often made up of the same things as potting soil, such as peat moss, perlite, vermiculite and various slow released fertilizers. There are multiple textures available depending on seed size and how fine of a media it requires.

  • what is the difference between a landrace and an heirloom variety?

Landraces are locally adapted or traditional varieties of plants. Heirlooms are varieties whose characteristics have been saved and selected by farmers over the course of many generations.

  • what is an open-pollinator?

Plants that will self pollinate and create plants identical to their parents. This is also known as “breeding true”.

Hydroponics, Aeroponics, and Aquaponics

Hydroponics, Aeroponics, and Aquaponics are all methods of growing plants without the traditional method of putting a seed in dirt and watering it until you have a harvest. Whether it be for lack of farmland, or to save on water, these methods are being employed more and more commonly.

 

Hydroponics, as the name would suggest, has to do with water, particularly abandoning soil in favor of a water based nutrient solution. There are many setups for this, such as ebb and flow where grow trays are sequentially flooded with nutrient solution then drained to allow the roots to breathe, running water methods where the solution is continuously pumped through the area giving the roots the nutrients they need, or even film techniques where very thin layers of the solution are applied to the roots at specific intervals.

 

Aeroponics on the other hand abandons water as well. This method suspends the roots of the plant in air, however definitely not open air. Closed spaces are used to shelter the roots from pests and imbalances. They are then misted very often or have their nutrients directly applied to them in other methods.

 

Aquaponics is the combination of Hydroponics with systems of growing fish, often tilapia due to their hardiness. This cuts down on nutrients needed for the hydroponics system by taking CO2, ammonia to be turned into nitrate, and other waste from the fish. Once an aquaponics system gets up and running, it can be very efficient and feel like an actual ecosystem.

 

While these benefits sound great, you may be wondering why these methods haven’t completely eliminated traditional farming, but of course, there’s a cost. It’s expensive. The increases in yield can pay for the initial costs, but that startup investment just isn’t available to many farmers, and it’s not a one time cost, as parts of the system need to be constantly monitored and fixed. Even getting the systems running takes time as well. Setting up an aquaponics system from scratch and allowing all the bacteria and plants to really acclimatize to that system usually takes at least half a year. Overall, while this method may be great for locations restricted on necessary resources, traditional farming is still much easier and cheaper for the majority of the world.

 

Horticulture Today Chapter 10 Response

What is a photoperiod and how does it relate to photo quantity vs photo quality. What is “leaf burn”? What is Etiolation ? What is photoperiodism? How can plants be “force flowered”? What is phototropism? What is the Q10 Temperature Coefficient?

 

A photoperiod is the length of day and the relation of light quality that a plant receives. This directly relates to light quality, which is the spectrum of light that the plant is currently receiving, as well as light quantity, which is how much of that light the plant is exposed to. Different plants have different sensitivities to both light quantity and quality. Leaf burn occurs when plants get too much light quantity, especially common in more shade loving plants.

 

Etiolation is plant growth in response to lack of light, often accompanied by a loss of chlorophyll in the affected areas, a good example is cauliflower. It’s covered for the sake of getting the white flesh in the center.

 

Photoperiodation is the plant’s response to the amount of light it gets each day, this mainly matters for flowering plants. Forced flowering uses this concept of photoperiodation to induce plants into flowering quicker than they naturally would by exposing them to specific cycles of light. Plants also react to light in other ways, such as phototropism, or bending away or towards light sources, like sunflowers do. Temperature also affects plants growth stages, the Q10 Temperature Coefficient for example is a relation between the temperature and the reaction rate of a growing plant. Warmer is better, though plants do have a limit of how hot they can stand.

 

Soil Lab

In this lab we got soil samples from around Shanghai to test for chemical compounds. I got my soil from a flower bed near the Portman Center in Puxi. The green ferns in the bed were doing fine, but the flowers looked like they were dying so I wanted to know how bad the soil was and if it might have been the cause.

After collecting my soil I let it sit in water for 48 hours, however as a massive mistake, I skated to school since I was late and the soil solution went from being completely separated to mixed up and cloudy again. This mistake would make the rest of my tests very unreliable.

First I tested for Nitrogen. The testing process was similar for all the tests, first filling the solution to the specified line, then adding the reactive compound from the pill and shaking the solution vigorously. Afterwards it is left to sit and then you check the color it becomes to see how much of the compound it has. My nitrogen sample looked deficient as shown below.

17106229_1417530261633070_1191923071_o

Next was the potassium test, which as seen below, the water was too cloudy to get an accurate reading, though it does seem to have turned more orange. According to the paper that meant my soil was full of potassium.

17036736_1417530311633065_581309009_o

Next I tested for phosphorous. Despite the cloudiness of the water, it looked to match with the adequate color.

17091187_1417530264966403_1894614114_o

Checking PH, it seemed extremely blue, so I marked it down as very basic (7.5+ pH). Thankfully this test used soil instead of soil solution so I could be more sure of this one.

17091157_1417530324966397_445445977_o

With all my tests done, the professor gave me another test to do, the heavy metal check. As seen below, their didn’t seem to be any lead in my soil.

17092904_1417530331633063_1095653409_o

Overall, it was interesting to check out the composition of my soil. I really wish the tests could be more accurate but that was my fault for mixing up the solution again. Hopefully I can do it again in the future with less personal error.

Lab 4 Infrared Distance Sensor

In this week’s lab we made a circuit that uses a distance sensor to check the distance and make a buzzer sound when an object is too close. Here is the circuit we ended up making.

17092955_10212436771095274_834446319_o

Putting it together was pretty simple, we just followed the diagram on the sensor documentation and wired a buzzer to it. Calibrating the values was pretty annoying at first, but in the test code they already had a sort of map function so we just used the range values that it was printing to serial already. Once that was figured out, setting up the if statement ranges was simple.

The final project ended up being this:

 

Overall this was a simple lab. If we were to expand upon it we would add more sensors to get a better feel of the environment the arduino is in dynamically.