Syllabus

 

KINETIC INTERFACES

Topics in Experimental Interfaces & Physical Computing

Instructor: Jung Hyun Moon
Email: jh.moon@nyu.edu
Office Hours: Friday, 11am–12, 1–3pm, Sign up here.

Monday/Wednesday: 9:15 – 11:30am
Classroom: 824

 

Description

Kinetic Interfaces is an introductory course that highlights on programming and designing an interaction with body movement, which will enable the students to understand the fundamentals of emerging kinetic user interfaces. Everyday we become more familiar with interfaces where our movement serves as input to control interactive media, and kinetic user interfaces allow users to interact with computing devices through the motion of bodies.

Students in this course will use Computer Vision and motion tracking tools and techniques to create kinetic interfaces that exploit the body’s movement capacity to control software and hardware systems. Both practical and creative applications of kinetic interfaces are to be investigated as students are challenged to design their own solutions. Webcams, the Leap Motion Controller and the Microsoft Kinect will all be considered as input devices. Students will be introduced to the topics of pixel manipulation, as well as face, hand, blob and skeletal tracking. Projection mapping, a technique that turns objects or surfaces within an environment into dynamic display surfaces, will be explored as an output method.

 

Pre-requisites

The class uses Processing and requires Interaction Lab or similar programming background. Three.js or C++ language in OpenFrameworks is also encouraged.

 

Learning Objectives

Upon completing the course, students will:

  • Understand digital images and the pixels that compose them
  • Learn programmatic methods for the manipulation of digital images and video
  • Learn the diverse methods of motion tracking
  • Apply motion tracking methods to user interfaces
  • Create a practical or artistic kinetic user interface
  • Have the knowledge to discuss and develop meaningful and effective kinetic interactions

 

Office Hours

By appointment at 5-8pm on Thursdays.

The instructor is also available via email and will respond to students’ questions as quickly as possible. Asking for help is strongly encouraged.

 

Assignments

Students are expected to complete assignments on time. Late assignments will not be accepted. Misunderstanding of an assignment is not an excuse for lateness. All assignments should be documented on the documentation blog. Undocumented work will not be considered complete.

Weekly Assignments

Weekly Assignments vary depending on the topics covered in class. The topics will be as written on the syllabus, but are subject to change, and more detailed information will be discussed in class.

All exercises should be uploaded to Kinetic Interfaces google drive folder by midnight every Saturday so that the instructor has enough time to review your work. You should upload not only your source code, but also all of your media assets, such as image, sound and video, so that the instructor is able to execute the program.

Additionally, your work process should be documented on the documentation blog. Your documentation needs to include sufficient amount of screenshots and video captures along with descriptions.

You can always find the example codes here: http://ima.nyu.sh/kinetic-interfaces/sample-codes/

Midterm Project

Create a screen-based application which interacts with body, facial movement or sound. It can be anything useful, interesting or fun. Make sure that you document your work on the documentation blog. You may choose to work alone or with a partner.

  • Midterm project concept presentation is in Week 7
  • Midterm project presentation is in Week 8

Final Project

Create a kinetic user interface utilizing any methods or devices covered during the class. Students are strongly encouraged to explore the diverse tracking methods and find their own ways how they effectively interact with body movement. The output can vary. It can be anything, such as a drawing tool, informative installation, entertaining game, dance performance, or musical instrument.

Individual assignments will be given accordingly after the concept presentation. Be sure to post it to the documentation blog. Include details about all phases of development, both successes and failures. You may work in groups of two.

  • Final project concept presentation is in Week 12
  • The deadline of the final project presentation and documentation is in Week 14

Required Readings

  • Learning Processing – A Beginner’s Guide to Programming Images, Animation, and Interaction by Daniel Shiffman
  • The Art of Interactive Design by Chris Crawford

 

Grading Components

  • Attendance & Participation: 25%
  • Weekly Assignments: 25%
  • Midterm Project: 15%
  • Final Project: 35%

It follows the grading practices of NYU Shanghai.

 

Grading Scale

  • A: Excellent performance showing a thorough knowledge and understanding of the topics of the course; all work includes clear, logical explanations, insight, and original thought and reasoning.
  • B: Good performance with general knowledge and understanding of the topics; all work includes general analysis and coherent explanations showing some independent reasoning, reading and research.
  • C: Satisfactory performance with some broad explanation and reasoning; the work will typically demonstrate an understanding of the course on a basic level.
  • D: Passable performance showing a general and superficial understanding of the course’s topics; work lacks satisfactory insight, analysis or reasoned explanations.
  • F: Unsatisfactory performance in all assessed criteria.

 

Attendance, Participation and Classroom Policies

Absences

Attendance in all class sessions is mandatory. It is expected that students will come to all classes, unless the instructor has been notified beforehand. One unexcused absence is permitted. Any unexcused absence after that will count as third of a letter grade off your final grade (A to A-, A-to B+, etc.). 3 unexcused absences will result in failure of the course. If you miss a class it is your responsibility to find out what happened that day in class (i.e. Obtain hand-outs distributed, assignments, etc.) Please let the instructor know ahead of time so that he/she can help you determine how to make up the material.

NYU policy permits members of any religious group to absent themselves from classes without penalty when required for compliance with their religious obligations.

Lateness

This course relies heavily on punctuality. The class starts promptly at the scheduled start time, so students are expected to be on time. Arriving more than 15 minutes late or leaving more than 10 minutes early will be considered an unexcused absence.
This class will be highly participatory with lectures being very conversational. Active participation is essential and expected in this course.

Use of technology

The use of mobile devices or personal electronic equipment is not permitted. Mobile devices must be turned off or silenced and put away during class. Anyone who is observed doing activities not related to the class, such as recreational browsing of the internet, including all social media websites, email and instant messaging, game playing, and work for other classes, will result in a reduction of your participation points.

Laptops are only allowed for note taking, in class work, as well as relevant research only. Your laptop should always be closed whenever a fellow student is presenting.

IMA Equipment

This course may require the use of equipment from the IMA Equipment Room. Policies and procedures for checking out, caring for, and returning equipment will be discussed during IMA Orientation as well as in class.
Keeping IMA equipment past due dates, abusing the equipment or failing to adhere to the policies of the IMA Lab will affect your grade in this course.

Academic Integrity

The presentation of another person’s words, ideas, judgment, images or data as though they were your own, whether intentionally or unintentionally, constitutes an act of plagiarism.

Students must retain an electronic copy of their work until final grades are posted on Albert. They must be prepared to supply an electronic copy if requested to do so by NYU Shanghai. Not submitting a copy of their work upon request will result in automatic failure in the assignment and possible failure in the class.

 

Weekly Course Schedule

 

Week 1: Introduction to Kinetic Interfaces

Lecture 1

  • Class introductions
  • Overview of materials to be covered and project expectations
  • Introduction to Kinetic Interfaces
  • Discussion about new media technologies, interaction design and kinetic user interfaces

Lecture 2

  • Review of programming basics in Processing
  • Iterations (for-loop, while)
  • Transformation functions (push/popMatrix)

Assignment

  • Video: Art that looks back at you, TED talk by Golan Levin
  • Reading: Computer Vision for Artists and Designers by Golan Levin
  • Video: What is Zero UI?, by Andy Goodman from Fjord
  • Reading: What is Zero UI?, by John Brownlee, based on Andy Goodman’s talk
  • Reading: Learning Processing – Chapter 2 Processing
  • Reading: Learning Processing – Chapter 4 Variables
  • Reading: Learning Processing – Chapter 6 Loops
  • Reading: Learning Processing – Chapter 14 Translation and Rotation
  • Research online about innovative and interesting interfaces. Pick one from your research and prepare a casual presentation for next week.

 

Week 2: Programing Fundamentals

Lecture 1

  • Kinetic Interfaces Research Presentation
  • Discussion about kinetic interfaces in depth, interfaces that can be created, technology that can be used, how to utilize them, how to improve them
  • Overall review of programming basics
  • In-class Exercise: Creating a simple drawing tool

Lecture 2

  • Review of Object-Oriented Programming
  • Variable – Array – Struct – Class
  • Array vs. ArrayList
  • In-class Exercise: Creating a drawing tool using particles (Objects and Arrays)

Assignment

  • Reading: Learning Processing – Chapter 7 Functions
  • Reading: Learning Processing – Chapter 8 Objects
  • Reading: Learning Processing – Chapter 9 Arrays
  • Reading: Learning Processing – Chapter 22 Advanced Object-Oriented Programming
  • Create a simple animation using Objects and Arrays.

 

Week 3: “Abstract” Button & Button States: Triggered, Pressed and Released

Lecture 1

  • Assignment review
  • Methods of creating an “abstract” button with programming
  • In-class Exercise: Creating a simple sketch with buttons and mouse interaction

Lecture 2

  • Methods of creating button states (triggered, pressed and released)
  • In-class Exercise: Creating button states (triggered, pressed and released) with mouse interaction and triggering sounds

Assignment

  • Reading: Learning Processing – Chapter 5 Conditionals
  • Reading: Learning Processing – Chapter 20 Sound
  • Program a simple interaction with “abstract” buttons and button states.

 

Week 4: Introduction to Computer Vision and Pixels

Lecture 1

  • Assignment review
  • Introduction to Computer Vision
  • Array(1 Dimensional data) vs. Pixels (2 Dimensional Data)
  • In-class Exercise: Practicing Pixel Iteration with an image

Lecture 2

  • Pixel Manipulation with an image
  • Real-time Pixel Manipulation with images from webcam
  • In-class Exercise: Practicing Pixel Manipulation

Assignment

  • Reading: Computer Vision for Artists and Designers by Golan Levin
  • Reading: Learning Processing – Chapter 1 Pixels
  • Reading: Learning Processing – Chapter 15 Images
  • Reading: Learning Processing – Chapter 16 Videos
  • Program an interactive or alternated portraiture with Pixel Manipulation

 

Week 5: Pixels: Color Tracking

Lecture 1

  • Assignment review
  • Methods of getting the center coordinates of an area detected by selected colors
  • Using a Threshold(Tolerance) variable to control the sensing area

Lecture 2

  • Using a Threshold(Tolerance) variable effectively to improve sensing accuracy and stability
  • In-class Exercise: Combining the methods of Pixel Manipulation and creating buttons in order to create a Color Tracking
  • Discussion about usage of Color Tracking

Assignment

  • Create an interactive sketch with Color Tracking

 

Week 6: Leap Motion and Face Tracking

Lecture 1

  • Assignment review
  • Introduction to Leap Motion
  • Practice the button states (triggered, pressed and released) with Leap Motion
  • Gestures with Leap Motion

Lecture 2

  • Introduction to Face Tracking
  • Overview of Face Tracking libraries
  • In-class Exercise: Creating a musical instrument with facial movements

Assignment

  • Prepare Midterm Project Concept Presentation.
  • Create a simple interaction with Leap Motion or Face Tracking, including the button states (triggered, pressed and released)

 

Week 7: Projection Mapping and Midterm Project Concept Presentations

Lecture 1

  • Midterm Project Concept Presentations

Lecture 2

  • Assignment review
  • Introduction to Projection Mapping
  • Basics of Syphon and MadMapper
  • Keystone library and Syphon library in Processing
  • In-class Exercise: Practice Projection Mapping onto multiple surfaces

Assignment

  • Develop Midterm Project

 

Week 8: Midterm Project Presentation

Lecture 1

  • Work on Midterm Project

Lecture 2

  • Midterm Project Presentations

 

Week 9: Introduction to Kinect and Processing GUI Libraries

Lecture 1

  • Introduction to Kinect
  • Depth Data and Point Cloud
  • Methods of utilizing Depth Data
  • Discussion about usefulness of Depth Data
  • In-class Exercise: Creating a visualization of Point Cloud from the Depth Data with Kinect

Lecture 2

  • Introduction to Processing GUI library
  • In-class Exercise: Applying and using the GUI library

Assignment

 

Week 10: Kinect: Point Cloud + Abstract Buttons

Lecture 1

  • Assignment review
  • Introduction to Kinect
  • Using Point Cloud to receive particular x and y coordinates
  • Method of getting the coordinates of the nearest point
  • Method of getting the coordinates of the center of a particular sensed area

Lecture 2

  • Method of creating an “abstract” button in 3D space
  • Method of triggering the button by detecting whether a body is in the area
  • In-class Exercise: Creating Motion Tracking with Point Cloud

Assignment

  • Create an interactive tool that includes the 3 button states (trigger, pressed and released)

 

Week 11: Kinect + OpenCV

Lecture 1

  • Assignment review
  • Introduction to OpenCV
  • Overview of OpenCV functions (blob, contour)
  • Blob Tracking and getting the ID of blobs
  • Contour Tracking and its usefulness
  • Practicing Blob Tracking with IDs

Lecture 2

  • Combining Kinect and OpenCV library
  • Programming a sketch to get blobs or contour with OpenCV Library
  • Improving the sensing quality by combining Kinect and OpenCV library

Assignment

  • Prepare Final Project Concept Presentations.

 

Week 12: Kinect: Skeleton Tracking and Final Project Concept Presentations

Lecture 1

  • Assignment review
  • Motion tracking with Skeleton Tracking
  • Comparison to other tracking methods

Lecture 2

  • Final Project Concept Presentations

Assignment

  • Develop Final Project.

 

Week 13: Final Project Development

Lecture 1

  • Work on Final Project

Lecture 2

  • Work on Final Project

Assignment

  • Develop Final Project.

 

Week 14: Final Project Presentations

Lecture 1

  • Final Project Presentations(1)

Lecture 2

  • Final Project Presentations(2)