Kinetic Interfaces – Midterm Project (Francie)

Among the assignments in the first half of the semester, I am interested in exploring the webcam and playing with pixel manipulation. Inspired by the built-in widget in my Macbook, I came up with the idea of combining pixel manipulation with the tile game. Basically, a flat image is evenly divided into 16 tiles, and one of them is replaced by an empty position. Each time the tiles next to the empty space can be moved into it, and we can organize the disordered tiles in this way. In my project, I want to replace the still image with a real-time video shot by the webcam.


I started with capturing the video images from webcam and tried to divide the pixels into tiles. Later it turned out that this step required a large amount of mathematic calculation. Let’s use the top left tile as an example. First of all, I need to give a range for both x and y values, and here the coordinates are limited to 0 < x < width/4 and 0 < y < height/4. Then I take out the pixels from the original webcam images using the function index = x + y * width. Now I have picked out the index of the pixels that belong to the first tile of the total sixteen. Next, I am going to display these pixels in a new canvas with a 1/4 width and 1/4 height. By transforming the function into _index = x + y * width/4, I can cut out the 1/16 images of the video and put the pixels into a container called img11.

After I figure out the first tile, I then derive the formulas of other parts from it. The values of x and y in the remaining need to restore by deducting their displacements from the top left. Here are how to get the coordinates of the second horizontal line. The y values in the formula become y – height/4, while the x values are individually adjusted to x – width/4, x – width/2, and x – width*3/4 according to their positions.

Another challenge in this project is to switch the positions of the tiles. I fill in the bottom right tile with white color instead of updating the pixels from the video so that it looks like an empty space. To simplify the interaction, I use the keyboard to control the movement. The “switch case” function helps me to achieve the separate instructions while pressing different direction keys. However, I was stuck in this step for a long time because the printed coordinates were correct, but the images did not change at all. Here I must credit to my friend Joe who helped me figure out the solution. On the one side, it is significant to pay attention to the sequence of codes. The values assigned to the variables were changing, and I must be clear about what the variables in each line meant.

On the other side, the declaration of array also confuses me a lot. My friend used the below method to finally make the tiles movable, but I still do not understand why it is necessary to do the extra assignment.

And there is one more mysterious operation. In the beginning I created the images line by line, and it looked a little wordy. So I used a for-loop to generate the images in order to make the codes clearer. But somehow it did not work anymore. I feel that the two ways are the same, but actually only the complicated one is valid. Why is it problematic to use for-loop here?

The demo of the tile game comes below:

I really love this project and I want to add more creativity into this classic game. Currently the tiles do not appear randomly and I have to disorder them manually before I start to play. So it would be better if I could randomize the tiles. I also plan to improve the interaction by using leap motion or kinetic rather than keyboards. Moreover, I am willing to add 3D perspectives and projection mapping for a more attractive user experience.

Leave a Reply