【Kinetic Interface】Kinect for UE4 TPS Blueprint (Kevin Xu and Jiayi Wang)

Final Preview

Build with UE4.17.2. Upward compatible until 4.18. Kinect plugin is required for the project which can be found here. The Kinect plugin is not compatible with 4.19 and any newer version by May 2018.

Project download link: https://drive.google.com/file/d/19AIs7NBNx7OkzGriS_T02x3lrI_zw4wf/view?usp=sharing

The final version of the blueprint is named as 3rdpersonKinect in the root directory.

The final product is a drag-and-drop ready character controller for all developer to use. The pawn comes with a local motion which means that the mesh actually follows the user around with responding animations. The Blueprint is constructed with a following camera and a gun that points at the direction of the player’s right hand.

The development of this project started with the “Kinect poseable mesh” BP that comes with the Kinect plugins. This blueprint consists of a mesh that will follow the relative movement of the rig. Which means that the mesh will not move around but rather stays at one point and do all the movement. Apart from this, since the original idea of the project is to make a multiplayer game based on Kinect. It’s necessary to assign the correct body number (body Enum as the variable in the blueprint)

As it’s recorded in the video above, the number is randomly generated and there’s no way to control that. As a result. we assigned whichever number that gets detected first to one body and the second to the other. Thus solved the random rig number problem.

As shown in the video, we now have to mesh that animate according to the two players. It will not mess up the animation between the two players. However, there’s still no local motion.

There’s a lot more calibration needed to the blueprint in order to improve the user experience. Since none of us have the experience with using C++ so it was really a struggle to construct everything by only using the node editor. If we had more time, we would definitely want to add functions such as reloading, teleporting (like VR pawn) or even combine this with VR.

【Kinetic Interface】Final Proposal (Kevin Xu and Jiayi Wang)

Presentation Slides:https://docs.google.com/presentation/d/1ai50ob6ezz_TZQRzUVHuS1AfxMMMpVi5pcNnVHDsvxo/edit?usp=sharing

For the final project, we want to make a sidescroller style 2 player fighting game with the Kinect as the controller. Different from traditional fighting games like Street Fighter that has a lot of combo attacks, we want our game to stick with reality. Which means the characters will attack each other in the way that the players actually move in reality. The difficulty of this project would probably be the collection-detection and damage based on that.

Capstone_The Oracle_Jiayi Wang


In mythology and ancient time, the oracle was an enlightenment given from the divinities high above to us, the ordinary human living on the earth. It’s a revelation that will guide the actions of the receiver. However, in modern days, these words come not only from gods. With the development of the internet, we are given the opportunity to communicate with people from a far distance away. The anonymous identity empowered us to be anyone. Sometimes, it is another self that is hidden in the social life of the real world.

Oracle is a voice interactive statue that will respond to the audience in a very unsympathetic and intimidating manner. When the audience approaches the statue, the microphone captures every comment from the audience and send to the “god”, the chatting system, and thus begin a conversation between the audience and the “god”. The indifference and alienation that magnified on social media from an individual that is unreachable is the inspiration of Oracle. According to research, in online social life, most internet aggressive behavior is linked with the nature of anonymity online. The carefree online behavior code brings the users courage of saying anything they want to anyone without having to be responsible for it. For Oracle, the ruthless words from the stony statue are judgments casts upon the audience from the standpoint where the statue, the spirit behinds it, doesn’t need to be responsible for its words and it’s beyond the reach of the audience. Destroying the statue won’t stop it from talking nor put an end to the malice behinds it. A statue is eventually a statue, and the code is only ones and zeros.

Internet violence is not an easy problem to solve. The distance between users leads to the question of “why should I even care”? Therefore, it’s almost impossible to let all users online to behave themselves. More importantly, as the victim of internet violence, how should we act upon and take in the “oracle” from afar. The “god” chatting system is being mean because it is a piece of code hiding inside the computer, beyond the reach of the audience.

【Video Games As Arts】Final Project-Tier

For the final project of this class, I proposed to finish my midterm project. The final product shifted greatly from the midterm project file. Initially, the game was constructed with a 2d platformer kit which the overall style of the game was sprite based with a pixel style. However, partially because of the ring menu system that I was trying to combine to the gameplay didn’t work well with the original 2d game system that I worked with and I’m not really familiar with constructing a 2d scene in Unreal, but most importantly, in order to emphasize on the feature of “exploring the city corners”, a 3d layout is much better than what I had before.

Project File:


Build with UE4.18

This is a demo of the ring menu system that I was trying to combine into the gameplay. I haven’t figured out the activating/deactivating mechanism that is going to work with the leveling up of the main characters.

Here’s a navigation through the map. I build a street that has corners and turns that has the potentiality of expanding to a full district that is big enough for the player to explore. The characters are tied with animation as suggested but didn’t have enough time to add walking animation and trigger animation.

For the final project, I dug too much into the trying to have the specific interface into the game that the overall mechanism. If there’s more time, I would definitely combine the mechanism in both projects together and make a complete gameplay.


【Video Games as Art】Final Proposal | Jiayi Wang


↑↑↑↑ Here’s the link to the presentation slides

I chose to work on the midterm project for my final. Mainly because I really like the idea of exploring the city and life through the eye of an animal. I can see a potentiality of this game and I really want to finish it.

Project Title


Description of project goals and purpose:

Tier is a 2d sidescroller game where the player acts as an abandoned pet that seeks survival in a big city. The player (cat) cannot understand human language nor their gestures and can only try its best to ask for food and water from the passersby on the street. However, a cat learns a lot of stuff through time. The player can acquire more skills after survived and spend more time with the human.  The more skillful the player is, the more likely the player will get adopted by kind people and no need to worry about the rest of its life.

The main focus of the game is not purely asking food from human but experience the city life as a small cat. The player will interact with NPCs from different social classes, from the wealthy to the homeless. Through the gameplay, the player will notice that not all human that lives in this city acts consistently as they seem to be. There are insidious “kind” people and kind-hearted “mean” looking people.

There will be 3 possible endings to the game. The player will probably get adopted by the passersby they meet on the street, or get adopted by (or rather chose to be the companies of) homeless people and offer help to other homeless animals, or they just couldn’t make it through the days.

Project Design and Production

  • Opening with the main character get abandoned and left on the street by a car
  • Human walking on the street
  • NPC animals that will help the player
  • Sprite animation
  • Rebuild level tile map and refine the city construction
  • Make more animal characters for NPC and possible further development of the game
  • Widget interaction interface
  • Refine HUD and lightings

What system is required to play the game?


【Kinetic Interface】Week11 Interaction with Kinect | Jiayi

I tried to create a drawing interaction with Kinect. There seems to be a problem with the Kinect working on my dorm pc stand. It disconnects about every 30 seconds or so and will reconnect after 10 to 20 seconds of delay. It is probably due to a power supply problem when multiple devices are plugged into the same plugboard since the same Kinect was working perfectly fine at school. Here’s the demo video

Capstone Progress 2 (Oracle)


Finished the construction of the inner kinetic structures. The shell of the sculpture is already cut apart and ready to handle movement. All parts are uncolored and remain as the white color of the plaster.

User Testing Result:

The user was tested with chatbot and presented with the assembled sculpture (couldn’t move at the point). The users all went through the dialogue and were asked if the appearance of the sculpture feel consistent enough.

Two of the users pointed out that the conversation with the chatbot is now lack of hints and indications so that they were kind of lost during the middle. The users suggested that the keyword detection of the chatting system wasn’t good enough. When they are saying sentences with similar meanings but not containing the programmed keywords, the bot wasn’t able to understand it and as a result, the conversation wasn’t able to move forward.

The user also suggested that it would be a nice and surprising thing to see. However, the user wasn’t content with the coloring of the sculpture. The pure white appearance of the sculpture wasn’t able to highlight the facial features of the sculpture and it was too pure and innocent according to the user.

What to do next:

Finish assembling mechanical arms with the sculpture over the weekend. Start to refine the chatting system starting next week. Consider recolor the sculpture and polish the details so that it is consistent with the tone of the interaction.

【Game Art Mod】Pushing Against Boundaries (Krom)

The player acts as the boss “Dragonslayer Ornstein”


In this mode, the player is able to control the boss in the original Dark Souls game. The active area is no longer limited to the boss fight zone but rather the entire map. Due to the vertical integration design of the Dark Souls map, the “boss” player is capable of moving around through almost the entire world of Dark Souls. The size, skill set, attacking methods, and moving abilities (flying, dashing, and etc) remain the same as the original AI-controlled boss. Meanwhile, the mod also keeps the original undead players.

One of the essences of the Souls games is the repetitive death of the undead(player) while exploring the map and fighting the tough ai enemies. As a result of the death, the player loses all of their “souls” (currency of the game) and in order to recover their souls, they need to fight all the way back to the point where they got killed previously. To create this process, the boss and the regular enemies that scattered around in the world are designed to be much stronger than the player. Usually speaking, they are capable of killing the player within 3 to 4 attacks. This mod empowers the players with greater ability than the undead player. Now that they are the ones who are able to mess around with the weak undead. It is interesting to see how the boss player will interact with the undead player.

For Souls games, beating the bosses and explore the storyline are the objectives that push the player to move forward since it is not an open world game. Therefore, having a tough boss staying around for the player to defeat if really important for the gaming aspect of the Dark Souls game. However, when the boss become player controlled characters, there will probably not going to be a boss waiting in the designated area. The boss players will probably wander around on the map and kill other undead players as they want. Or on the contrary, the players that act as the boss will use their overpowered abilities to help other undead players to reach their final destination. Ultimately, this mod would probably turn into the boss players and the undead players dancing around together around the bonfire.

For players, is the most important thing of playing a game to defeat all other players and become the strongest one in the world? Or simply playing around with other players and enjoy the fun? When an individual possesses absolute power over the rest of the people, does this individual chose to use this power to help others, or to reinforce the supreme status? These are the questions that my mod wants the user to discuss and think about.

【Kinetic Interfaces】Midterm Project – Puppet


The project was initially designed to be a 3d puppet that’s built with  Unreal Engine. However, I encountered plenty difficulties during the process of creating a custom player pawn and noticed that there’s no way for me to finish this in time. As a result, the project switched back to Processing and the main mechanism of the puppet is using push and pop matrix as well as the translation from linear movement to rotation.


Unreal Progress:

To enable leap motion input, the user only needs to simply check the box in the plugins menu. After that, create a game mode BP and set the default player pawn to be leap motion related ones. Then under the world settings, change the game mode to the one game mode BP that’s just created. When hitting the play button, the hands will appear in the game.

one set back that I encountered was the plug in files are not displayed in the countet browser as default. Therefore, most of the production time was spet on trying to create that leap motion player controller pawn with skeleton mesh imbeded from skratch. 

It’s possible to add a cable component to the skeleton mesh in the BP therefore, it’s possible to create strings that tie fingers to the puppet.

Capstone Progress


Chatbot code structure. Robot hand, head printed model. Linkage design.

The hands and head are paid to print on Taobao which saved a lot of time and come with a better quality.

In progress:

Body under construction

To do:

Finish the construction of the body.

Refine the chatting system.

Make the physical linkage structure


The 3D printer (largest) is highly likely to make errors while printing. People that have 3D printing involved in their project should take that into concern.