Augmented Reality Storytelling by Tyler Rhorick

Reading Response

Blippar Intervention

Live Broadcast AR

For the Live Broadcast AR assignment, I was part of the team that tried to augment the IMA equipment room to tell the story of a student who was murdered by a cat for turning in equipment late.

Personal Reflection– Overall, the process of converting the space to tell our story with the green screen went pretty well. I would say that our biggest challenge in creating accurate scale, perspective, and lighting. As for the scale and perspective, we were able to achieve a believable enough positioning of the “victim” student after moving the camera angle and Diana multiple times, but the lighting was one thing we could never remedy. I think this means that for the future we should pay better attention to lighting conditions to give our final image a better overall effect. I think we could have figured it out if given more time, however, so I am not too sad walking away from this assignment.

Your Photogrammetry

Photogrammetry remained to be one of the most difficult assignments of the semester for me, for reasons I still cannot understand. What I was trying to do in this project was create a 3D model of the meowspace to further our story we created in the Live Broadcast, but this proved to be more difficult than anticipated because of the following challenges:

    • The real MeowSpace couldn’t be used– Because the meowspace was under modification when this project was assigned, my original plan failed. I was lucky to find a 3D model of meowspace in the lab that I ended up using, but this did cause some slight panic in the beginning.
  • Creating an accurate scan- The biggest problem with creating a photogrammetry model persisted to be difficulty in capturing images that could successfully be used by the program. I think I had a difficult time because the image I was trying to scan was pretty uniform in texture and lighting was hard to control against the surface of the structure.
  • Software- Another big problem that I had was in using the software. Even if the pictures were bad I could never figure out why the program never showed me a model after following the steps in the tech template. I had shown this problem to Christian, but we still couldn’t figure out what was happening.

Here is a folder containing all of the countless cat pictures I took trying to do this assignment.

Your Game Design Document

Here is our Game Design powerpoint.

Your Core Mechanic Documentation

Here is our Core Mechanic powerpoint.

Your MR interview with Storyboard and Scan

Because there was a misunderstanding when the groups were making their way into the green screen room, Matuez and me got split from our larger group where we had made a storyboard to play off the idea of the Sims. Because of this, we had to make a new model and storyline on the spot. To make the figure, we chose Adobe Fuse because it was quick and simple. We decided to make Vladimir Putin wearing makeup because of the recent ban on such imagery in Russia. As for the interview, it was decided that I would interview Matuez acting like Vladimir about such topics like Russia , the Ukraine, and his makeup.

Here you can see the video and Matuez’s perspective of the experience.

Immersive Sound

For the immersive sound project I decided to use Unity because my Max MSP trial gave out. To do this in unity I watched several tutorials on youtube. Basically the lesson of these tutorials was basically that you need to turn on 3D sound by changing the spacial blend. Using this technique I created a simple player and audio track of the NYU Shanghai alma matter. This player walks around the scene and the sounds get fainter as the player walks away. Here is a screen shot to see what I changed in the audio part to make it 3D.

Screen Shot 2017-05-24 at 9.49.18 PM

Final Project

Project Title: Shanghai Storylines

Partner: Mateuz

Elevator Brief: Shanghai Storylines is an Augmented Reality history experience that communicates the history of Shanghai’s Pudong area. Using their phone, the user can explore the Pudong area of Shanghai and learn more about the history that has often been untold of the area. The project imagines what Shanghai would have looked like from the NYU Shanghai academic building throughout history.

Extended Description: Shanghai Storylines is made technically possible using Unity and Vuforia. To start the experience the user walks up to an old Shanghai style window. Upon scanning the Vuforia marker, the user is introduced to the experience. The first view the user is made privy to is our imaged view of Shanghai from the NYU Shanghai Academic Building in the early 1900’s. The landscape was made in Unity.

Technology: Unity, Smart Phone, Vuforia

Development: Before we could start the project, we had to do a great deal of research concerning the history of Shanghai. To start this research, I met with Anna Greenspan, as a professor that has focused on the urbanization of China. She shared with me very interested texts talking about the historic foliage of Shanghai, which was research we used in the final project when we selected to make all of the trees broadleaf evergreen models, in accordance with Shanghai’s historic ecosystem. After this research was done, we needed a better idea of what was actually built in Pudong. Though we had the idea that it was just going to be fishing villages based on widespread “knowledge,” we still decided to research to make sure that was the correct narrative.

The first lead we got on the prior history of Shanghai came from finding a map on google images of the old Shanghai area. After researching more about this map, it showed that this map was one of very few of the area at the time and is widely considered one of the most reliable models of the area at the time.  Here is that map below:

Shanghai 1945

This map proved to be monumental in moving forward because it gave us the information that Pudong was not always called Pudong, but rather was formerly called Pootung. This information helped us find much more information about the area because this is what scholars have always referred to the area as. In searching for Pootung, we came across one book by Paul French called Old Shanghai. With a very detailed description of the Pootung area, we decided to dedicate our project to his research, which included a very colorful history of old Shanghai that included a foreign cemetery for those that died at sea, foreign occupation of a land controlled by the Chinese government, and animal warehouses that doubled in the nightly trade of prostitutes.

After we finished all of the research which took up most of our time in the first weeks to make sure that we were telling a compelling narrative, we began working on the technical side of the project. In hindsight, we should have started this part of the project way sooner because Matuez and me both had no experience in Unity nor 3D modeling.

Because of this we decided to split up the Unity work. I decided to work on getting the core mechanism of Vuforia working, while Matuez worked figuring out how to get the landscapes started. When it came to getting Vuforia working, we first decided that we wanted to markerless markers, but this proved to be more difficult than we anticipated, so we went back to using a marker. I also worked on getting the core mechanism of buttons and text boxes working so that we could communicate the story of Shanghai. While I was doing this Matuez was learning how to make terrains in Unity. He sent me a working model with the terrains started and then I watched the same tutorials to finish up the models. To modify what he gave me I decided to shape the terrain actually like that of Pudong. He had given me a square terrain, but I decided to be truer to the history we should try and get the right shape to the terrain. To do this I made a plane layer of maps from new and old and sculpted the landscape. Here is how that process looked.

Screen Shot 2017-05-16 at 1.03.41 AM


I also decided to add water to the scene, which I also attached a script to make the water move. In addition to this I flushed out some of the areas of the experience to give it a better sense of history like the docks, graveyard, and factory part. Here are some screen grabs of the finalized look of some of these areas.

Screen Shot 2017-05-16 at 3.46.47 AM

Screen Shot 2017-05-15 at 5.39.47 AM Screen Shot 2017-05-15 at 5.39.40 AM Screen Shot 2017-05-15 at 5.40.29 AM

All of the assets were made from other elements of the free asset store. For instance, I made docks out of extremely large and distorted pallets from a warehouse collection on unity.

In the end, our mechanisms definitely worked and I think we gave a great history of the region with the time we had. For the future, I would like to expand the historical content of the project and work to make the buttons and menus feel more integrated in the experience.

Here is a video of the mechanisms working:

Leave a Reply