RAPS – BirthDeath

BirthDeath

Partners: Maxwell, Vivian

Project Abstract

A live audiovisual performance in collaboration with Maxwell and Vivian. This performance explores the limits of the human body.

Project Description

BirthDeath is about the limits of the human body, and pushing those limits even further, until the body collapses. Maxwell and I created a realtime audiovisual performance following this concept.

With this performance, our purpose is simply to gradually provoke an overwhelming feeling in the audience and even increase their heart rate as the overall rate and feel of our piece speeds up. This performance is not necessarily meant to make you reflect on anything, it is just a sensorial experience.

Our inspirations to create this piece came from our interest in working with dance performance and the body. Maxwell and I have both danced and we really wanted to explore with implementing the body into our audiovisual performance. We originally wanted to use a heartbeat monitor and an accelerometer to modify some values in our Max patch, but since this was too complicated for the little time we had, we decided not to use it. However, we kept the dance performance.

Perspective & Context

Our performances fits into the historical context of visual music and abstract film in the sense that we really wanted to create a correlation between the sound and the visuals although not all of the audio factors were modifying the visuals. Our communication during the performance was essential.

I think that nowadays, because of our constant need to maximize time and productivity, we push our bodies with the last bits of energy we have everyday. We forget that our bodies have a limit and act as though we are invincible.

Development & Technical Implementation

From the beginning, Maxwell and I had a clear idea that they were going to work on the audio and I would work on the visuals. Maxwell created the audio patch alone, but in my case, I found that working on my patch while Maxwell played around with the audio served as a guide for me to create the visuals.

Part of the inspiration for this piece came from Maxwell’s and my interest in using sensors in the performance. Thus, we also wanted to implement Arduino in our piece so that we could use sensors such as a heartbeat monitor and an accelerometer. We did research on different types of heartbeat monitors but the only ones available to us were not reliable at all and were quite complicated to use. We considered using different buying a nicer sensor, but still, we did not know how to use it and we did not have enough time to make it work. Thus, we decided to only have the accelerometer. With Eric’s help, we got a patch that could send Arduino data to Max, so getting the accelerometer values was not too hard. However, attaching the accelerometer to Maxwell with a bluetooth Arduino did not seem to be very reliable either, so we officially decided to leave our idea of using sensors.

Instead, since I still wanted to show a correlation between the heartbeat and the visuals, I made the amplitude of Maxwell’s audio determine the size of a 3D model of a heart, and the redness of the screen in the beginning of the piece. This is essentially what I wanted to use the sensors for anyways, but this was definitely a much better and faster way of going about it.

As to the rest of the visual components, I originally meant to generate all of the visuals in Max. However, since I do not really know how to create graphics in Max, I decided to use screen recordings of sketches that I had previously created in Processing. In the patch, I switched between 4 videos, one of a red background, another of the 3D model of the heart, and the two other ones for the videos of my Processing sketches. I used functions such as rotation, zoom, scramble, multiplier, etc. When it came to modifying the visuals live, the MidiMix was fundamental to the success of the piece. I cannot imagine having the same results without it. It really made all the values easier to access and to alter.

Overall, we had two different patches, one for the audio, and one for the visuals. This means that we used two different laptops in the performance, and they interacted through our own improvisation and through the amplitude of the sound.

Performance

The performance was the first time we all ran through the piece. It went way better than what we expected. I was terrified because I was not sure if it was going to go well and I did not know when the performance ended, so much of it was improvising on the spot and trying to make the visuals fit the sound and the dance.

In terms of what could go better, Maxwell kept walking in and out of stage to work on the audio, which we realized was not a great idea because it did not add anything to the piece and in fact was a bit distracting. So we decided to cut the dance off of the performance and only focus on the audio and visuals.

Even though Maxwell and I made the greatest contribution to the project, Vivian was very helpful during the performance because if we wanted to keep the dance, someone had to control the sound. So Vivian took that role.

Maxwell and I had the opportunity to performance in Miki’s show at Extra Time Cafe & Lounge. Here is a picture of us during the performance.

Conclusion

Overall I am very happy with how this project turned out. At first it seemed a bit chaotic because we did not really know in what direction to go. But we ended up figuring it out. Working with Maxwell was great, they did an amazing job with the sound, which really helped me develop my part of the project. And Vivian was very helpful during the performance because she was able to control the audio while Maxwell was dancing. I would have not been able to control the sound and the visuals at the same time. I really enjoyed doing this project and hope to create more live audiovisual performances in the future.

Being able to performance at Extra Time Cafe & Lounge was an amazing experience.

RAPS Final // Konrad Krawczyk

GitHubhttps://github.com/krawc/RAPS/blob/master/visuals.amxd

Title: Untitled

Abstract:

If a musical texture is generated through an artificial intelligence, then who is the author? Can a neural network be more than a mere mechanically reproductive sampler? What does it mean for a musical experience to be “natural”? What does “natural” or “organic” mean anyway?

“All the modern things have always existed.” – Bjork

Project Description:

The project is an experiment with forms and tools in music and visual interpretation. The performance uses Google NSynth as a live musical instrument. NSynth is an ongoing project by the Magenta team, which specialises in artistic applications of machine learning. In short, it is a synthesiser based on a deep neural network fed with data of “natural” musical instruments, such as violin, bass or flute. This means that any texture produced by it is a function of the data that has been the prior input, as opposed to e.g. an oscillator, which produces pure almost sound waves. It is almost like using a sampler, but continuously. The resulting textures are often striking, unexpected, and unlike any real instruments – which is a direct result of how neural networks often make mistakes in modelling data accurately.

In this project, I used Google NSynth to produce ambient music, and use it further as an input to a visual generator in Jitter, within a Max for Live environment.

Background / context:

Abstract audiovisual storytelling are certainly not new, and they started appearing almost as soon as new technologies made them possible. Attempts of abstracting storytelling from material symbolism are often about “pushing the boundaries” of art and narrative storytelling. I think this boundary has been pushed to its absolute limits after absolute film was made out of black and white total frames. After that, abstract storytelling needs to be mindful of the fine line between openness for interpretation and mere reductiveness.

However, it is still a fact that minimalism and abstraction allows for more space for experimentation. I think that when experimenting with new technologies, it is better to start small and slowly increment instead of setting the stakes too high. However, there have to be some stakes, or at least a sense of intentionality behind the work. In the case of my project, it was to explore the intersection of organic and synthetic through an organic-synthetic instrument and visual effect.

Development:

I knew since the very beginning that I wanted to work with NSynth. I just didn’t know exactly what to expect, as all I hear about it was wither promotional videos or people experimenting with it. I saw the following video: https://www.youtube.com/watch?time_continue=8&v=QKCRANEsWB8 and after that I knew exactly what I wanted to achieve. NSynth was just extremely good in ambient textures. However, it seems like I needed a physical interface to begin with. I looked up all the components, and ordered some of them.

Eventually, it turned out that the instrument is available in a technically equivalent version for Max for Live. I decided to consider it a back-up plan, as I really wanted to get the physical interface up and running. I started with trying to get the image onto a raspberry Pi. That did not work either on RPi 2 our RPi 3. So, it’s either that none of them were working RPis (unlikely) or simply I was doing something wrong. This turned out several days before the performance was supposed to be ready, so I eventually gave up on the idea and decided to use a MIDI mixer instead.

Meanwhile, I ran NSynth in Ableton and started to explore different textures and sounds. I got really quite amazed by the variety of sounds. Some “instruments” sounded pretty realistic and orchestral, other sounded like ambient synthesiser, and others sounded like faint echoes of natural sounds, or church interiors. This was really satisfying. I decided to introduce a simple rhythm to make the music a bit more progressive and interesting. The two rhythms would closely resemble a heartbeat, and one of them was simply a richer version of a pulse.

I also started to experiment with Jitter. I followed this tutorial: https://www.youtube.com/watch?v=yqI39xxTiDs to create simple, geometric shapes that would change with different frequencies (low, mid and high). Instead of using a funky torus, I used a symmetrical cube. There is an interesting feature of the renderer I discovered, namely the mesh mode, which allows to render the 3D shapes using transparent meshes, points or full closed shapes.

The final setup uses six switches. Three make the installation responsive to the incoming sound from Live (low, medium and high frequencies), one introduces a slow rotation, another one introduces fast rotation and mesh mode change, and the last one draws a plane – white background behind the 3D shape.

Performance:

I spent a late night before the performance trying to feed the Jitter data into a Vizzable plugin, which was supposed to work with Max. The goal was to make blurs and glitches to make the visuals look more detached and historical. That didn’t work, as the plugin was meant to work only with a camera input and a chain of other plugins of its own type. The Jitter input just didn’t get detected.

I gave up pretty late – an hour before the performance, in fact. I decided to work with what I had instead. The visuals slowly progressed, as I introduced new elements, chords and textures to the Ableton loop. I determined the loops and chords way before the performance, but the textures themselves were a matter of improvisation – I only knew how the sounds would start, but not how they would end up.  I was honestly quite lucky to have worked with NSynth, because the random changes in texture I made sounded quite pleasant.

I made a last minute decision to also change the cube into a sphere IN THE CODE halfway through the performance. I’ll never dare to do that again, but it worked apparently.

Conclusion:

The overall goal – generating ambient sound and simple, reactive visuals in Jitter – has been accomplished. Not having the physical interface did not make any difference at all, and in fact has made it easier as the MIDI controller was integrated seamlessly with Ableton. I am highly satisfied with the sound of NSynth. Of course, at this stage it does need a bit of extra editing in a convolutional reverb (to make it a bit more “ambient”) but the sheer amount of textural possibilities makes it an extremely good tool for improvisation.

I do wish I figured out how to create more complex matrix transformations in Jitter ahead of time. Perhaps if I started this stage the week before performing I would have gotten there. Jitter is much more complicated than Vizzie as it’s basically a programming environment similar to Processing – but I think it is more performant and versatile than Vizzie.

I would like to continue experimenting with both this sound and (perhaps more visually appealing) visuals in the future.

RAPS Final Blog Post

Title
Duality

Project Abstract

In our project, we present the duality of Shanghai, which refers to the modern and the shabby side. Furthermore, we can never judge a city unilaterally so we start with showing separate pieces of videos with corresponding audios alternately, moving to the mixture of the two sides gradually, to demonstrate the duality of such a super city.

Project Description

Our project is basically about a process of a better understanding of Shanghai. Speaking of the beginning and ending, actually we used the same video about an approaching man with an eventual focus on his eyes as the starting point. Nevertheless, when it comes to the ending, the video is expected to be played backwards, which means that there are a thousand Shanghai in a thousand people’s eyes.

Except the beginning and ending, the body of our project can be divided into three parts. First, the two sides of duality are shown separately in turns, when the duration is narrowing from 20 seconds to 5 seconds. In this way, we want to create a kind of confusing feeling between two sides step by step. Second, we selected every three videos as a group to mix them together with the emphasis on the old ones as the purpose was to focus on the hidden side of Shanghai. Third, the videos emerged on the screen one by one and finally occupied the whole picture, showing that there are quite distinct parts of the same city from different angles.

Perspective and Context
Compared with historical context of visual music and abstract film, our project also demonstrated a corresponding relationship between audios and videos. Through the coherence of various materials with a main concept, our project aims at conveying more meaningful things more than just the color effects and extreme abstract images.

In terms of contemporary practices of live cinema and live audiovisuals, the first thing occurring to our mind is mostly the byproducts such as the background of concerts and clubs. However, I think the meanings of audiovisuals are far more than entertainment as discussed in The Audiovisual Breakthrough, instead these art works can inspire the audience to think beyond the surface of performance itself.

Development & Technical Implementation
At the very beginning, we wanted to get inspiration through seeking for various kinds of audiovisual performance videos. What impressed us most was a video that demonstrated the city views from morning to night and all of the frames were inside the eyes of a man. The visual effect really stimulated us that we decided to focus on the versatility of Shanghai through eyes. As shown in our final performance, we used a shot of one man moving from far to near with a ending freeze-frame of his eyes as our beginning and ending. The man can be interpreted as anyone of the audience who would rediscover Shanghai through our performance.

In terms of visual part, on one hand, we shot some videos in old residential areas including street stalls and pedestrian along the roads and clothes hangers outside windows. When shooting them, we retain some floats and shakes in videos to strengthen the feeling of old neighborhoods. On the other hand, we shot some traffic flows around AB by time-lapse photography to show the intense and modern side of Shanghai. Also, we found some videos from website including aerial images, but we were very sorry to realize eventually that they were not in public domain and we couldn’t use them without a license.

In terms of audio part, we downloaded music from music sites whose rhythms and emotions are corresponding to our videos. But in the same way, we didn’t pay attention to their license and used them directly in our project, which was really a serious mistake in our project. Thanks again for Eric for pointing out our misunderstanding of the license! We are sure to pay more attention to this point when making the art project in the future.

The biggest problem we met with was how to handle too many videos at the same time to avoid the unexpected crash of MAX. With the suggestion of Eric that we don’t need so many AV Players which lowered the speed of the running of MAX, we edited some videos as a whole and omitted the unnecessary to make our setup more concisely and easier to operate.

In the whole process, as a group, we divide the works into three parts and everyone was mainly responsible for one part. Generally, In general, Krystal took the audio part, Selina mainly focused on MAX and I was in charge of shooting and finding visuals. After the preparatory phase, we all concentrated on MAX. Krystal added proper effects to audios while Selina and I were devoted to the setup of visual part. In the final performance, Selina and I were responsible for the MIDI while Krystal took control of the music in another computer.

Here is the link to our code in Github: https://gist.github.com/EEElisa/1f29c1603a534027b5e7cdb9de8f347f

Videos:

https://www.youtube.com/watch?v=Y4lWRiwNwRc

https://www.youtube.com/watch?v=q3F3nlPkAlE&t=115s

Music are downloaded from KUWO music app.

Performance

Generally speaking, our performance wasn’t up to our expectation. The biggest problem is lack of practice, which results mainly from the fact that we spent a lot of time fixing the unexpected problems of patch the night before the final performance till midnight. Therefore, we didn’t have enough time to practice our setup so that the transitions between each part and the match of video and audio are not that smooth.

Also, for lack of practice and test, we were not so confident to present our project very well with intension. Therefore, the performance seemed to be a little hurry-scurry. For me, I felt regretful that I even failed to turn the speed to negative one to play the video backwards.

Despite these disadvantages, I preferred the mixture part, which provoked the further thought of audience to consider about the influence of rapid development of Shanghai. As for me, I really appreciated the chance to perform our own project in the auditorium as an audiovisual performance player. When I was controlling the system and turning the knobs, I felt like that I was absorbed in the world built up by ourselves. If there is another chance, we will spare no efforts to make up for the imperfect to present a better performance.

Conclusion
In a word, I learned a lot from out project and RPAS. Looking back the whole process from beginning to ending, what I appreciated was far more than many of the terms in Max but the experience of creating an audiovisual performance system by ourselves, which I can’t imagine at all before. Also, I had a deeper understanding of IMA, which inspired me to further discover other courses in the future.

What’s more, full preparation and consciousness of copyright license really matter. Only with a detailed time table and reasonable arrangement can the process runs well as expected. Also, the consciousness of copyright license is essential in every field and every subject, which shows the respect for others and deserves more attention in the future. At last, thanks a lot for Eric and all of our classmates who presented us a really cool show!

RAPS Final Duplication

Title:

Duplication

Project Abstract:

Our project was meant to create a sense of horror and suspension of the concept of the parallel universe using a narrative tone. The films were recorded on phone and the audio&visual mixing was operated through max.

Project Description:

In the beginning, the brainstorming stage, we were planning to film a horror film with a storyline behind it. However, after actually recording some early footages of the film, we found out that it is unfavorable to create a project like that for this course. This is RAPS, not a drama class. Our original concept relied too much on the narrative itself which requires acting and filming skills rather than max applications.

 

Acknowledging this issue, we decided to shift the focus of the project. The story was about the character getting frightened by a duplication he saw in a daily setting. We shifted the theme of pure horror to the concept of the parallel universe. By doing so, it also eliminated the boundaries of a narrative story where we can implement our imagination and understandings of the parallel universe.

 

The scene where the screen was split into three equal portions and each portion was looping the same video with different beginnings and rates. The purpose was to show that parallel universes might be multiple timelines flowing at the same time. The timelines might lead us to different branches and outcomes but for some parts it’s immutable. In the end, we mixed a running and walking version of the character to express the unlimited possibilities among the infinite amount of parallel universes. In this universe, I might be the prey but in the other one, I’m the catcher. This uncertainty breaks up the limitations of our imagination.

Perspective and Context:

On page 83 of The Audiovisual Breakthrough, the author quoted Makela that live cinema is exempt from cinema’s chief constraints, namely narrative continuity and a fixed spatial arrangement. This was also the trap that we stepped in at the beginning. Live cinema should focus on the aesthetics produced from spatial movement. During the process of producing the piece, the artist should use experimental technology on a chosen theme. To us, although max doesn’t count as “cunning edge” technology, it is a new experiment to us since it is a new medium that we just started to test upon.

 

However, we did not want to abandon all narrative traits. There is still an abstract storyline flowing through our performance. Just as Chris Allen describes their work as a “deconstructed, exploded kind of filmmaking that involves narrative and storytelling”, our work starts from the character being haunted by his own duplication till the end where the “meet” each other. Mixing some pieces of logical elements into the piece gives the audience a basic guideline to build upon their understanding of the piece.

Development & Technical Implementation:

The development process could be could be separated into two main categories: film recording and max. For the film recording process, we experimented multiple times to get the best angle we could to create the effects that we wanted. For example, the first scene we had to find the angle showing the turning point of the hallway to show that the character encountered something unpleasant. The last scene we had to squeeze at the corner of the room to get a full view of the walkway. My contribution to this part is being the actor of the film

For the max part, we started by creating the audio part and the video part in one patch. Later on, since we had too much video players in one patch, the workload on the computer became too high which made it hard to operate with. Considering this outcome, we decided to sacrifice some automatic synchronizations of the audio and video for fluency of both parts. My contribution to this part is the video mixing part on max. The link provided below is the link of my video part:

https://gist.github.com/hjn1999/9d580d14e43d80987ab53921b3b010d2

For the audio part we used the following audio files:

The following mp3 files are all licensed under the creative commons Attribution License

Knight riders, https://freesound.org/people/Link-Boy/sounds/420544/

Horror Soundscape, https://freesound.org/people/Headphaze/sounds/170653/

Atmosphere- Horror (loop), https://freesound.org/people/julius_galla/sounds/206176/

Performance:

For the performance, there were moments that stood out but there are also flaws. Since we separated the control of video and audio, there are times that we didn’t perfectly synchronize our actions which highly corrupted the audience’s enjoyment of the piece.

 

However, the creation of digital and disturbing sound with some improvisation did create the effect of suspension and depressing effects that we expected. Overall, even though it wasn’t perfect, we are satisfied with what we presented to the class.

Conclusion:

Throughout the process of brainstorming, developing, testing, reediting, not only did I learn the explored the field of live cinema and VJ culture which I’ve never touched before, but I also learned basic daily skills such as communicating with my teammates of the effects we want and dividing work upon ourselves.

 

There are some things that we can improve. First, if our performance is mainly based on film rather than the generation of spatial elements, we could’ve spent more energy on capturing the footage. The videos could’ve been taken somewhere much more meaningful or cooler than a setting where we are all familiar with: school. That would’ve brought our performance to a next level. If there is a chance I would like to experiment such methods in the future for a more complete performance.

 

Thank you, Eric, for teaching us this semester. The ideas and skills weren’t easy to grasp but it did open up my vision. Seeing art from a perspective that I have never seen before. Merry Christmas, Happy New Year, and enjoy your winter break :).

RAPS FINAL-Duality

TITLE:Duality

PROJECT ABSTRACT

The purpose of our final project is to project the duality of Shanghai, a city where full of white-collars working in hi-tech or financial centers and also the relatively poor people living in alleys.

PROJECT DESCRIPTION

What our project tries to convey is the seeming conflict and contradiction within our city. However, this contradictory city is where we live. We witness the conflict but we have no means to change that. At the very beginning of our performance, we shoot a man and focus on his eyes as the representative of our perspectives. What follows that is what we actually see in our daily lives.

First, we make sharp transitions between the modern part of our lives and what we often see when in old communities. What inspired us to make these kinds of transitions is that sometimes when we look up to the sky, we can see a great number of skyscrapers and our prosperous future, but when we cast our eyes to ordinary street, especially those in Puxi, what we see is actually a sharp transition between these two. However, they indeed coexist.

The next part of our performance shows how they coexist in this society. What kind of make me do these kinds of overlapping effects is a post I once saw on WeChat, saying that there are great bunch of people in Shanghai who use all of their salaries to buy luxurious bags and clothes to wear to office, squeezing with people in metro on their way going home, which is in those old communities. Even within one’s life, there are full of these contradictions.

PERSPECTIVES AND CONTEXT

How our project links with the notion of visual music is that the interaction between images and audio. Since visual music is derived from the idea of synesthesia, the idea of having images when hearing a song or hearing sounds when seeing images. When the modern part of life was shown on the screen, we chose “songs linking with prosperity”; when the lives of the most vulnerable was shown, we chose “songs representing sadness and oldness”.

As mentioned in the book The Audiovisual Breakthrough, one crucial element in live cinema is linear storytelling, particularly those “based on actors or verbal dialogues (Cavalho 81).” Our project in particular, make all ordinary people who were shot actors, to portray a drama of ordinary life. In addition, we also make these kinds of images with a view to evoking sort of complicated feeling of contradiction from audiences’ minds, getting them think more about something in our lives subtle but also crucial.

DEVELOPMENT AND TECHNICAL IMPLEMENTATION

To develop our idea on the duality of Shanghai, we thought of how to bring up the actual contradiction in lives and how can we best address their means to coexist in our current society.

We first took footages by ourselves, trying to shot things that are indeed around us that we can actually feel the reality. To convey the contradiction and sharp transitions, we classify our footages and make them fall into two main categories of old and modern, and make two different effects on them. For the modern part, we choose to have the effect created by faders, to create a sense of high density and rapid speed of lives. For the old part, we make up an analogy effect of old television. To make sharply changes, we switch between these two effects by the control of 4mixer by hands.

In the second part of our project, we mixed three videos together by focusing on the old part of Shanghai, having other two videos of transportation overlapping it, to create a sense of that the rapid development is trampling on top of the ordinary people, since some of them indeed sacrificed for the development and urban renewal. In the last part of our performance, we made all these “lives” pop up one after another in order by controlling the size of these video through knobs randomly to create a sense that they coexist and we see them together. Sometimes we may only see the bright side and neglect the lives of ordinary people (when the modern parts zoomed in), but when we get in touch with them, we can feel them greatly and may sometimes feel compassion in mind (when the old parts zoomed in).

Git Hub: https://gist.github.com/EEElisa/1f29c1603a534027b5e7cdb9de8f347f

References:

VIDEOS:

My Destiny. “Top 10 Shanghai Shopping Malls! Stunning Architecture & Shopping Vibes – China.” YouTube, YouTube, 7 May 2018, www.youtube.com/watch?v=q3F3nlPkAlE&t=115s.

Zijiang, Kim . “航拍上海 Shanghai,China.” YouTube, YouTube, 5 Dec. 2018, www.youtube.com/watch?v=Y4lWRiwNwRc.

AUDIOS:

from KUWO Music

SETTING:

Video                                                                           Audio

      

PERFORMANCE

Our way to final presentation is kind of tough actually. First and foremost, as a group, we divided our task into three parts: video footages shooting, audio making, and the Max mixing. I was in charge of the last part. We made up a schedule at first, however, we did not meet our goals, which makes the time for setting up really short. In addition, since we did our work too separately, we sometimes lost connection with each other, which make the whole process even tougher. Nevertheless, we made it at the last minutes.

On account of our lack of practices, things didn’t go that well on stage. On stage, me and Elisa controlled the mixing of videos and Krystal controlled the audio part. Since our setting of video and audio were on different computers, we only focus on our screen but neglect that we need to sync audio with video, which make our first part kind of disconnected. However, in spite of that, I think the video part alone looked great! When hearing the music on the stage, we actually played with these effects in our interests and had great fun! Indeed, the effects we made exceeded what we thought of when practicing!

CONCLUSION

To conclude, this final project for me is a great turning point. In terms of creation, I think we exceed the ordinary field of the original kinds of audiovisual performances that we watched throughout the course (I’m not saying these performances are bad) by linking what we learned in this semester in RAPS and other notions we learned in other courses. In terms of research and material collecting, I learned how to better convey our ideas by collecting videos shot by ourselves and based on our thoughts.

What I think I can improve is that when doing group work, I should focus on our group more, to focus on cooperation more, instead of focusing only on my task. Since the wrong concentration may delay the whole process and make group member disconnected with each other. To wrap up, in my opinion, what I succeed in doing is the better way to convey my idea through performances, trying to make myself understood through images. What I can improve is to cooperate and practice more!

RAPS Final Project

Title
UTOPIA|CHAOS|DISTORTION

Project Abstract
This is an audiovisual live performance piece which discusses people’s behavior in a highly organised society.

Project Description
This is a realtime audiovisual performance piece which discussed people’s behaviors in a highly organised society. We are questioning what would happen when people begin to overthrow this highly organised society, and what would happen when people fail and everything goes back to the normal state.
In the beginning of the piece, people behave normally. We used different perspectives and scenes of people walking around to show people’s life in a highly organised society. Gradually, everything enters a chaotic state, with people’s wish to overthrow the current society. People’s facial expressions go crazy, while the visual composition of also goes crazy. After the chaos, everything goes back to normal. People’s wish failed, and everything become distorted.

Perspective and Context
The form of our performance was inspired by the chapter of Granular Synthesis. Miki and I really like the /Modell 5 – Granular Synthesis/. So we made a lot of close shots of people.

In Miki’s post, she mentioned a lot about the message. I will add something about the content of video. The concept of walk and stand was inspired by some post modern performance art. We watched John Cage in class, which gave me a lot of inspirations. He walked around the stage and made some meaningless actions to make sound. In other post modern performance pieces, we can also see a lot of walk. I am curious about what walking means to people. Does it have a specific meaning? Or it’s just a normal daily action.

Development & Technical Implementation

Storyboard:
We brainstormed some key words and then made a list of content to shoot based on these keywords. We also created several storyboards for shooting.

Video Shooting:
We spent two days to finish our video shooting. The footages include facial expressions, legs with two shoes, running, throw shoes, walking, and a time-lapse video. We also recorded audio in different places.

Footage selection:
We selected some footages and categorised them into three folders: utopia, chaos and distortion. We also found some audio and video footages online.

Version Iterations:
For the first version, we created a simple patcher based on our footages and played with these footages and some color effects. We are not satisfied with the effects. After talking to Eric, for the second version, we played with some crazy Vizzie effects, including time effects, color effects and composition effects. And we also found better audios. We don’t want over-effects in the whole performance, so for the third version, we tried audio and effects with all of the footages, and decided only to use them for the second and third categories.

After figuring out transitions, we created presentation mode and assigned input signals to our MIDI controller. We distributed tasks and practiced together.

In the whole development process, Miki and I  worked together.

Code: https://gist.github.com/mikibinxue/d6e25701b71dd2e8c82fc22ceba7b2f8

Online Video Source:

https://videos.pexels.com/videos/person-walking-on-pedestrian-lane-857593

https://videos.pexels.com/videos/walking-on-the-street-857592

https://videos.pexels.com/videos/people-walking-857589

https://videos.pexels.com/videos/white-footwear-857834

https://videos.pexels.com/videos/white-sneakers-857849

https://videos.pexels.com/videos/black-and-white-sneaker-857581

https://videos.pexels.com/videos/person-walking-with-support-857812

https://videos.pexels.com/videos/man-crossing-the-street-857584

https://videos.pexels.com/videos/person-walking-on-pedestrian-lane-857593

https://videos.pexels.com/videos/busy-people-crossing-the-street-857586

https://videos.pexels.com/videos/black-and-white-video-of-people-853889

https://videos.pexels.com/videos/man-step-on-a-puddle-of-water-854031

Online Audio Source

Title: Kiasmos, Artist: Franq
http://freemusicarchive.org/music/Franq/nx2016-01/nx2016-01_02

Title: Binary Dawn, Artist: Filmy Ghost + Peter Edwards Ambient
http://freemusicarchive.org/music/Filmy_Ghost__Peter_Edwards_Ambient/

Title: Fuite En Tête, Artist: Ayato & Kecap Tuyul
http://freemusicarchive.org/music/Ayato__Kecap_Tuyul/When_we_talk_about_love_and_happiness/05_fuiteEnTte

https://freesound.org/people/2hear/sounds/317533/

https://freesound.org/people/tec_studio/sounds/98491/

https://freesound.org/people/Robinhood76/sounds/318900/

https://freesound.org/people/bsumusictech/sounds/62333/

https://freesound.org/people/klangfabrik/sounds/160001/

storyboard

video folders

max patch layout

presentation mode

Performance
The performance was good. I was responsible controlling audios while Miki controlled visuals. We were pretty nervous, but it didn’t hurt the performance. In the opposite, the performance was better than our rehearsals. One thing that could be stronger is how we distributed time to each sections. We could rearrange the time and give some sections more time.

Conclusion
This project was a good experience. I learned a lot through the whole process, from brainstorming to final performance. I realized it’s always good to start with a strong concept. And all of the video and audios should be guided by the concept. Besides, collaboration is also important. Teamwork makes production and performance much easier and more efficient. For performance, it’s always important to have rehearsals and practices. People can’t find the errors unless they do rehearsals and practices.

For our project, all of the sources are pre-recorded. It would be great if we can add some live elements next time, like live sound or body movements.

 

Tyler Roman – RAPS – Live Cinema- Reflection

It seems from this reading as well as last weeks and the documentary the difference between the categories of VJing, Live Cinema, and Live Audiovisual Performances are as follows: VJing is the mixing and crafting of video often times to match and auditory experience such as a DJ, this sometimes evokes itself as becoming a “visual wallpaper”, this differs from Live Cinema as Live Cinema endeavors to be a complete experience in and of itself, it is less about matching a separate party as it is creating a self-contained world often times held to a higher standard of cinematography and dramaturgy, and both of these somewhat exist within and yet differ from the Live Audiovisual Performance which more or less exists as a catch-all term for the new world of audiovisual performances as they have grown, evolved, and diversified from existing as “mere” VJing, VJing and Live Cinema may fit within the category, but also everything else that seeks to find a balance between what is audio and visual in a “live” context exists within this category.

Looking examples I found three: “Vanishing Lines” for VJing, “On the Other Side” for Live Cinema, and “Octovalent” for the Live Audiovisual Performance. For “Octovalent” it did not look like the video was clear-cut attached to the audio, though they often worked in tandem, they were strongly synced to exact notes maybe. There seemed to be multiple parts with individuals on different parts working together as a collective though at the times the meshing seemed to be a touch off. With “Vanishing Lines” there is a pretty polished performance where it seems as if the visuals have been handmade for the music, the movements and transitions seem to match the mood beat by beat at least until around 2:40 all the while the VJer controls the visuals as they stare intently at them. Finally, for “On the Other Side” there seems to be an event bigger disconnect from the sound as there an immense priority placed on the visuals and the cinematography, as smoke is used to enhance the visual experience as the sound helps to set and amplify the mood as well tell a story, the sound is not nearly as reactive to the video as it exists more as a singular experience.

Links to performances:

On the Other Side – https://vimeo.com/23097637

Vanishing Lines- https://vimeo.com/176196153

Octovalent- https://vimeo.com/286253580

Tyler Roman – RAPS VJ Culture Reflection

Interestingly enough its hard to remember where I might have seen VJing outside this class. It’s one of those things for me that I know I must have seen before, but cannot quite place when or where. And I think that it is this exact thing that The Audiovisual Breakthrough talks about when it refers to “visual wallpaper” in the past my experiences with VJing must have largely been experiences where I did not quite know what I was looking at, it was not the focus, perhaps the music or some other aspect of it was and in that sense VJing carried the tone of “wallpaper” obvious to the knowledgable or attentive observer and meticulously chosen, crafted, and designed by the owner/performer and yet to the masses it is taken for granted.

In the past I might have see visually responsive works at concerts or clubs, the video would sometimes change and shift to match the music or perhaps the mood of the music. But I do not believe that I ever singled it out as its own thing, I focused on the music and may have wondered which genius was able to match the aesthetics of the musical piece so well, but I would never have imagined that there would be an individual, a performer, and craftsman behind the work I saw. Rather than seeing through this guise of a “wallpaper” I think I stopped short and for that and many other reasons I truly believe that this class has opened my eyes. Where I once might have seen visuals that seemed to match the music and granted the merit to the sound for finding its match, now, I might see a crafted performance, designed and intentional, as it works to add another universe to the experiences of its audience

Tyler Roman- RAPS Final “Shanghai Dissonance”

Title

Shanghai Dissonance

Project Abstract

Shanghai is a city of lights and sounds, there is beauty, but also chaos to be found within. Experience the dissonance as movement and sound collide.

Project Description

The idea behind Shanghai Dissonance is to make music from some of the ambient dissonant sounds of the city to create music that in turn creates a visually reactive video. The concept stems from my own experiences in Shanghai hearing these sounds and beginning with the question if it was possible to make somewhat passable music while utilizing the skills I have learned this semester in both this class as well as my Music Structure class. Having watched videos online about how to make and compose music and learning about scales, harmonies, melodies, perfect authentic cadences, motives, etc. I wanted to combine this idea born of curiosity, passion, and new found knowledge within the context of a audio-visual performance inspired by the idea of “My Shanghai” that seems to embody my projects this semester.

In “Shanghai Dissonance” one experiences three different videos of movement and sound within Shanghai, particularly ones that correlate to my day by day; the metro arriving, the bike ride home, and the city skyline that passes by in the journey in between. The videos play, stop, buffer, loop, and start in relation to the pressing of button on the Akai MidiMix. By taking the sound of construction hits and car horns sounds, normally dissonant sounds that mark the Shanghai Experience and turning them into musical components a melody is played with accompanying drums and bass line, and in turn, the button presses change and shift the the video being displayed into several different effects and combinations.

Perspective and Context

“Shanghai Dissonance” definitely fits more into the modern side on the front of visual music and abstract film. The cityscape is modern and I think the commonality of the video is representative of the more modern idea to capture the realistic and mundane rather than works such as the works of lumia and the clavilux or the ideas of cosmic consciousness or the over-developedness of early animation that all sought to explore the extraordinary. I do not believe that my piece is one that progresses the medium in many ways and rather that it is the point, this is my experimentation into the world of Audio-Visual performances and live audiovisuals and so it serves more as a marker of what is possible in todays day and age, a marker for the progress of technology and my own development within the field. In some ways the style is somewhat similar to the stuttering effects of works such as “Granular Synthesis- FORM” as the layering and buffering of the videos lend themselves to a similar style.

The piece in a sense describes a journey, at least visually, in the sense that all the videos display motion and travel through space during different legs of the day to day experience, whether that is enough to fit into the realm of live cinema at the level it was developed is hard to say, but in the future if such a thing could be better developed I would personally want to endeavor to such an idea.

Development & Technical Implementation

The melody was created as by changing the pitch of the car horns (the sound for both the car horns and the construction hits take from freesounds.org which I will add the links to below) using Audacity I was able to create notes that follow a pentatonic format, roughly following the Chinese Pentatonic format of the Shang, Jue, and Yu scales (with one note modulations in the first two scales) for the main melody as well as an accompanying drum progression. The notes were also manipulated to make a sort of “walking bass” based the G,E,D,F keys as well as various sharp variations that serve as leading notes. The initial inspiration behind both the walking bass (which actually took forever to figure out as I am not trained in piano so figuring out the key notes and names and then order took a more than a few listens) and the notes come a youtube video on simple jazz improvisations (link below,) but car horns do not lend themselves well to matching with piano notes so some of the notes were transposed down in order to better fit i.e. not sound ear-piercingly high or inaudibly low. The walking bass as well is supposed to well, walk, and that means its supposed to somewhat live and breathe along with the notes played, matching chords and whatnot, but in my piece it turned more into a steady repetitive backdrop in which to layer the melody on top of, not modular in most senses of the word.

The setup works as (with the help of Professor Eric) I was able to figure out how to use bangs as “momentary switches” instead of toggles when activated through the midi mapping feature of Max 8, they in turn send a signal on the press of the mapped buttons playing there respective note on one of three different scales D-FGA-C, E-GAB-D, A-CDE-G. In the original iteration, as I utilized one of the coding resources provided to us by Professor Eric (which will be linked below) the midi toggles were mostly hooked up to the visual effects of the three videos, however as I wanted to control the video more through the playing of the piece rather than toggles I mapped the buttons to also control the start and stop of the avplayers, as well as connecting wandrs to the loop end, and buffer in order to create more random and erratic visual patterns, controlled random chaos i.e. dissonance if you will, and then adding wandrs to those wandrs to make it even more so) at this point this would be the performance iteration, later due to utilizing IMA laptops which I had to return I actually lost the performance patch iteration as I seemed to have uploaded the wrong folder to my google drive, thus I recreated the patch from scratch and updated it, this time adding wandrs that would also change the start location of the videos so that way there was an increase of variety that I noticed was lacking during my performance. As well as this, I changed the music playlists to be a three and then one list instead of 15 separate files as unfortunately the car horns did not really create effective or audibly pleasurable chords when played together, instead sounding rather messy, I find this updated version to be a lot better on both counts of visuals and sounds and thus have recorded a shorter performance to display the new system.

 

Link to Github Improved Iteration: 

https://gist.github.com/TylerRoman/07f947dbd908a8e3d7094df302300b3e

Links to sound samples:

Drum (Construction sounds)-

https://freesound.org/people/Mujyo/sounds/384075/

Melody (Car Horns)-

https://freesound.org/people/fkunze/sounds/203167/

Link to Jazz inspirational video: (I am not sure if this falls under the idea of free use, but I used this video mainly for inspiration and the actual components came out vastly different and nothing was directly sampled so no issue I think (?), but still credit given where credit is due)

Link to updated performance on google drive: (I think this works)

https://drive.google.com/file/d/1QFgoqwmvEAEg9ZSDWlb9C_rH_gcWptLt/view?usp=sharing

Performance

The performance went…alright. Most likely due to a lack of practice, but also potentially the choice of instrument the sound design does not really fit into the realm of amazingly aesthetically enjoyable. Even though the notes themselves have some sort of musical structure and theory to them, putting into practice is a different story. I tried to embody some of the advice I received last time working with Konrad to show some of the plain, unedited footage, but receiving the feedback that perhaps I could have show even more and that it was a lot of what seemed like the exact same makes a lot of sense.

In regards to what went well, I think the system work as instead minus some errors in the sound design and though it could use some definite improvements it was a neat experience to perform an entire performance and have it go somewhat according to plan. In terms of what could have gone better, as I’ve mentioned, practice, sound design, more variations in video choices I think there are definitely things I mistook as “aesthetically enough” that instead came off as “lacking” the sound design could use more work and though putting all the wavs into a singular playlist helped with that I think the idea of car horns in general could be scrapped in the pursuit of more “enjoyable” music. As for the unexpected, nothing went too out of hand, the piece is more about dissonance and controlled chaos so it went rather smoothly on that front.

Conclusion

In conclusion, I learned a bit about composition and audio editing, getting familiar with the midimix and midi-mapping in and of itself was a treat. I discovered that yes, one can indeed make music out of car horns and construction sounds, but whether or not that is a good idea remains largely to be seen and heard, perhaps from someone else or in a more nuanced context.  I failed to make something that really makes a statement as some others did, and while my piece is indeed modular, and fresh each time, it could stand to use from some more practiced structure and preparation.

I have mentioned a lot that it could be improved in the area of sound design and if I could have the full range of musicality as expressed in the jazz video it would be cool to be able to have a somewhat improvised, nice sounding jazz variation of the performance, especially if the scales were note for note matching with the Chinese scales they are based on. In terms of video, having the piece tell more of a story would be better, perhaps by staggering the videos in the chronological story in which they appear, and on that front, even adding two or three or four or plus videos would help a lot in making the piece seem a lot more full and developed than it comes off as. All of these possible improvements would be great things to accomplish in the future, and perhaps in a different way and iteration “Shanghai Dissonance” will come about in a more practiced, fluid, and meaningful manner.

RAPS -final – documentation – UPTOPIA | CHAOS | DISTORTION

Title
UTOPIA | CHAOS | DISTORTION

 

Project Abstract

Suppressed by an ideal world, through a series of effort to break free from the rigid boundaries, they finally consent and returned to the world with a distorted mind.

Project Description

Our project was inspired by the process of people being suppressed by an ideal world, through a series of effort to break free from the rigid boundaries, they finally consent and returned to the world with a distorted mind.

It magnifies people’s relationship to the world, and through an abstract interplay between music and visual, it encourages people to introspect their own relationship to their surroundings.

Perspective and Context

As a group, our project was really inspired by the chapter of Granular Synthesis. For the style where an original video was so heavily effected that people could not tell its original form, we referred to the video, /Granular-Synthesis – FORM/. The use of close shots of people was inspired by /Modell 5 – Granular Synthesis/.

Our message delivery was inspired by the reading, The Audiovisual Breakthrough (Carvahlo & Lund, 2015). It talked about the concept of live cinema, which encourages collaboration with musicians to form AV-groups (Rechenzentrum, Telcosystems, Pink Twins, etc.), marking the fact that the visual was not just for a wall-paper like for most VJs in a club setting, but needs to be properly appreciated with full attention in a proper, cinema space. We also wanted our piece to be something more than wallpaper, but we want to tell an abstract story through the sequencing of video effects, and how they were effected.

For example, since our project wants to portray the emotional status and transition of people living in today’s highly ordered world, we shoot different original video footages in iconic places in Shanghai, including Lujiazui, the financial area, the park, and abstract video footages in the dance studio. By time-lapsing the crowd flow on the highway, we created a sense of fast-paced life, and how it emerged in the close shot of the legs were trying to magnify our presence against the society.

Development & Technical Implementation

GitHub link to project:

https://gist.github.com/mikibinxue/d6e25701b71dd2e8c82fc22ceba7b2f8

Song:

Title: Kiasmos

Artist: Franq

http://freemusicarchive.org/music/Franq/nx2016-01/nx2016-01_02

Title: Binary Dawn

Artist: Filmy Ghost + Peter Edwards Ambient

http://freemusicarchive.org/music/Filmy_Ghost__Peter_Edwards_Ambient/

online footages:

https://videos.pexels.com/videos/person-walking-on-pedestrian-lane-857593

https://videos.pexels.com/videos/walking-on-the-street-857592

https://videos.pexels.com/videos/people-walking-857589

https://videos.pexels.com/videos/white-footwear-857834

https://videos.pexels.com/videos/white-sneakers-857849

https://videos.pexels.com/videos/black-and-white-sneaker-857581

https://videos.pexels.com/videos/person-walking-with-support-857812

https://videos.pexels.com/videos/man-crossing-the-street-857584

https://videos.pexels.com/videos/person-walking-on-pedestrian-lane-857593

https://videos.pexels.com/videos/busy-people-crossing-the-street-857586

https://videos.pexels.com/videos/black-and-white-video-of-people-853889

https://videos.pexels.com/videos/man-step-on-a-puddle-of-water-854031

Sound Effects:

https://freesound.org/people/2hear/sounds/317533/

https://freesound.org/people/tec_studio/sounds/98491/

https://freesound.org/people/Robinhood76/sounds/318900/

https://freesound.org/people/bsumusictech/sounds/62333/

https://freesound.org/people/klangfabrik/sounds/160001/

 

Our project was first drafted out with a story plot. We listed the storyboard and setup in the google doc before shooting. Having gathered the raw materials, we started to play with the effects. We also download 2 music from the Free Music Archive that matches the overall atmosphere of our project.

storyboard

video folders

max patch layout

presentation mode

 

After meeting with Eric, he suggested that we should stay away from just playing with the color effects, but trying to explore other video effects, such as feeding the video to itself. Diana and I created a separate patch, just to explore different possibilities of effects on video footages, labeled them, so that they would be easier to be put in a bigger patch. That is also the part that we realized that our current audio is too plain to hold through the entire performance. Diana found several other online free music, that can be linked to the video as it played.

Having sorted out the transition of the videos, Diana and I practiced the patch together and created the presentation mode for the patch. Practicing gave us a better collaboration, and we came up with a map for the audio and visual control on the MIDI controller on separate sheets of paper, which helps to practice a lot.

Many visual effects were affected by the music being played. Such as the frequency of its motion. However, half of it was also improvisation live. Diana determines when certain sound effects come in, and they gave me the cue for the visual progression.

Performance

 

The performance during the final review was very good. It was better than any of the practices, and I got better control of the videos and knobs overall. I was very nervous, but that does not hurt my performance. One thing I would improve was the abrupt shift in one of the video effect transitions. I could have made it smoother by fading in the video effect.

 

Conclusion

Overall, this project was a very good exercise experience. I learned how important it is to have original footage content, and sometimes it is better to work in teams: I don’t think I can manage both the visuals and the sound at the same time. I also learned how important practice was. Playing around the effects and the patch is really important since that determines whether or not you can get good control of the video transitions.

Currently, I work mostly with videos, but it would be great if I can also incorporate footages of me live generating patterns, or affecting with 3D models, just like Eric taught us in class.