Sound + Vision

By Ivan Toth Depeña, 2015/2016 13-Month Affiliate Artist

Last fall, the Charlotte Symphony reached out to McColl Center regarding a series that they organize called KnightSounds. They wanted to pair a visual artist with the conductor and symphony for a one-day set of performances. 

The Charlotte Symphony’s Music Director Christopher Warren-Green and [Director of Artistic Planning] Tanya Sparks visited my studio at McColl Center. I walked them through and showed them things I was working on and examples of past projects. Christopher was intrigued by some of the new media and interactive work I had done in the past—particularly, my project at the Pit Arena in New Mexico. In that project, the large-scale light installation on the facade of the building responds to the sounds of the crowd inside during games and mimics the colors of sunset every evening. 

We agreed that light and responsiveness to sound would make for a compelling performance. From that point, it sounded like there was a strong possibility that I would be working with the Charlotte Symphony on a project for the KnightSounds event, ¡Sinfonica!. A couple of months later I received information that they wanted to move forward.

I have wanted to work on visuals for sound and performance for quite a while. I have always been very interested in music. Music affects me on a daily basis in various ways. For instance, sound and harmony inform the color and compositions of my work. Music is constantly playing in my studio. I play music (guitar) as a hobby and a sort of meditation. I have an enormous admiration for the musicians in the Symphony—the rigor and the technical aptitude it requires to play the complex musical compositions perfectly. Another fascination of mine is fusing different disciplines. This project was a perfect combination of all of the above.

We were on a very tight timeline, so I quickly started to sketch and think about what would make for an interesting collaboration/performance piece. I returned to some of my past research and projects that dealt with real-time interactivity and responsiveness through motion capture. I thought it would be interesting to capture the maestro’s movements in real time throughout the performance and have those movements affect video projections of my work on a large screen behind the performers.


Ivan Toth Depeña real-time artwork behind the Charlotte Symphony, January 29, 2016 (Photo by Michael O’Neill, Courtesy of Ivan Toth Depeña)

I called my programmer, listened to the music, and started the process. We used a motion sensor at the front base of the conductor’s podium. This sensor sent a signal to a computer that was stationed within 3-4 feet from it. The motion signal was captured, then sent via Ethernet to another computer on the balcony where two HD high lumen projectors (synced together for more brightness) were positioned to fill the back screen. This computer on the balcony ran custom software that processed the sensor's signals into the visuals seen on the screen. 

The software was designed to take the form of the conductor and abstract it by revealing 2D digital art work that I created. We carefully choreographed a combination of stills, video, and real-time movement acting as a "mask" to reveal and create abstract compositions in real time. The visual compositions were painstakingly cued with the musical performers to create an overall cohesive and seamless audio/visual experience. We ran tests. We practiced during the rehearsals. We revised and refined rigorously before the final performances on January 29, which went on very successfully (in my opinion).


Ivan Toth Depeña artwork behind the Charlotte Symphony, January 29, 2016 (Photo by Michael O’Neill, Courtesy of Ivan Toth Depeña)

This project with the Charlotte Symphony has opened the door to many other ideas and possibilities for me. I plan on pushing the audio/visual theme further in my work. I am also looking forward to the possibility of working with the Symphony again on more innovative projects in the near future. The whole process was inspiring on many levels.