Thomas Demand presented his work at William & Mary recently. His pieces span sculpture and photography, and, a number of them are based on exploiting some compelling properties of paper. His work entitled “Poll”, seen at MOMA, presents the relationship between becoming U.S. President-elect and measured change in polling paper (see below).
Museum of Modern Art Thomas Demand’s “Poll” (2001), based on a photograph of a Florida recount station.
Poll, 2001, C-Print / Diasec, 180 x 260 cm© Thomas Demand, VG Bild-Kunst, Bonn / ARS, New York
But his pieces that most compelled me were the videos. Click here for his website featuring his simple but mesmerizing escalator video. Below, view a still from his video entitled “Rain”, which can be viewed partially here.
Here’s the trick with the “Rain” video: the image was created using candy wrappers (which you can just about make out in the still above) and the background sound is not of rain but of cheap oil frying (the cheaper the better, or so Mr. Demand says).
There wasn’t any indication from Mr. Demand that psychophysical phenomenon were drawn from in his artistic endeavors but, for me, the alignment is too symmetrical to not consider.
There is a large literature on the discord between sight and sound, referred to as intermodal
asynchrony. Aside from thinking about what happens in Demand’s “Rain” video, there is a well known effect that we can draw from to find our answer, the McGurk effect (McGurk & MacDonald, 1976). The McGurk effect always yields an “oooh” resound from the classroom when it’s revealed because the Va Va Va auditory sound heard when eyes are closed appears to be Tha or Da or Ba when the speaker’s mouth is visually observed creating such sounds while the Va sound is heard. Here’s a video demonstrating this effect.
So, there is a primacy for the role of vision in auditory processing. And this is useful. We’ve all been in the loud restaurant (or, years ago, the bar with pounding music) where we can’t hear the person talking to us unless we can see their mouth moving. In such cases, owing to context, the visual signal typically matches the auditory and there is success in hearing the words “Let’s get out of here” or “Wanna dance?” correctly.
More recently, a different version of the McGurk type effect has found its way onto the tele wave: bad lip reading. In this case, auditory sounds are generated based on a misinterpretation of mouth movements. The results are pretty hilarious. See here, and here, and here for some great examples.
Like the McGurk effect, in bad lip reading, the auditory sounds are generated based on what the observer visually perceives the mouth as pronouncing. Unlike the McGurk effect, the soundtrack in bad lip reading is then created to match that visual perception. The errors between what is interpreted as being said and what was actually said have to do with complicated lip movement patterns inherent to quickly spoken or muttered sentences. The McGurk effect proper doesn’t suffer from a bad lip reading problem because it deals with clearly visible, single syllables beginning with v, or d, or g, or b (each having a unique mouth movement associated with them) and ending in a.
Essentially, Mr. Demand’s “Rain” video is an artistic rendering of the McGurk effect! The sight of the rain drives the auditory interpretation. One might suppose then that the opposing video could be made: the sight of oil frying in a pan while the sound of rain is played. In this case, it would be the pan frying sound that would be heard revealing, again, the value of the sight over sound.
One can imagine pushing this effect even further, though. The sound of a dog chewing a toy while watching a car wreck; the sound of bowling ball falling while a bird lands on a tree limb; the sound of applause while popcorn pops. The question then becomes: how much jurisdiction does our visual perception have over our auditory system? Or, when and where are we willing to associate sights and sounds and, more importantly, how might this impact or subsequent thoughts and behavior? The answer may be found in a psychophysical lab. On the other hand, the answer may just as easily be found in a museum.
visualparticles.org 