The B'GARK! effect

Here’s something interesting.

It’s called the McGurk effect, and it is evidence that our brains use visual information to adjust how we percieve sound.

When you are presented with audio of someone saying “BA”, and video of someone saying “FA”, you hear “FA” or “VA”. The illusion is pretty strong - understanding how it works doesn’t weaken the effect. Very interesting.

I’m really interested in how visual feedback influences what a composer hears - in fact a lot of my honours thesis was about this topic. For example, when you are creating a melody on a piano roll or sequencer, would you make different decisions about it than if you were coming up with it on a piano or guitar?

For me, I definitely find that writing music in piano rolls or notation makes me obsess over harmony, and I feel I have to make everything very harmonically complex & interesting. Whereas if I sit down to the piano to write, I feel comfortable writing much simpler harmonies, because the simplicity of them isn’t foregrounded visually. When I play guitar I focus more on interesting rhythms and timbres.

This is a simple example, but I think it’s probably more complicated than that. If the first time I heard a song was watching the band play live, and the first time you heard that song was on headphones while watching one of those windows media player visualiser things, then we may have very different perceptions of what that song sounds like. But it’s almost impossible to compare, because we are listening to the same sound as it comes out of the speakers, and we don’t really have a discourse for describing our subtly different perception of that sound.

I think the McGurk effect is a nice bit of evidence to back up this theory that whether we realise it or not, our brains are always going to percieve sound differently depending on what visual information corresponds with the sound.