The McGurk effect: how you can hear with your eyes

Have you ever watched a badly dubbed film, where you can easily make out that the sounds from the actors mouth clearly do not match the visual movements of the lips on the screen. That is to say, even though both are auditory and visual senses are stimulated during the film at the same time, our brain can easily manage to pick up on differences between the audio and the visual, i.e. the lip movements that are supposed to go with speech sounds.

Similarly, it is also possible that are brain be fooled into hearing a false sound by watching the movements of an individual’s lips. This type of illusion is known as the McGruk Effect.

WHAT IS THE MCGRUK EFFECT

This type of illusion was first coined and described by Harry McGruk and John McDonald in 1976. The McGurk effect is a type of communication phenomenon that takes place when someone perceives that another person’s lip movements don’t match up with what they are saying (Tiippana, 2014). In other words, what people hear is entirely different than what is actually being said as a result of the visual input overriding what they are actually hearing. The brain is thus convinced that they are hearing something else entirely. Thus, there is a perception of a whole new message.

RESEARCH ON THE MCGURK EFFECT

Principal research on the effect has been conducted by neuroscientists at the Baylor College of Medicine, Houston. In the study, which attempted to demonstrate how the McGruk effect occurs, the participant was first asked to keep their eyes closed while listening to a video. The video was of a person making a ‘ba ba ba’ sound. Thus, the participant too heard a ‘ba ba ba’ sound when their visual sense was not stimulated.

After that, the participant was asked to open their eyes and to watch the mouth of a person in the video saying ‘ga ga ga,’ with the sound off. Thus, the participant visually saw the person saying ‘ga ga ga’ without their sense of hearing being stimulated. For the final step, the same video was played for the participant, now with the sound on along with the participant keeping their eyes open. People experiencing the McGruk effect reported hearing ‘da da da’- a sound that did not match any of the previous visual or audio cues.

The effect occurs because the brain tries to resolve what it think it is hearing with a sound that is closer to what it seeing (Nierenberg, 2017).

The McGruk effect has interesting implications given the current coronavirus pandemic. Given that we now need to wear face masks, this effect provides a reason as to why communicating with people might now be difficult given that we cannot see their lips when the mask is on. The effect clearly shows that people need both their ears and eyes to understand what people are saying clearly. While it may be true that the effect doesn’t take place much when an individual has their mouth covered, it does show that when one’s mouth is covered, a vital piece of communication has been lost for the person on the receiving end.

Further research on McGruk effect is important as it can help children with cochlear implants, as it can help one understand how visual speech can affect what one actually hears.

References

Nierenberg, C. (2017). The Strange ‘McGruk’ effect: how your eyes can effect what you hear. Live Science. Retrieved from: https://www.livescience.com/58047-mcgurk-effect-weird-way-eyes-trick-brain.html

Tiippana, K. (2014). What is the McGruk effect. Front Psychol, 5 (725). doi:10.3389/fpsyg.2014.00725