Waseda WeeklyWaseda Weekly

Gestures are essential for freestyle rap! What is the role of gestures?

Associate Professor Sekine, what kind of research field is multimodal communication?

"Multimodal" refers to multiple "styles." In other words, the exchange of information using various modalities, such as gestures, gaze, facial expressions, and intonation, in addition to words, is called multimodal communication. In our daily lives, we naturally combine multiple modalities to exchange information.

Research into non-verbal communication has been active to date. However, the reality is that each modality has often been treated as a separate research subject. When we look at actual communication, we notice that multiple modalities are used simultaneously, and I am interested in the generation and understanding of such messages. In particular, gestures produced while speaking are often used to convey information that cannot be conveyed by words alone. There is a possibility that gestures contribute to the generation of one's own messages, and I am also interested in the mechanism behind this.

How do you approach research into multimodal communication?

Based on the research findings of various researchers in psycholinguistics and cognitive science, we are exploring the role of language and gestures not only in transmitting information, but also in thinking, learning, memory, and speech processing.

Psycholinguistics is an academic field that studies how people use, understand, and acquire language. For example, while linguistics studies language structures and rules such as grammar and vocabulary, psycholinguistics focus on how people process and produce language, and it is known that non-linguistic knowledge and cognitive and physical activities are also involved.

​Cognitive science, on the other hand, is an interdisciplinary field that seeks to elucidate the mechanisms of mental information processing, such as how humans perceive, remember, judge, and verbalize information. For example, when a person is faced with a puddle and decides whether or not they can jump over it, an inference process is at work that predicts the possibility by comparing it with their physical abilities and past experiences. In this way, cognitive science focuses on the perception, thought, and behavior that arise from interactions with the body and the environment. In that sense, cognitive science is a research field that is unique the School of Human Sciences, where I belong, as it explores "what it means to be human" from various perspectives.

Books on psycholinguistics and nonverbal research

What topic have you been focusing on recently?

Gestures are not just a means of transmitting information but are thought to be actively involved in speech production and the thought process itself, so we are conducting various experimental verifications of how hand movements promote speech production. For example, we are verifying the mechanism by which moving one's hands while speaking promotes vocabulary retrieval and sentence construction and enables one to speak fluently. Recently, we have been analyzing how brain activity during speech changes with the presence or absence of gestures through a neuroscientific approach using MEG (magnetoencephalography).

In a related study, I have also conducted research into the relationship between hand movements and highly improvisational language activities such as freestyle rap. In an experiment in which I asked professional rappers to rap while limiting their gestures, I found that the number of rhymes decreased.

In another study, the quality and quantity of gestures were compared between improvised rap and singing an existing song, and it was found that there were overwhelmingly more gestures in improvisation, especially gestures that depicted movements and objects. This result suggests that unlike singing existing songs (similar to the process of speaking lines from a memorized script), improvised speech activity (similar to the process of everyday conversation) requires verbalizing images and searching for vocabulary that fits the intended message, and that hand movements play an important role in this process.

Left: A plot of gesture movements and their trajectories measured with motion capture
Right: In another experiment, participants were asked to hold down the keyboard while speaking to suppress gestures. The data shown is a classification of speech disfluency.

Communication skills are developed through "observation" and "feedback"

How do people acquire communication ability during the developmental stage?

One of the major factors is "observation." Children learn how to convey information and emotions by observing the gestures and facial expressions of the adults around them. We formed an international research team with a research collaborator from Italy, a country known for having a "gesture-rich culture," and conducted comparative observations of how Italian and Japanese infants use gestures. The results showed that there was no significant difference in the frequency of gestures up to around 2 to 3 years of age, and that Japanese children tended to use slightly more gestures.

This result suggests that cultural differences in expressive styles may be formed at a later developmental stage than infancy. In fact, in Japan, when children enter elementary school, they are explicitly and implicitly taught to maintain good posture and are taught communication etiquette that prioritizes harmony over self-expression. For example, when listening to someone else, they place their hands on their thighs, sit upright, and listen with a serious expression. Through this posture education and observing the behavior of the adults around them, it is possible that the gestures of Japanese elementary school children gradually become more limited.

On the other hand, in Italy, children are educated under relatively liberal rules regarding how they use their bodies, and the adults around them use a surprising number of gestures in their daily lives. In this way, it is thought that communication specific to each culture develops from early childhood onwards through observation of the people around them.

A comparative study of gestures between Italian and Japanese children. A Japanese girl (left) and an Italian boy (right), both aged 3, use the same gesture when explaining a comb.

The other is "feedback from those around us." People learn how to express themselves through how others react to their words, gestures, and facial expressions. In psychology, there are things called "display rules," which refer to culturally shared ways of expressing emotions, such as "don't laugh when you're scolded" or "suppress negative facial expressions in public." These are social rules that have been established in specific cultural spheres as a result of the accumulation of reactions (feedback) from those around us.

The same is true for individual communication; it is thought that a communication style is established through repeated experiences in which an individual realizes that their behavior is not good if others react negatively to it and recognizes that it is acceptable if they react positively.

The effect of bowing was investigated. It was found that bowing or not changes the speaker's pronunciation and the impression given to the other person, and that bowing conveys more politeness even over the phone.

Also, in general, people tend to think that communication skill is the ability to talk to anyone or to sustain a long conversation. However, Professor Emeritus of Kobe Women's University Tatsuru Uchida states that communication skill is about "how to get out of a situation when communication doesn't go well." When you can't get your point across to the other person, you can use a different expression, add facial expressions, or use gestures to make adjustments. In other words, I think communication skill comes from the accumulation of experience in trying different methods to convey one's feelings to others. 

I want people to continue practicing "connecting" without worrying about what others think

What do you think "communication" is?

I think it is a "practice for connecting." Only by connecting with others can you gain empathy and trust. For example, by sending out a message, you increase the chances of being found by people who come from the same place or have the same hobbies. In that sense, even a T-shirt or keychain with a message can be a chance to connect with others.

In addition to language, it is also possible to connect with others through small gestures such as eye contact, body position, and bowing. For example, even a slight bow when receiving your change at a store creates a momentary sense of connection with the store clerk. I believe that the accumulation of such "practices for connection" is what creates society.

Finally, please give a message to Waseda students.

I want you to express your thoughts and feelings without worrying about what others think. In order to connect with others, you have to express something inside of you. It's okay to fail. University is also a time when you are encouraged to learn from your failures. Throw away your shyness, boldly take on new challenges, and make big mistakes.

When you see an elderly person on the train and want to offer them your seat, you may be afraid of being rejected in front of others, or you may not be able to speak up or ask questions in class. These are probably all related to embarrassment. If you can overcome this embarrassment, I believe your student life will change dramatically. If you are at a loss for words, try using non-verbal means such as gestures and facial expressions. First of all, let's practice connecting without worrying about what others think.