Abstract:
Subtitles, which convert spoken dialogue into written text on screen, have become a widely used tool for making audiovisual content more accessible to diverse audiences. This seminar explores how subtitles—one of the most common forms of audiovisual translation—offer a valuable lens through which we can better understand multimodal language processing. Processing language in multimodal contexts, such as reading subtitles while watching videos, is cognitively demanding. It requires the simultaneous integration of information from multiple sources—including images, auditory input, and written text—all of which compete for limited cognitive resources and require complex coordination of different cognitive systems. Drawing on findings from eye-tracking research, the seminar presents empirical evidence that enriches our understanding of the cognitive mechanisms involved in subtitle reading. Specifically, it examines how cognitive systems responsible for language, memory, attention, and eye-movement control work together to support this complex task in different situations. The discussion will also highlight how viewers adjust their reading strategies in response to increasing task demands, and how reading behaviours may vary between adults and children.
About the Speaker:
Sixin Liao is an Assistant Professor in the Department of Translation, Interpreting and Intercultural Studies at Hong Kong Baptist University. Her primary research employs eye tracking to investigate how viewers process language in multimodal contexts, such as reading subtitles while watching videos. A second strand of her research explores how subtitles can support the development of reading skills in slow readers and second language users. She is the recipient of the 2025 ESIST Prize for the Most Outstanding PhD Thesis and the First Laureate of the 2022 CIUTI PhD Award.
Background Readings:
Liao, S., Yu, L., Kruger, J.-L., & Reichle, E. D. (2024). Dynamic reading in a digital age: New insights on cognition. Trends in Cognitive Sciences, 28(1), 43–55. https://doi.org/10.1016/j.tics.2023.08.002
Liao, S. & Kruger, J.-L. (2023). Cognitive processing of subtitles: Charting the future by mapping the past. In A. Ferreira, & J. W. Schwieter (Eds.), The Routledge Handbook of Translation, Interpreting, and Bilingualism (pp. 161-176). Routledge.
Liao, S., Yu, L., Reichle, E. D., & Kruger, J.-L. (2021). Using eye movements to study the reading of subtitles in video. Scientific Studies of Reading, 25, 417-435. https://doi.org/10.1080/10888438.2020.1823986
Lopukhina, A., van Heuven, W. J. B., Crowley, R., & Rastle, K. (2025). Where Do Children Look When Watching Videos With Same-Language Subtitles? Psychological Science, 36(4), 223–236. https://doi.org/10.1177/09567976251325789
