The Surprising Benefits of Subtitle Use for Literacy Development – A conversation with Professor Jan-Louis Kruger

In this insightful interview with Professor Jan-Louis Kruger, a leading researcher at Macquarie University (Sydney, Australia) in the field of subtitling and eye tracking, we explore the fascinating topic of subtitles (captions) and their potential benefits for children and individuals learning to read.

Professor Kruger shares valuable insights from cognitive and linguistic perspectives, shedding light on how subtitles influence our reading behavior and comprehension.

Traditionally, we associate subtitles with accessibility for individuals with hearing difficulties. However, recent studies suggest that subtitles can also support literacy development. Eye-tracking studies have shown that people, including children, automatically engage with subtitles, even if they claim not to read them!

Discover more about the vast potential subtitles have for literacy,  and about the exciting opportunities to leverage subtitles as a strategic tool for improving reading abilities.

Watch the interview to Professor Jan-Louis Kruger on YouTube:

 

 

Interview Transcript:

(All links mentioned during this conversation can be found at the end, under “Resources to Explore Further”)

Laura, Learning Reading Hub:

So, today I have the pleasure of interviewing Professor Jan-Louis Kruger from the Department of Linguistics at Macquarie University in Sydney, Australia, to talk about a very interesting, even fascinating topic, but, probably, an unexpected one as well: subtitles, and how we can use them to our advantage with children (and with anyone, really) learning to read. Today, I’m very lucky to have a leading researcher in the subtitling world. Professor Jan-Louis Kruger, welcome to the Learning Reading Hub!

Professor Jan-Louis Kruger:

Hi Laura, thank you very much. I’m very happy to be here.  I’ve seen the work that you’ve done and it’s really impressive. So, thank you for having me.

Laura, Learning Reading Hub:

Thank you.

So, this is fascinating, because, normally, when we think of subtitles, in our own language, what comes to mind is accessibility, really, and how they can help maybe people with hearing difficulties, for instance. We don’t think of subtitles in the context of learning to read.

However, there are studies, especially -I believe- a study that was carried out in India that used animated books to increase literacy rates in rural areas that indicate that they can support people in the process of learning to read.

It looks pretty incredible, so Professor Jan-Louis Kruger, taking into account all that you know about subtitles and eye tracking movement in audiovisual formats, can you share with us what exactly is going on from a cognitive perspective. Do we even read the subtitles? Do children read the subtitles?

Professor Jan-Louis Kruger:

Yes, that’s quite a lot of ground to cover, but I’ll see what I can do. I think, first of all, if we if we take what you said is exactly right. Subtitles, the first thing you think about is accessibility. That they are there for people who deaf and hard of hearing, to give them access to the soundtrack.

That’s why when you switch on the subtitles, you often see things that you don’t need, you see a description of sounds or you see maybe a label for who’s speaking, things that you don’t need if you’re a hearing viewer, but that you need if you are deaf or hard of hearing.

Now, so, yes that is true, but what’s also happened and there’s been quite a lot of media attention to this lately is that people are increasingly switching on the subtitles in any case, even if they don’t need them; and I know I do it myself and there was an article in “The Atlantic” recently (I’ll send you a link to this later), where somebody was discussing this: that their friends are all doing this and I don’t quite understand why. I think that I can talk about that for a long time, and I know your focus in on reading development, so I’ll try to limit myself mostly to that, but the thing is that, even if you don’t need the subtitles, subtitles can give you support.

They give you a stable, a more stable, source of information than the auditory input, if you are a proficient reader. If you’re an expert reader and you watch something with subtitles, they become like a safety net you can, whenever you don’t hear something well, you can, just jump down to the subtitles and start reading them.

And we know quite a bit about this, not as much as we know about reading of paper, book paper, or static text on screen, but we have an idea of how people read subtitles, how they divide their attention between what’s on the screen and what’s in the subtitles, and how things like speed influence it.

But to take a step back, the colleague that you mentioned, Brij Kothari, in India, has actually been working on the use of subtitles in literacy. In other words, teaching people to read for almost two decades now. And this was initially… He did that with music videos.

So, rural areas in India, you would, they would start using them in some villages. They did experiments where they would have maybe just one television set with subtitles

on all of the music videos, where people would watch all of those videos over and over with the subtitles, and because they know the words already, there is a very clear gain in their literacy.

So they start recognizing words when they see them, and I think that’s the essence of this, that’s really what this is about. It’s about exposing people especially.

Although it’s more effective for efficient readers, you can with people who are beginning to read, and I think that extends even to children, who are just beginning to read.

You can use subtitles quite effectively to help them to start recognizing words more easily. So, what you get is, and I think this is the main benefit, is that you get a double exposure. You hear a word and, at the same time, that you hear it you also see that word on screen. And, after a lot of repetition, you will start recognizing, you’ll know, when I hear this, this is what I see.

If you know a little bit about the phonics, of if you’ve already learned that, then it becomes easier for you to recognize those words. And we know, you know, it’s a topic of a lot of what you do on this channel. It is a lot about practice, about repetition.

You have to start recognizing those words, and you have to start making those links with the with the sounds, and that’s how reading development progresses, and that’s why I think subtitling can be really, really effective, even at a very young age, but I can say more about that in a moment.

Laura, Learning Reading Hub:

Yes, well, this is very good because children like to watch the same things over and over, so there’s that repetition there. Do they read the subtitles even if it’s a show they’ve seen… Or people, in general, they’ve seen several times… Or do we skip subtitles when it’s something we’ve watched several times?

Professor Jan-Louis Kruger:

Yeah, no, it’s a very good question because if you if you ask people, do you read the subtitles?  Then, you always find some people who say: “No, I don’t look at the subtitles. I ignore them.”

But there’s been a number of eye tracking studies where we found that, not me necessarily, but other colleagues have found that even if people say they don’t read the subtitles they can’t ignore the subtitles, it’s really hard not to look at the subtitles. Even if you don’t realize that you’re looking at them, you are looking at them.

Now, whether you’re going to have benefits if you only look at them like a nuisance on screen. In other words, subtitle appears, you automatically look down because there’s something on screen that wasn’t there previously and as soon as you realize what it is you look away again, then of course you’re not reading it, so it’s not really making, it can’t have any benefit.

It could have a harmful effect because it’s distracting you from maybe other sources of information, but, in general, there is a researcher in Belgium who’s been working on this for a long time and he and his colleagues have found in adults, as well as children, that there is something that’s very similar to automatic reading behavior.

Even if you have subtitles in a language in that you do not understand. You can’t read that language, let’s say the soundtrack is in your first language, and the subtitles are in a language you don’t read, you will still try to read that. You’ll try to find something that’s recognizable, and this is what I think will happen is with children, with adults.

When you’re beginning to read, you’re going to start looking for things that you recognize and we know, in reading literature, there’s a lot about the word frequency effect. If there’s a low frequency word, in other words, if there’s an unfamiliar word, that you haven’t really seen before, it’s going to take you longer to process that.

If it’s a high-frequency word, you can process it really quickly. Little words like “of” and “it” are words that are very high-frequency. You don’t even necessarily have to look directly at them. Even if you look at the word just before them, you can recognize them in your peripheral vision, and you can you can carry on without having to look at that word specifically.

So, I’m not going to go into those mechanisms, but what this means is that as you start recognizing the form of a word, the shape of a word, of those letters together, you’re going to start making more and more words higher frequency, so that you can read them more efficiently, and this I think applies to adults as well as children.

Now, there are a few critics who’ll say it’s senseless to show captions to small children because they can’t read yet, you only start getting benefits once somebody is a proficient reader and I think, to some extent, there’s definitely truth in that.

If you’re going to put your 5-year-old, 7-year-old in front of a television with a normal program with subtitles directed or aimed at adults, they’re not going to be able to cope with those subtitles.

They can’t read that fast, we know how slow reading is at first. So, you can’t expect any benefit from that, and if anything, it’s going to make them panic, because they can’t keep up. But if it is a program that is at a pace that is aimed at children, then, then, it becomes a different matter. Now, anything with normal speech rate, even if it’s a slow speech rate,

I would say you probably need to be like in year 4 or up to start getting benefits from that. But if you’re talking about children’s programs, if you’re talking about nursery rhymes, children’s books that have been made into videos, like Planet Read, what they do, then what you have is you have a very slow pace, you have a lot of repetition.

And we know this from all of our little, all of the storybooks that I used to read my children when they were young, the one thing that you always find in books for beginner readers is a lot of repetition. It’s like a little rhyme that keeps going, and just one word in the sentence changes from page to page.

So, they start recognizing and they start forming that memory for those words in their long-term memory. So that they can, then, by the time you get to page ten all of the words have been there for ten pages, and only one word changed, and they start recognizing those words.

So, I think that’s the principle behind this, and why I do think children will read those words. They will, even if they can’t read it, they will try to read it. And the more they try, the more the chances will increase of eventually recognizing those words and being able to read it. So, I think it’s as simple as that.

Laura, Learning Reading Hub:

It’s funny how it’s at the same time unnoticeable for us, because I use captions as well at home. For us, it started when we first moved to an English-speaking country, my husband, well, boyfriend at the time, and myself like 15 years ago or 16 years ago, we couldn’t understand everything that was said on TV. So, whenever available, we would have the subtitles on. And they weren’t always available and they weren’t so good at the time, like sometimes out of sync, horrible experience. Anyway, fast forward 15 years, 16 years, we still use the subtitles and we don’t even know why, we don’t even notice they are there. And it was only once that my older daughter, I think like two years ago or something like that, she asked me: “Mom, why is  our TV so weird?”

I was like: “What do you mean our TV set is weird?” “Yeah, there’s always those letters below and I don’t see those letters when I go to my friends’.

You know like: Their TV sets are normal; they don’t have those letters. Why do we have those letters? I was like: “Ah, okay, do they annoy you? Do you want them off?” And, she was like: “No, no, it’s fine, you can keep them on, but I just want to know why our TV set is so weird.” She just wanted to understand, you know, she was curious about it, and wanted to make a point that our TV was very weird.

But, anyway, are we the norm? That we get used to them, it gets to a point that you don’t even notice they are there. And it’s the same for me and my husband and my children, we don’t notice they are there anymore.

Professor Jan-Louis Kruger:

Yeah.

Laura, Learning Reading Hub

That’s the experience we have, so it’s funny how we don’t notice them but, at the same time, we can’t help ourselves but read them. You know, like your eye tracking movement experiments say that, that we always try to read them, that we cannot ignore them. It’s funny, isn’t it?

Professor Jan-Louis Kruger:

It is funny, and it is, we talk about salience, and there are certain things that they are so salient. When we look at the world in front of us, there are certain things we will always prioritize, you try and you always look at that whenever you can. So, if you’re  talking to somebody you’re gonna look at the eyes, you’re going to look at their mouth, because you’re trying to get information from wherever, you know, the most salient information will be.

If you’re looking at…  Just look in front of you and something suddenly moves very fast, then your eye is going to be drawn to that, so that’s automatic. And we talk about top-down and bottom-up processes. Bottom-up is automatic, you can’t, it’s very difficult to avoid doing it, and top-down is where you are in control.

You know, I want more information about something, so I’m going to look at a particular place to find that information. So, the centre of a screen is very salient; anything that’s marked by contrast is salient; when I move my hand, that’s salient, you’re going to look at it even if it doesn’t mean anything.

You’re going to look at movement, definitely, but then there’s also a text bias. If there is text on the screen and you can read, you’re going to try to read it. If you can’t read, you’re going to want to be able to read it, and you’re going to look at it in any case, I think.

So, yes, it’s very difficult not to look at it and once you’ve gotten used to it, you’re going to find it very difficult not to do it, and you’re going to miss it when it’s not there.

Whenever I switch on the television and the subtitles are not there, I feel like I’m walking on a bridge, and there’s no hand railing.

It kind of feels like something is not right here, so you want to switch it on.

Laura, Learning Reading Hub:

Same, same. Same experience. I watch them in Spanish, as well. I want them there.

I’m just too used to them so… You know, it’s funny as well because there are so many opportunities to add reading to our lives, maybe not in a conventional way, not literature, but in a way that is also valid.

Especially when we’re learning to read, so it’s crazy the number of hours we can sneakily add up, right? So, I mean, the potential is huge, especially if we want to be strategic about it. For instance, this got me thinking maybe we could create shows for struggling readers maybe using lots of high-frequency words, going at a slower pace…

For beginner readers, with lots of simple words, repetition, CVC words, going slowly as well… I mean the possibilities are endless and it’s very exciting, it’s very exciting

What do you think about this? Maybe there’s something available, like some shows, some initiative that I’m not aware of that you want to share with the audience… I’m not sure.

Professor Jan-Louis Kruger:

I have to admit I’m not aware of any specific initiatives. I do not know whether Planet Read, for example, is available outside of where they’re using it, but what I do know is that there’s been… I’m gonna try and stay focused here, because there are a few topics I want to cover.

The one is that there are a number of initiatives underway at the moment in the UK, in Europe, in America, in India as well, that are aimed at driving policy around the use of subtitles; to say subtitles provide exposure to reading in a world where children read less and less.

There’s been surveys in the UK that says only about half of children read on a daily basis, or like reading; sorry, I have some notes on that. That’s…Where is that?

The National Literacy Trust in the UK did a survey and they found only half of the children surveyed enjoyed reading and less than one-third of children read daily, on a daily basis. So, that’s not a lot of reading. But, then, if you ask how much television do children watch, how much screen time do they get from the day (And I know many parents control that, many people say television is bad for you… That’s not an argument I’m going to get into at the moment). But the thing is if you expose children to television, and you take it as a learning opportunity, and you make sure that they have the subtitles on, then you’re actually giving them exposure to reading as long as you make sure that what you’re showing them is pitched at the right level for them.

So, you don’t need to go in and look for expensive solutions out there because these things are there, that’s available, subtitles are available, but then yes but what you said is exactly true. If you want something that will, where you can see gain, specific gains and use it in a goal-oriented way, then you need to start looking for resources that either can, if you’re a teacher or a parent, that you can use or something that you can manipulate or you can develop as a group.

Or things where you have, as you said, a lot of repetition. You don’t need to look very far for that, anything with children’s music videos, you’re going to find that. It’s there already. If you have something that’s karaoke style, especially with the subtitles highlighted as the words are sang. That’s going to be really good, because it helps them make that link.

If you want to go a little bit more towards an intervention, then you could design these things, so that you have the words repeat, and you can have the words highlighted in a certain way, underlined, make them flash… Anything to help that beginner reader to know which word to look at, at a particular moment.

So, if you want to teach a specific word, then you can maybe just highlight that word repeatedly, so that that image is kind of captured in their long-term memory. That word identification with the meaning attached to it.

Now, I think this is maybe one of the main reasons that subtitling can be beneficial in this context… When you’re using subtitles, you’ve been exposed to language, contextualized language, very richly contextualized language.

It’s different from just putting down a page with text in front of a child. Even if it’s an illustration, if you put down a page in front of them, they can pace themselves and that’s brilliant and that’s very good. But if you have something like audio-visual content, multimodal input (and picture books count as multimodal as well). But if you have something of sound, images and pictures and words, then it means that it helps them to integrate those things.

When they see a word, that word comes with a rich context associated to it. It’s not just the word, it’s not just the meaning of the word. They know who said the word, they can hear the intonation of the person saying the word, so there’s a lot of rich information that comes with that.

And I’m pretty sure that this is something that will have long-term benefits in creating those pathways in the brain, because we need to form those connotations between things.

That’s how, we, that’s how cognition works. It’s that you get the input in your sensory memory and, then, in your working memory, you do the word recognition. You understand the word and, then, you see it in context, and you start integrating it into sentences and, then, next time you see that word in a particular context, it will be very easy to recognize.

We’ve drawn our long-term memory, whatever is thought there makes it easier to process new input and that’s where something like captions (I’m not saying captions is going to be miracle worker in teaching children to read. It’s not like stop teaching children to read because the television will do for you, that won’t help), but there are a number of ways that you can do this; and there are a number of initiatives out there

I’m sorry I don’t have the specific examples, but Google is very useful for things like that.

Laura, Learning Reading Hub:

Yes, yes. I will share. I’ll leave links on the video description for further reading reading for people that are interested in digging into this more. For sure. Now it’s very interesting, and it’s about repetition, isn’t it?  To get things into our long-term memory, so definitely a very good way to do it, and a sneaky way to do it… And, also, you touched the point of karaoke-style subtitles, and I was gonna ask you about the different types of subtitles, because now that I pay more attention to subtitles, I’m more observant of them, I’ve noticed that there are different types of subtitles.

For instance, you’ve got subtitles in which words are presented as they are said this is normally auto-generated, like in YouTube, or, then, you’ve got subtitles where you have the whole sentence maybe in a line or two, and, then, you’ve got the karaoke-style subtitles where words are highlighted as they are being said.

So, I just wanted to know which ones are superior? What are the best practices here?

Professor Jan-Louis Kruger:

Okay, maybe, sorry, can you just repeat the ones that you’ve mentioned now?

Laura, Learning Reading Hub:

Yes, so… Words are presented as said like auto-generated, like in YouTube. Then, you’ve got the whole sentence, where you see a line or two with the whole sentence. And, then, karaoke-style subtitles. Those are the three styles I’ve spotted, but maybe there are more that I haven’t.

Professor Jan-Louis Kruger:

So, the different styles… If you just look at this context, then, yes on YouTube you get a lot of auto-generated subtitles, which means that they make use of algorithms for speech recognition to basically do a transcript of what’s being said. And, then, they use other algorithms to time that, so that whatever is being said is then linked to the timer that it said. Because that’s the thing: in order for us to be able to integrate the information, as soon as somebody says something, whatever they say has to be on the screen. So, it has to be synchronized.

Now there’s a lot of algorithms out there who can do that can do a really good job with this, so that, what you see is not that terrible. It’s getting better also. The big problem with those auto-generated subtitles is that there’s going to be errors in there, because of speech recognition, because of the fact that people speak in different ways. It’s never going to be perfect; it’s never going to be without errors. There will be errors. People will misspeak, so that’s the one problem, there’ll be errors. The second problem is that whenever you transcribe spoken language, it becomes a little bit less coherent, because spoken language is less coherent than written language.

When you write down something, especially in an email or something like that, you don’t just type randomly and send off. Well, some of us do, but and you’re always sorry if you did. But, typically, what you do is: you write it, you read it again to make sure it’s correct, and then you press send. That’s what we should do.

Anything that you read has typically been written in a way that is easy to read. When you look at spoken language, there are a lot of false starts; where people start saying something and then they say ‘um’ ‘uh’, and then they start reformulating. Before you know it, there’s a sentence that’s 20 lines long, because they haven’t put in any punctuation.

So, if you transcribe that (and this is what I’ve just been doing now), it’s still not going to be very readable, so that’s the problem with automatic transcription.

Karaoke-style, you typically only find it with something like music videos, and I think that is useful.

So for the reasons that I mentioned, I would not recommend exposing children to captioned YouTube videos… Unless you have vetted that, unless you’ve checked to see whether those were post-edited captions or subtitles, or whether they were just auto-generated ones.

If they were auto-generated ones, they tend to be really, really… Very fast. So, it’s almost impossible to read them, in any case. And they’ll be full of errors, so would you run the danger there, especially for somebody who’s beginning to learn to read, that they’re going to learn the wrong spellings, the wrong forms of words?

So, I don’t think that’s great.

I’m not saying that you won’t find things on YouTube that have proper subtitles, but the auto-generated ones are not that.

So, what I would say is the best thing to do is to look at the subtitles. I would say if you, as an adult, can look at the subtitles and read the subtitle that’s on screen about three times before it disappears (This is just a rough guess, but something like that, then you know that it should be okay for a beginner reader).

In any case, if you struggle to read the whole thing, then you know that your child won’t. So, this brings me to the one of the main things to consider when you look at the use of subtitles. I already said it has to be appropriate for the age, for the user.

Laura, Learning Reading Hub:

Yes.

Professor Jan-Louis Kruger:

The other thing is that you’ve established that the speed at which the subtitle is presented is really important. We’ve done a lot of work on that, to see what happens when the subtitles are really fast. Companies, streaming companies, have guidelines. They say subtitles cannot be faster than 20 characters per second. Characters per second is just the way to express the speed (how many characters were presented in one second). That is kind of designed to say this is how much people can read.

Conventionally, in countries like Scandinavia, they recommend something like 12 characters per second, even for adults.

Netflix, sorry, I shouldn’t mention specific companies, but I know Netflix has specific guidelines, between 20 characters for adults, and I think something like 16, 17 for children, or it could be a little bit less than that.

But it’s much faster than the conventional one.

So, if you want benefits from subtitles, I would say that you need to try and find subtitles that are really slow, because if you can’t read the subtitle to completion, it’s not going to benefit you.

If it disappears before they’ve started reading it, and ideally for beginner readers you want them to be able to read it more than once.

You need them to be able to, when they get to a word they don’t know, to stay there for a little bit longer and figure out what it means, maybe come back to that word after they finished reading the other words.

Karaoke-style, great if you can find something with good karaoke-style subtitles. That will help with reading along, because they can, as the word is sung specifically (because karaoke-style is normally with music videos), it is highlighted, so they know which word is being sung at the moment.

So, yes, that’s definitely useful. Other than that, as long as the subtitles are not too fast and don’t contain errors, I think they will be useful.

Laura, Learning Reading Hub:

Okay, perfect.

Professor Jan-Louis Kruger:

I don’t think parents should worry too much about whether this is exactly at the right speed either, or right-pitched. It can be a little bit above, because that’s that gives incentive.

I mean, if you show karaoke-style nursery rhyme to a 10-year-old or a 9-year-old, they’re not going to like it, they’re going to get really frustrated with it.

So, it can’t be too easy. It has to be a little bit above what they can do and, then, they will aspire.

Laura, Learning Reading Hub.

Okay. Makes sense. What if what is said by the characters in the show, in the movie, doesn’t match exactly with the words that are you see on the captions? Because I’ve noticed this is happening sometimes. They may not transcribe every single word, they may use a synonym… And I have my own little theory about this. I think this happens a lot in shows that have been translated from one language into another language; and it feels like the doubling team doesn’t really speak to the captions team, sometimes. Well, anyway, if this happens: Is this good? Is this bad? Can this be confusing for children, for adults?

What’s your opinion?

Professor Jan-Louis Kruger:

Well, I think people should realize that when… I mean, I’m talking about commercial things that you see on television, on streaming platforms… Very rarely will it be a mistake.

In most cases, the reason that you find a different word is simply because the subtitler was trying to make it possible for the reader to finish reading the subtitle before it disappears.

So, as a subtitler, I teach students to subtitle and the majority of the time we spend on finding ways to bring the same meaning across in fewer words, fewer characters.

So that’s why we use synonyms, because it’s a shorter synonym, it’s maybe a slightly higher-frequency word that will be easier to read.

So, that’s the reason this happens.

And, also, we have to keep in mind that subtitles are in the first place intended for people who cannot process the soundtrack. So, it’s either because they are deaf and hard of hearing or it’s because they don’t speak the language of the soundtrack.

So, if you are fluent in English, but your first language is Spanish, and you read something in English or Spanish subtitles or vice versa, then this was not meant for you. It wasn’t meant for you to be able to or to check whether it says exactly the same thing.

Because, I think in 90% of the times when you notice that difference it doesn’t really change the meaning.

It was a different word, but it doesn’t change the meaning.

So, that’s the idea. It’s that when we subtitle, we try to find ways to respect the technical limitations, and they are time limitations, you want to keep your subtitles in a specific speed.

So, if you have two full lines of text for adults, you want at least four seconds for that to stay on screen.

You cannot use more than two lines, because then you start covering too much of the screen. You cannot have too many characters in a line because on some devices that won’t be visible.

That’s why when I teach students, we still use slightly outdated characters but spaces, but we probably talk about between two lines total, so that’s 78 characters in total for a subtitle.

If you think English has an average word length of five, then you can do the math,  how many words that is. But it means that if you transcribe whatever’s in the audio and it goes over those counts, you have to limit, you have to cut it down, you have to find ways, you have to maybe remove an adjective or an adverb, or an article… Reformulate the sentence, so that it’s just flipped over in a way to reduce the volume of text and to be able to leave that text on screen longer.

So that’s the two reasons.

It’s either there’s too many characters and we have to reduce it, or the speed will be too fast. There’s not enough time to read all of that, so I need to reduce the number of words.

So, that’s the reason.

But your other part that’s important of what you asked is, I think that for an adult who can understand two languages, it’s going to be a distractor.

You will start maybe distrusting the subtitles, if you know that the subtitles say something different from what’s in the audio.

Laura, Learning Reading Hub:

Okay.

Professor Jan-Louis Kruger:

In a learning context, I think it is important to not have that happen.

If you intend to use subtitles for, especially reading development, you really want those subtitles to be verbatim. You want them to have exactly the same words as in the video and no mistakes either.

Laura, Learning Reading Hub:

Okay, no, makes sense. And that got me thinking… What’s the difference between closed captions and English subtitles? Because maybe there’s a difference there, and one of them is for an audience and the others are for a different audience.

Professor Jan-Louis Kruger:

Yes, so closed captions. It could be a bit confusing because some people talk about captions and others about subtitling, and some people think captions are only for deaf or hard of hearing, whereas in other contexts people use captions for anything that’s at the bottom of the screen…

So typically closed captions, when we talk about open and closed, then closed captions would typically be subtitles you can switch on and off, so you need a key to unlock them. So, they’re not there. They’re not always there.  You have to press the button on your remote control to select subtitles, otherwise it won’t show.

So, that’s the closed bit. So, it’s purely whether you can them switch on and off.

Open subtitles, open captions. You can’t switch them off, they will be there.

But, typically, when they talk about closed captions on media or the net, on YouTube, Netflix, anything like that…  Then that refers to subtitles for the deaf or hard of hearing, which means that it’s not just the dialogue. It’ll also be sound effects, there will also be maybe speaker identification, the name of the speaker will be there. And this varies.

You will have seen that on different shows, and sometimes and you have the person who speaks is named, sometimes not. Some shows will even have color codes, so that one character’s words are in white, the others are in maybe red or different. There’s a hierarchy of these colors that they use.

But, normally, what most of us will be exposed to is… You will simply see in square brackets the name of the character. Or you’ll see the sound effect.

So, those are captions.

Whereas subtitles or English subtitles would typically not include the sound effects. It will just be the English and the audience there would be maybe people who are second language users who want to see sign language subtitles, English audio with English subtitles, and that audience is quite big. There’s a lot of people who want to see the English subtitles.

They don’t need the sounds, but they want the English.

So, you would select… And you will see it in any DVD or streaming platform or anything like that. There are often choices:  you can choose different subtitles, you can choose different languages, you can choose English or you can choose closed captions, and that’s the difference.

Laura, Learning Reading Hub:

Yeah, makes sense, okay.

Another question… While I believe that most studies have been done with children that have received some sort of reading training, that have started reading instruction already, you also mentioned that there could be a benefit for really young children, but should they at least know the alphabet or something to  benefit from this?

Professor Jan- Louis Kruger:

Oh, yeah, no. I would say… I mean, I’m not a reading expert. In that sense, there are people here at Macquarie who specialize in that kind of thing, and I think you’re a bit of an expert yourself on phonics and how we teach children to read.mI, just as a novice on that level or on that topic, I would say it would make sense that for people to be able to read, they must have the tool set to be able to recognize sounds, what sounds look like, and that’s why you need to first teach them what the letter looks like, what that letter sounds like, what are the different sounds that letter can make, or whatever.

I know I sound like an idiot now.

Laura, Learning Reading Hub:

No.

Professor Jan-Louis Kruger:

But phonics I think is important, you need to have that, at least. Otherwise, they’ll be looking at, it’s like if you were to see subtitles in hieroglyphics, or in Chinese characters, if you don’t speak Chinese.

We can’t read Chinese, it will just be pictures. It will always just remain pictures. I can look at Chinese characters as much as I like, it’ll still remain pictures. Until I started, at least, making some association, something like that it’s a very difficult thing to do, especially when you’ve grown-up already.

But I do think that the benefits will start escalating the more of a foundation there is. So, again, that’s why I wanted to say this is not a miracle cure for illiteracy. It’s something that can be really beneficial, but it has to be used responsibly, and you have to have realistic expectations.

Laura, Learning Reading Hub:

Yeah, it’s not a magic pill, no.

Professor Jan-Louis Kruger:

It’s not a magic pill, no.

Laura, Learning Reading Hub:

I also understand, and you mentioned a little bit about second language learners, that there’s another huge benefit with subtitles for learning a second language. Any insights on this that you’d like to share with us? Any tips?

Any studies that you know of?

Professor Jan-Louis Kruger:

There are quite a few studies on subtitles and language learning, and  I have a student who just completed the PHD on the same topic as well.

So, there is some evidence. It tends to be a fairly frustrating thing to study, because the benefits that you get from something like this is benefits that you get over a long time with a lot of exposure. So, if you show somebody one video of 10 minutes or even an hour, and you expect them to have learned words from that, then you’re not realistic.

You have to… Again, I think the benefits will start escalating the better your knowledge of that language that you’re learning is. If you have really intermediate skills and maybe more higher level skills, advanced skills, you’re going to start learning much faster.

But, at the same time, I think anybody who has ever watched something in a language that they don’t understand at all will know, you can have the subtitles on for as much as you want, it’s not gonna make it stick. Maybe there’s one or two words that you’re going to start recognizing.

I mean, I watched “Narcos” and this particular word that I won’t repeat that they say all the time, and I can recognize that word now in Spanish, but I’m not going to recognize a lot of the other stuff.

I mean, some things you recognize because, of course, you have something similar in your language or a language that you know. So, again, not a magic pill, but I do think if you have a basic knowledge of a language, or if you already know the language (you’re just not fluent in it), then I think the benefit starts increasing.

If you’re a second language speaker, it means you already have a fairly high proficiency. It’s not a foreign language you don’t know much about.

I know a little bit of German and French, and I wouldn’t call myself second language in either of those, but I would be able to benefit a little bit, but I’m gonna struggle. I’ve tried that with German. I did five years of German at school.  I can read it relatively okay, if it’s not too difficult, but if it’s in subtitles or if I hear German and I have any subtitles on, that processing takes too long, so it’s very difficult to learn something.

And, again, it’s only with repetition that you’re going to be able to get that benefit.

Laura, Learning Reading Hub:

Totally. You need to bump into the word over and over. That’s my experience. That’s my personal experience. And you need to be a fluent because, otherwise everything is gonna be new and you won’t be able to remember all of those words, in my experience.

Professor Jan-Louis Kruger:

Yeah.

Laura, Learning Reading Hub:

Okay, let’s get very practical, what exactly can we do with what’s available right now?

What specific entertainment platforms are especially good for this? Even though maybe you don’t want to give names, and that’s totally fine, no pressure. And which ones are not so good? But again, no pressure, and maybe you don’t even know.

And, what can we demand from entertainment platforms? Because, at the end of the day, we are their customers, and we can make changes happen if we ask for changes.

Professor Jan-Louis Kruger:

Yeah, that’s a very good point. First, I have to start by saying I can’t give you names. Not because I don’t want to, but because I don’t know. It’s just not something that I use personally, it’s not something that I’ve explored enough.

What I can say is that there are a number of groups lobbying for subtitles to be switched on in places like the UK and America and in India.

They’re doing they have a big project now, called BIRD (Billion Readers) where they’re trying to roll it out in the different languages of the different regions.

So, I think there has to be kind of a lot of pressure from different angles. Political pressure would be nice, but we know politics doesn’t necessarily work like that. If there’s not money in it for somebody, then it’s going to take a long time before people will actually do something about it.

But we’re busy trying to collect as much data as we can, evidence as we can, so we can tell people: “If you do this, it’s a cheap solution.” So, I think turning on the subtitles, the thing you can do is spread the word.

Turn on the subtitles. That will be beneficial. The thing to maybe exert some pressure on the providers would be if there are people who provide children’s programs, who create them, produce them, if you can talk to those companies and say, this is something that could benefit children, so, please, add captions and when you add them, make sure that they correct, and make sure that they are slow, but they’re not too fast.

That will mean that there will be a benefit.

It won’t cost them much to do that, but it will be something that will increase the value of the product that they have tremendously, and it will be something that will provide a lot of benefits to a lot of people making use of that. For parents, I would say, just look at what’s out there, just do a little bit of research and see… If you control the types of things … We control what our children watch in any case, in terms of content, so if you can just maybe then extend that to saying let’s try and find things that have captions on, that have subtitles on, and then get them used to that idea that like your daughter said… That they will suddenly say:  “But we visited the house and there was something strange, there was no text at the bottom of the screen.” So that it becomes the exception not to have it on, cause that will benefit. Then we will start seeing benefits of this.

Laura, Learning Reading Hub:

Let’s talk about some projects you are involved with, so people can understand your background and your expertise. So, you are on the editorial board of the Journal of Audiovisual translation, you’re on the academic Advisory Board of the “Turn On the Subtitles” campaign, and this is how I personally discovered your work, and I think -and correct me if I’m wrong- you’re involved in the Billion Readers project -the BIRD project- that you mentioned, an initiative to increase literacy rates in India, which is a big problem, I believe. Like 900 million people are illiterate in India.

Anything else that you’d like to share about these projects with us? Any other relevant projects that you’re involved in?

Professor Jan-Louis Kruger:

Yeah, so the BIRD project (the Billion readers Project), we are definitely talking myself and my colleague Sishim Lau… We are speaking to them, and we are busy developing a small project where we can start collaborating and we want to start working with them much more extensively, so that’s definitely the one project that we are engaged in and in early stages at the moment.

We’re doing more fundamental work on just understanding how people read in multimodal contexts with eye tracking. What happens when there’s complicated things on screen that they have to process, what happens when theyhave sound or not sound.

I’m also a participant in, one of the researchers, on a big Polish grant or project called “Watch Me,” which is really just about understanding how subtitles are read, and we will be in Hamburgh in a couple of weeks doing presentations on a number of things there: if you have sound on and off, or depending on what your first language is, how do you read subtitles, the speed of subtitles, how does that influence the way we read it. So, those are some of the things that we are busy with at the moment.

There are things that we want to do.

And I’ve spoken to you about some of those to start looking at maybe putting together a project, where we work with parents to have an assessment of their child’s reading using eye tracking, using some of the standard measures, but then also looking at eye tracking search, you can see if you have…

If you show your child something to read, I’m not talking about subtitles necessarily, just normal text or subtitles. You can start looking at the eye movements and you can see: are there specific areas where they need more development? Even maybe just getting an initial idea of what it looks like at a particular age and, then, say six months later, or a year later or two years later, what has improved, what hasn’t, and then trying to find ways working with people who are much cleverer than I in terms of reading instruction to find ways to use that to diagnose where the deficits might be, and then to target those things to develop further.

So, if we do get a project like that up and running, I will definitely let you know, and we can maybe see if we can recruit some willing parents and their children to participate.

Laura, Learning Reading Hub:

Very exciting, very exciting.

The diagnosis, I had never thought of that.  Eye tracking for diagnosis. There’s a lot of potential. So many avenues to explore.

If people want to learn more about this topic, what do  you suggest? What’s the best place to go to?

Professor Jan-Louis Kruger:

I think your channel.

Laura, Learning Reading Hub:

Thank you!

And maybe the “Turn On the Subtitles” website?

Professor Jan-Louis Kruger:

That’s right. If you just Google “turnonthesubtitles,” just one word, turnonthesubtitles.org, you’ll get information there. The slogan they use is “Kids Read More When We Turn On the Subtitles,” so that’s fairly self-explanatory.

We also have another campaign that is called “captions on.” And that’s simply just captionson.org And their slogan is “Make Screen Time Reading Time.”

Laura, Learning Reading Hub:

Very good slogan.

Professor Jan-Louis Kruger:

And, then, the Billion Readers project. If your viewers can maybe just Google “billionreaders.org”, they’ll see there, in all three of these websites, that there are actually resources available. They explain the principles more and there’s some research on some of them, findings, and, then, they have some resources as well.

So, I think that’s a really good place to start.

The Macquarie University has a reading clinic that specializes in reading development, so there’s definitely something to explore there as well. That parents could maybe reach out, if they feel that their child has specific need for an intervention, then there are things that can be done. They sometimes have holiday programs, where you can use the creative kids’ vouchers, for example. Something like that.

Laura, Learning Reading Hub:

Perfect. Anything else that you’d like to add? Anything that I’ve missed? Anything else that is relevant?

Professor Jan-Louis Kruger:

No, I think that was fairly comprehensive. I think we’ve covered a lot of ground.

I’m quite passionate about subtitles, just  because I’ve been doing this for quite a long time,  but also because I just think it’s just a bit of a no-brainer. That there are so many benefits to be had from this, from having subtitles available.

And it’s a good thing maybe for people just to know why subtitles look the way they do sometimes, to know why there are differences sometimes, between what you hear and what you read, to know when to trust the subtitles, when not. If you can’t finish reading the subtitles before they disappear, switch them off because, they’re just going to frustrate you.

Laura, Learning Reading Hub:

Yeah, no, totally, It’s very, very frustrating.  Out of sync, horrible!

Professor Jan-Louis Kruger:

Out of sync. Yes, exactly.

Laura, Learning Reading Hub:

You cannot concentrate on the show, you cannot read the subtitles. It just doesn’t work. At least, in my experience.

Professor Jan-Louis Kruger:

That’s true, and that’s why I teach my students also that they have to try to make the subtitles invisible.

And that just means, if they’re easy to read people won’t notice that they’re there. You will read the subtitles as if you are listening to them with your eyes, if you want.

As soon as there’s something like the synchronization is out, somebody has already said something and there’s nothing on screen, or there’s something on screen before they’ve said anything… Or if the subtitle is too fast, or if there’s a spelling mistake, or if the line distribution is really odd, and it might take you a while to figure out what’s being said, if the punctuation is terrible… Those things will draw attention to the subtitles and will take your attention away from the screen.

And, let’s face it, when we watch audio visual content, we don’t want to do that to read. You want to watch it for the full experience. So, you want to be able to read the subtitles and look at the screen and not compromise either of those.

Laura, Learning Reading Hub:

Yes, totally. So, thank you so much for your precious time. It’s being a really insightful conversation  and a real pleasure to learn from you. I think our audience is going to love this topic, they are probably also gonna be surprised, because maybe this is not something they knew of. Not many people are talking about this… yet.

So, thank you so much.

Professor Jan-Louis Kruger:

Thank you very much for having me.

Laura, Learning Reading Hub:

Thanks.

Resources to explore further:

• Turn On the Subtitles campaign: https://turnonthesubtitles.org/
• Captions on: https://www.captionson.org/
• Planet Read: https://www.planetread.org
• The BIRD Project: https://billionreaders.org/
• Article mentioned in the interview: https://www.theatlantic.com/ideas/archive/2023/06/watching-movies-tv-with-subtitles/674301/
• Department of Linguistics at Macquarie University: https://www.mq.edu.au/about/about-the-university/our-faculties/medicine-and-health-sciences/departments-and-centres/department-of-linguistics
• Reading Clinic at Macquarie University: https://www.mq.edu.au/about/facilities/hospital-and-clinics/macquarie-university-reading-clinic