Wednesday, December 24, 2014

Laptop vs. IMAX: An eyetracking experiment

As part of BBC Radio 4's Cells and Celluloid Christmas special on the science of film I conducted an experiment investigating the impact screen size had on viewing behaviour. Using head-mounted eyetracking equipment (SMI Glasses) I recorded the eye movements of Francine Stock and Adam Rutherford as they watched Fly Me to the Moon 3D (2008) either in the Science Museum's IMAX or on a 13 inch laptop.

Full discussion of the experiment will be available on the BBC iPlayer here:

As you will see from the video, the IMAX screen filled the entire field of view of the head camera (>60 degrees) where as the laptop screen only filled less than half of the image even when it was viewed at a typical viewing distance (~60cm), This difference in viewing angle meant that the gaze had to explore more of the image in the IMAX presentation producing a greater number of saccades into the periphery. The difference in gaze exploration of the image can clearly be seen in the scatter plot below. All the gaze to the laptop (green crosses) is tightly centred relative to the head camera whereas the gaze during the IMAX viewing (blue circles) is higher (possibly due to the viewer being positioned in the back row of the auditorium) and covers much more of the image.

These data give us an initial indication of how the viewing behaviour between small screen film viewing and large format (such as IMAX) may differ but to check whether these results are robust we would need to conduct a more thorough empirical investigation using exactly the same section of a movie and a larger number of viewers. A lot has been claimed about the "immersive" effect of large-screen viewing and these sorts of studies can begin to identify exactly what these differences might be in terms of viewer experience. There have not been enough real-cinema studies for us to be able to draw strong conclusions but those studies that have been conducted indicate that even when the amount of visual field occupied by a screen and the amount of light reflected off the screen is equated between a home cinema presentation and a large screen (e.g. at the cinema) our knowledge that the screen is physically larger creates an increased experience of immersion (Troscianko, Meese & Hinde, 2012). More research is needed to identify exactly what aspects create the "cinematic experience" and to predict what the future holds for this experience given the increased competition for our attention.

Thursday, June 26, 2014

EPSRC 3.5 year PhD studentship: Eye guidance in real-world scenes (UK students only)

Can't recommend this opportunity more highly. This is a chance to work with two pioneers of active vision (Nuthmann) and computer vision (Fisher) in one of my favourite cities on the planet!

Dear students,

A (last-minute) fully funded(!) PhD studentship is available to work with Dr. Antje Nuthmann (principal supervisor) and Prof. Bob Fisher (co-supervisor) at the University of Edinburgh. 

Details can be found here:

Please don't hesitate to contact me if you have any questions.

Best wishes,
Antje Nuthmann

Sunday, November 03, 2013

BIMI study day: Cognition at the Movies

I am proud to announce the first 'Cognition at the Movies' study day hosted by myself and Birkbeck Institute for the Moving Image (BIMI).

Saturday November 9th 10am to 6:30pm (B35, Birkbeck, Malet Street, London, WC1E 7HX; number 1 on the map

Since cinema’s inception filmmakers and theorists have been interested in the relationship between film and its audience. How do directorial decisions influence what we see on the screen and how does a viewer’s prior beliefs and interests influence how they experience a film? Cognitive Science, the interdisciplinary investigation of mental phenomena using theories and techniques from neuroscience, psychology and philosophy has recently begun to be applied to these questions of film cognition. This workshop will bring together film theorists, cognitive psychologists and philosophers in an exploration of the relationship between film and its audience.

Keynote presentation ‘by Prof Torben Grodal (Copenhagen) author of Moving Pictures and Embodied Visions.

Comic Entertainment, Film, and the Embodied Brain
The lecture will first provide a short description of how muscles and action is important for the embodied brain and for our experience of narratives.  The basis for the standard narrative reflects the Brain’s PECMA flow: Perception, Emotion, Cognition, and Motor Action. Characters and viewers want to modify some states of the world by motor action, including verbal actions. The lecture will then discuss the embodied brain’s three ’bail out’ mechanisms where the modification of the world by action is supplanted with self-modification: Crying, as in sad melodramas, laughter, as in comedies, and freeze reactions as effects of sublime submission to the exterior world. The lecture will especially focus on comic entertainment and discuss the processes that allows the brain to evaluate something as ’not real’, as ’not a cause for action’ and redirect the arousal from a given scene from tense world-directedness to laughter. It will finally discuss the social nature of comic entertainment and those mammalian play-functions that serve as facilitators for the reality status evaluations in comic entertainment that makes it possible to experience shame, failure and other negative events with a strongly positive hedonic tone.

Event is free but please register as spaces are limited:


9:30-10 Registration

10-10:15 Welcome and Introduction
10:15-11 Prof. Ian Christie (Birkbeck) - Psychology in the dark: just what is it we want to know?
11-11:45 Dr. William Brown (Roehampton) - He(u)retical Film Theory: Cinema and the Brain
11:45-12:30 Prof. Sheena Rogers (James Mason U.) - Towards Transcendence: Cognitive Components of the Sublime in Art

12:30-1:30 Lunch break

1:30-2:30 KEYNOTE: Prof. Torben Grodal (Copenhagen) - Comic Entertainment, Film, and the Embodied Brain
2:30-3:15 Dr. Paul Taberham  (Kent-Canterbury) - Avant-Garde Film in an Evolutionary Context
3:15-4:00 Prof. Murray Smith (Kent-Canterbury) - Murder Ballads

4-4:15   Coffee break

4:15-35 Steve Hinde  (Bristol) - A Study of Attention  while People Watch Movies.
4:35-5 Parag K. Mital (Goldsmiths) - Resynthesizing Perception
5-5:45 Dr. Tim J. Smith  (Birkbeck) - Cinematic Universality: Do I see the same movie you see?
5:45 - 6:30 Discussion

6:30 + Wine reception

For further information email 

Tuesday, October 01, 2013

Where’s Walter? How the finale of Breaking Bad used your eye movements to build suspense.

Where’s Walter? How the finale of Breaking Bad used your eye movements to build suspense.
By Dr. Tim J. Smith and Rebecca Nako

WARNING: If you are a fan of Breaking Bad and have not yet watched the finale do not read on. Spoilers ahead!

So it is all over. The season finale of Breaking Bad has been eagerly devoured by fans in America and across the world. We now know the fate of the anti-hero, meth-cook extraordinaire, Walter White and the extended group of characters both good and evil. The final episode is a masterful end to an exceptionally crafted series that has always found the perfect balance of intensity, humour and nuanced plot. It also serves as a wonderful demonstration of how the loyal Breaking Bad viewers have often been complicit in the creation of the tension. As Walt’s character develops across the series he becomes a menacing figure who’s actions are less and less predictable. Unlike some of the gangster characters he encounters (and usually defeats) he is not prone to irrational outbursts or  sudden violence. Walt’s menace comes from his intellect and cool planning. We know his actions are morally wrong but throughout the series we continue to empathise for Walt and see the action from his perspective, rooting for him to succeed. This is perfectly exemplified in the season finale in which the action builds slowly to an ultra-violent crescendo in which Walt’s ingenuity triumphs over those who have wronged him and his family. The slow scenes that build to this climax show us how he first gets his own back on the Schwartz’s, the couple who he believes profited off his early ideas, Lydia, Walt’s meth distributor, and then says goodbye to his wife, Skyler. Each scene is incredibly tense to watch but, unlike most mainstream TV or Cinema the tension is created by the viewers, not typical cinematic devices.  The action is often framed in shots that linger on the screen for an uncharacteristically long amount of time for TV. Hidden within the shot, unbeknownst to either the viewer, the characters or both, is the menacing figure of Walt. The director is creating tension by playing a game of ‘Where’s Walter?’ with the viewer.

For example, in the sequence in which Walt confronts Lydia, Walt seems to appear from nowhere in the café after Todd, the new meth cook arrives. Walt’s appearance behind Lydia and Todd shocks the audience but not through a traditional use of sudden cut to close-up or dramatic change in accompanying music. The shock comes from the viewer surprise that they didn’t notice him in the scene earlier. However, if we look back at two earlier shots of the café, Walt can clearly be seen sitting off to the side.

When we first watch this sequence our interest is in Lydia and her conversation with Todd. Due to physiological limits in what we can attend to and see at any one moment we have to choose where to fixate in the scene. By predicting our interest in Lydia, the director (Vince Gilligan, the creator of the series) ensures that our eyes linger near her and do not locate Walt in the periphery. There are many techniques for influencing where people fixate in a film, as I have discussed in length elsewhere (Smith, Psychocinematics, 2013) but one of the strongest is to make the viewer want to look somewhere. If the viewer is complicit in their choice of fixation location they will be even more surprised when it is revealed that they failed to see something. This is a technique of subtle misdirection that magicians have used for centuries and we have recently shown can operate in magic tricks even when visual cues are used to try and force the viewer to look at the source of the trick (Smith, Lamont, & Henderson, Perception, in press). In this scene from Breaking Bad, the director uses this inattentional blindness to play with the viewer and reward the active viewer who discovers Walt before Lydia and Todd do. Along with this episodes dense use of back-references to earlier plot points, subtle character cues and symbols (such as Jesse’s box), this game of hide-and-seek with Walt serves to reward the committed viewer with a sense of discovery and enrichment of what will be their final glimpse of this world.

The impact of this knowledge on how people watch this scene is evident if you record their eye movements. Using a Tobii TX-60 eyetracker, I recorded the eye movements of two participants watching the café scene. One participant had never seen Breaking Bad before (Yes, I ruined the whole of Breaking Bad for her by showing her the finale!). The other participant was an avid fan who had already watched the finale the night before. If we visualise their eye movements as red dots on top of the video (see below) we can see how their eyes and their attention shift across the screen. Each red dot signifies the location of the viewer’s gaze during one sample of the eyetracker (1/60th of a second). When the gaze clusters together in one place their eyes are in a fixation. When the gaze suddenly jumps to a new location they are performing a saccade.

At the beginning of the clip we can see how both the experienced (bottom video) and novice viewer (top video) track Lydia’s bag as she drags it through the café and then saccade up to her body and face once she sits down in the next shot. If we were to plot the two gaze patterns on top of each other we would see a striking degree of coordination between the two viewers. This synchronisation of attention across viewers is characteristic of how we watch most TV and film. Although we think we are highly idiosyncratic in how we watch a program most of the time the director is ensuring we all look in the same place at the same time as I demonstrated by eyetracking multiple viewers watching There Will Be Blood here and here.

After Lydia is seated at the table the camera then cuts across the table to reveal the rest of the café to Lydia’s left. Immediately following the cut, the experienced viewer saccades directly to Walt seated in the background. The novice viewer only looks at that part of the frame once the waiter enters the shot and blocks our view of Walt. After watching this clip the experienced viewer stated that he had not known Walt was in this shot until watching the scene during the experiment. His direct saccade to Walt suggests that knowing Walt would appear in the scene at some point had primed his attention and made it easier for him to find him earlier than the novice viewer.

The camera then cuts to Lydia and a series of close-ups of her face and the Stevia sweetener which will later play a critical role in the scene. We then cut back to a longer shot of the café in which Walt is now lurking discretely. At this point both viewers saccade directly to him  even though he hasn’t yet moved and the main action is still taking place in the rear of the shot. Both viewers now know Walt is present in the scene and about to approach Lydia and Todd who are still unaware of this presence. This mismatch between what the viewer knows and what the characters know creates tension about what will happen next.

 An even more impressive use of camera positioning to create tension occurs later in the episode when we overhear a phone conversation between Skyler and her sister, Marie. The sequence begins with a slow camera pan across Skyler’s new apartment as the phone rings and the answerphone picks it up. We see Skyler smoking at the kitchen table. Our view of the kitchen is complete except for a small patch occluded by a column at screen centre (image above). The slow pan of the room and this final long shot suggests that Walt is not in the scene. However, after several cuts back and forth between Skyler and Marie as Marie informs Skyler of Walt’s presence in town the viewer begins to get the impression that Walt may be either coming for Skyler or already be hiding somewhere in the scene. After Skyler hangs up, the camera cuts back to the earlier long shot and we again see that the scene is empty. This belief is trashed as the camera slowly moves into the scene and Walt is revealed behind the column. We were denied knowledge of Walt’s presence in the scene by the director due to the clever choice of camera position. The tension we feel after discovering Walt’s presence is due to the mismatch between what we have known up until that point and what Skyler must have known all along: Walt is present in the room. What will Walt do next and why is Skyler hiding his presence from Marie? Our sudden awareness of Walt creates a flurry of questions and an interest in how the scene develops.

Watching the eye movements of our two participants viewing this scene reveals the strong influence knowledge of Walt’s presence has on our experienced viewer’s eye movements. Both viewers begin the scene by saccading around the apartment, fixating and tracking objects as they are revealed by the panning camera. As soon as the column behind which Walt is hiding comes into view the experienced viewer (bottom video) becomes obsessed with this boring piece of architecture. His gaze dwells on the column and saccades around its edges, trying to find some evidence of Walt. By comparison, the novice viewer saccades directly to Skyler and focuses on the phone conversation.

After the conversation finishes and the camera cuts back to the long-shot (02:19) the experienced viewer immediately saccades to the column and then saccades back-and-forth between the column and Skyler, waiting for Walt’s reveal. The novice viewer glances briefly at the column but mostly concentrates on Skyler. It is only once the camera begins moving in that her attention to the column increases and finally peaks once she catches a glimpse of Walt’s jacket sticking out behind the column. The novice viewer is actively viewing the scene; trying to check that Walt isn’t present given the suspicion Marie has just created but she can only see what is visibly present in front of her. The experienced viewer perceives Walt behind the column in the very first shot due to his memory from previously watching the scene. The experienced viewer’s gaze interrogates the column seeking out confirmation of Walt’s presence even though for the majority of the scene all  you can see is a bland wood column.

These example scenes demonstrate how film and TV can create suspense by withholding information from either the viewer (e.g. Walt’s presence in the kitchen with Skyler), the characters (e.g. Lydia and Todd’s knowledge of Walt’s presence in the café) or both. Often such suspense is created by not cutting to a detail that we desperately want to see. However, such techniques can often appear heavy handed and position the director at odds with the viewer. In the scenes discussed above the director cleverly plays around with what the viewer can and cannot see whilst always giving us the impression that we have access to the full scene. This false belief makes Walt’s eventual reveal all the more powerful. This is further evidence for why Breaking Bad was such exquisite TV.

*If you are interested in seeing more examples of how our expectations about a dynamic scene can influence where we look check out my recent study published in Perception. This study used a simple card trick to bias participant’s gaze towards one part of the screen whilst the trick occurred in plain sight elsewhere on the screen.  Eye tracking revealed that participants completely fail to look at the location of the trick during the first viewing due to their own belief about what is relevant. During a second viewing all participants look in the right place and see how the trick worked. These findings (and the Breaking Bad examples above) are completely at odds with most current theories of how attention is guided in dynamic scenes which state that basic visual features such as motion guide attention (see my article on the topic here; Smith & Mital, JoV, 2013).

** CAVEAT: The two participants tested above may be extreme examples of how a novice and experienced viewer might watch these sequences and a full empirical study would require a larger sample of participants in each group. Effects such as the bias of the experienced viewer’s gaze to the column are unlikely to be absolute but may prove to be statistically significant if the gaze was quantified  across more participants. The precision of the eyetracking, the synch of the audio during playback and the image quality is also not adequate for a full empirical study (hence why the gaze sometimes seems to be offset from objects in the scene). However, this quick and dirty demonstration allows us to quickly analyse the scenes whilst the episode is still fresh in people’s minds.

Wednesday, July 17, 2013

Melbourne's Eye Tracking and Moving Image Research Group

The application of eye tracking technology to questions of moving image spectatorship has risen in popularity over the last couple of years. Aided by the dropping cost of eyetracking hardware and the ease of use of presentation and analysis software eyetracking is becoming practical for researchers across a broad range of disciplines.

For example, the recently formed Eye Tracking and Moving Image Research Group lead by Sean Redmond and Jodi Sita in Melbourne has two central goals:

" bringing the group together; we wanted to utilise eye tracking technology more centrally in the analysis and examination of the moving image; and we wanted to draw together scholars and practitioners from the Sciences, and the (Creative) Arts and Humanities so that different modes of enquiry and theoretical and methodological apparatus were placed in the same analytical arena."

I very much welcome groups like this and hope their research proves fruitful and informative. Only by spreading the research questions across a broad range of researchers can we hope to tackle the complex questions of spectatorship and by sharing research methods/techniques we can avoid each of us reinventing the 'eyetracking and film' wheel. To that end I hope my recent publications on the topic can serve as a useful starting point for researchers beginning to apply eye tracking to these questions:

  • Smith, T. J. (2013) Watching you watch movies: Using eye tracking to inform cognitive film theory. In A. P. Shimamura (Ed.),Psychocinematics: Exploring Cognition at the Movies. New York: Oxford University Press. (pdf)
  • Smith, T. J. (2012) The Attentional Theory of Cinematic Continuity, Projections: The Journal for Movies and the Mind. 6(1), 1-27. (pdf)

Thursday, June 06, 2013

Reaction videos capture the heart (and twisted soul) of Film/TV

Videos of viewer reactions to film and TV have been cropping up on Youtube for several years and I have always been fascinated with how they capture what, for me is the heart of the the cinematic experience: the power to manipulate our minds and emotions. Watching a grown adult shriek, cry, and laugh in response to artificial patterns of light and sound on a screen demonstrates the power film and TV have over us and why we seek it out as a source of entertainment, escapism, and emotion that in everyday life would be considered too extreme or dangerous.

I've been meaning to post on this topic for a while but had to act this week in response to the TV event that was the Game Of Thrones 'Red Wedding' (series 3, episode 9). For fans of the books, the events of this episode came as no surprise as they had already been traumatised by the shocking and brutal events when reading the third book in the series, 'The Storm of Swords'. *spoilers* For the fans of the TV series, subtle manipulation of plot events and characters by the producers of the series meant that fans of the show had no way of knowing the duplicitous nature of Walder Frey and what was about to happen in the episode. As the Stark's celebrate the wedding of Edmure Tully to one of Walder Frey's daughters they are brutally slaughtered by the Frey's and the Bolton's, swarn bannermen of the Starks who have secretly made an allegiance with their enemies, the Lannisters. Three of the main characters of the series are brutally killed in a matter of minutes: Robb Stark (the King In the North), his mother, Catelyn and Robb's new bride, Talisa who was pregnant with their first child. The brutality and graphic nature of the murders is shocking even to readers of the books who knew what was going to happen. For viewers of the series without prior knowledge of the events, the murders were..... well judge for yourself *end of spoilers*

The facial reactions, gasps, screams, and comments directed to the screen and to the person filming (who had obviously read the books and knew what was going to happen, filming their friend's/partner's/parent's reactions with sadistic glee) show how involved they are in the series and the characters. For a brief moment at least, they are as moved by the deaths of these fictitious characters as they are to the news of a friend's death. Emotionally, they are across the fourth wall.

Some other classic reaction videos are the reaction of children to the reveal at the end of Empire Strikes Back that Darth Vader is Luke Skywalker's father:

Not surprisingly most of the videos that are used to elicit these shocking reactions are either horrific, shocking, pornographic or sometimes all three at the same time. This brings me to the reaction videos that started the whole craze: People's reactions to '2 Girls 1 Cup'. I'm not going to say anything about the original video as it is too vile to mention but there are several reaction videos on-line from which you can infer the general events of the stimulus. Here are selection:

The whole gamut of human emotions are right there in those videos.

Thursday, March 21, 2013

Film reflected in the human eye

Simply beautiful!

I could spend days watching the human eye move in response to film.....Oh, that's right, I do!