An fMRI study by a group from UC San Diego led by Ayse Pinar Saygin has investigated the phenomenon known as "The Uncanny Valley": that eerie feeling you get when watching a robot or CG animation that is attempting to be photorealistic but fails. we have all experienced this phenomenon when watching films such as Final Fantasy, or Robert Zemeckis' performance capture pieces, The Polar Express, Beowulf, A Christmas Carol and the recent film that sounded the death-knell for Zemeckis' studio, Mars Needs Moms. Given the popularity of using performance capture and even facial expression capture in films like Avatar and in the upcoming Tintin, there is a lot of interest in how to present naturalistic motion capture animation without wandering into the Uncanny Valley. The results of this new study seem to suggest that the discomfort comes from the perceptual mismatch between the authentic human motion and the inadequate appearance. We are highly sensitive to human appearance and motion and have more brain areas dedicated to the processing of these features than any other visual category. Motion capture gives us the ability to trick the brain into seeing human motion even in the absence of the corresponding appearance. This is clearly displayed in point-light walkers (check out this fun interactive demo http://www.biomotionlab.ca/Demos/BMLwalker.html). However, when the authentic biological motion is combined with an appearance that doesn't match the authenticity of the motion it results in cognitive dissonance. This new study shows the brains response in such a situation.
I am very intrigued to see how Steven Spielberg and Peter Jackson deal with this issue in the upcoming Tintin movies. In the first teaser trailer he avoided showing faces, perhaps to avoid the audience's negative response to seeing an Uncanny Tintin. In the first full trailer (below) the faces walk an interesting line between naturalism and cartoon, very accurately capturing the character of the original Herge cartoon. Perhaps Spielberg has dodged the bullet by mismatching the motion and appearance enough to avoid cognitive dissonance. We'll have to wait until Christmas to find out.
From the press release:
"[Image: Brain response as measured by fMRI to videos of a robot, android and human]
Your Brain on Androids
July 14, 2011 By Inga Kiderra
Ever get the heebie-jeebies at a wax museum? Feel uneasy with an anthropomorphic robot? What about playing a video game or watching an animated movie, where the human characters are pretty realistic but just not quite right and maybe a bit creepy? If yes, then you’ve probably been a visitor to what’s called the “uncanny valley.”
T! he phenomenon has been described anecdotally for years, but how and why this happens is still a subject of debate in robotics, computer graphics and neuroscience. Now an international team of researchers, led by Ayse Pinar Saygin of the University of California, San Diego, has taken a peek inside the brains of people viewing videos of an uncanny android (compared to videos of a human and a robot-looking robot).
Published in the Oxford University Press journal Social Cognitive and Affective Neuroscience, the functional MRI study suggests that what may be going on is due to a perceptual mismatch between appearance and motion.
The term “uncanny valley” refers to an artificial agent’s drop in likeability when it becomes too humanlike. People respond positively to an agent that shares some characteristics with humans – think dolls, cartoon animals, R2D2. As the agent becomes more human-like, it becomes more likeable. But at some point! that upward trajectory stops and instead the agent is perceived as strange and disconcerting. Many viewers, for example, find the characters in the animated film “Polar Express” to be off-putting. And most modern androids, including the Japanese Repliee Q2 used in the study here, are also thought to fall into the uncanny valley.
Saygin and her colleagues set out to discover if what they call the “action perception system” in the human brain is tuned more to human appearance or human motion, with the general goal, they write, “of identifying the functional properties of brain systems that allow us to understand others’ body movements and actions.”
They tested 20 subjects aged 20 to 36 who had no experience working with robots and hadn’t spent time in Japan, where there’s potentially more cultural exposure to and acceptance of androids, or even had friends or family from Japan.
The subjects were shown 12 videos of Repliee Q2 performing such o! rdinary actions as waving, nodding, taking a drink of water and picking up a piece of paper from a table. They were also shown videos of the same actions performed by the human on whom the android was modeled and by a stripped version of the android – skinned to its underlying metal joints and wiring, revealing its mechanics until it could no longer be mistaken for a human. That is, they set up three conditions: a human with biological appearance and movement; a robot with mechanical appearance and mechanical motion; and a human-seeming agent with the exact same mechanical movement as the robot.
At the start of the experiment, the subjects were shown each of the videos outside the fMRI scanner and were informed about which was a robot and which human.
The biggest difference in brain response the researchers noticed was during the android condition – in the parietal cortex, on both sides of the brain, specifically in the areas that connect the part of the brain! s visual cortex that processes bodily movements with the section of the motor cortex thought to contain mirror neurons (neurons also known as “monkey-see, monkey-do neurons” or “empathy neurons”).
According to their interpretation of the fMRI results, the researchers say they saw, in essence, evidence of mismatch. The brain “lit up” when the human-like appearance of the android and its robotic motion “didn’t compute.”
“The brain doesn’t seem tuned to care about either biological appearance or biological motion per se,” said Saygin, an assistant professor of cognitive science at UC San Diego and alumna of the same department. “What it seems to be doing is looking for its expectations to be met – for appearance and motion to be congruent.”
In other words, if it looks human and moves likes a human, we are OK with that. If it looks like a robot and acts like a robot, we are OK with that, too; our brains have no difficulty processin! g the information. The trouble arises when – contrary to a lifetime of expectations – appearance and motion are at odds.
“As human-like artificial agents become more commonplace, perhaps our perceptual systems will be re-tuned to accommodate these new social partners,” the researchers write. “Or perhaps, we will decide it is not a good idea to make them so closely in our image after all.”
Saygin thinks it’s “not so crazy to suggest we brain-test-drive robots or animated characters before spending millions of dollars on their development.”
It’s not too practical, though, to do these test-drives in expensive and hard-to-come-by fMRI scanners. So Saygin and her students are currently on the hunt for an analogous EEG signal. EEG technology is cheap enough that the electrode caps are being developed for home use.
The research was funded by the Kavli Institute for Brain and Mind at UC San Diego. Saygin was additionally supported by the Cal! ifornia Institute of Telecommunication and Information Technology (Calit2) at UCSD.
Saygin’s coauthors are Thierry Chaminade of Mediterranean Institute for Cognitive Neuroscience, France; Hiroshi Ishiguro of Osaka University and ATR, Japan; Jon Driver of University College London; and Chris Firth of University of Aarhus, Denmark.
In the aftermath of the recent cuts arguments about arts funding are becoming increasingly heated, yet crucial discussions as to the value and place of art in our world are distinguished by their absence. As the wider society experiences the kind of structural economic changes unseen in the UK for sixty years, the time has never been riper for a serious investigation of the role of art in our lives, and of its relationship with the individual, the state and the market.
For 2011, in collaboration with University College London, we are developing a sequence of six events on the theme ‘what is the point of art?’, each featuring a single speaker, to take place in London on dates throughout the year. The aim is to have accumulated by the end a compelling portfolio of perspectives on the value of art in our society.
How we perceive spatiotemporal continuity across edited sequences of film is clearly a major interest of mine (hence the blog name!). Several theorists have written about this topic going all the way back to Munsterberg (1916), Hochberg & Brooks in the 70s and early 80s, and the pioneering Change Blindness studies of Levin & Simons in the 90s. In my thesis I discussed in depth the paradox of perceiving continuous visual scenes across visually discontinuous shots. For instance, why is a series of shots all filmed from the same side of an action such as a conversation perceived as being spatially continuous where as a shot that crosses to the other side of the action leads to spatial confusion? This filming and editing convention is known as the 180 Degree Rule. You can see clear demonstrations of it in this video:
Now a recent paper on the perception of continuity in film by Berliner and Cohen (2011) wonderfully brings together psychological evidence and film theory to provide an accessible and insightful overview of the topic. I highly recommend this article if you are looking for a quick reader on continuity perception.
Berliner and Cohen (2011) clearly outline the psychological motivation for the continuity editing rules but also acknowledge the role of intuition and creativity of filmmakers in choosing the right techniques to convey their particular story.
"Continuity conventions have remained relatively stable for about ninety years. The primary reason for their stability is not, as some scholars think, Hollywood’s marketing dominance or other externalities but rather that the early filmmakers who first developed the conventions were guided by their intuitive understanding of space perception and the reactions of cinema spectators. Just as expert pool players learn— not through direct study but intuitively, through trial and error—the principles of Newtonian physics that govern pool playing, as well as matter and energy generally, the filmmakers in the early twentieth century who first developed the conventions of the classical editing system, without directly studying psychology, discovered the structure of human perception."
Berliner, T. & Cohen, D. J. (2011) The Illusion of Continuity: Active Perception and the Classical Editing System.”Journal of Film and Video 63.1: 44-63.
I had a great conversation with Michael Bhardwaj, the science correspondent for CBC radio Canada last week about eyetracking and film, the There Will Be Blood research, and my visit to Dreamworks Animation. Michael did a great job of describing my work on his radio show last Friday (10/06/11) and was kind enough to send me an mp3.
I was very fortunate to have the opportunity to discuss my research with Alok Jha on the Guardian Science Weekly podcast. If you would like to hear my views on visual attention, eye movements, the film experience and how it relates to magic check out the podcast:
My guest post on David Bordwell's blog last week was a roaring success. I could never have imagined that it would capture the interest of so many people across so many disciplines. You can get a sense of the interest by looking at the comments and statistics for the main eye movement video. In the week the post has been up it has been viewed 145,000 times and pages embedding the video have been read 728,000 times! The video cropped up on twitter (thanks Roger Ebert and others), facebook, numerous blogs, websites and newspapers. I am incredibly happy that my research reached out to film makers, theorists, and eager consumers to inform their appreciation of film. Hopefully, you can all now get a sense of how miraculous and complex our perception of film is and how we can inform our understanding by applying methods from empirical psychology.
I plan to build on the momentum created by the blog post by posting similar cognitive readings of films here on my own blog. In the meantime, I can point you to my existing publications on the topic:
For information on the Dynamic Images and Eye Movement project (DIEM) and its analysis of the influence of visual and cinematic features on how we watch movies as presented in my analysis of There Will Be Blood, check out:
Finally, if you want to see more of the DIEM eye movement videos, new videos as they are created and download the analysis software (i.e. CARPE) go to the DIEM project page and subscribe to our Vimeo channel. As a taster, here is a showreel from the DIEM videos. Enjoy!
As mentioned in my last post, David Bordwell asked me to write a guest post on his much read blog. Skip over to davidbordwell.net to see my overview of how recording eye movements during film viewing can inform our understanding of how we watch and make sense of films. As an example I analysed a sequence from PT Anderson's There Will Be Blood and visualised the gaze data using tools created as part of the Dynamic Images and Eye Movements (DIEM) project. To give you a sneak preview, here is a "peekthrough heatmap" made from the gaze data of 11 viewers.
Details about how this was created and what it tells us can be found on the blog post.
Cognitive Film Theorist extraordinaire, David Bordwell has recently posted two wonderful blog posts about the power of eyes in film. His first post on emotional communication and acting in Fincher's The Social Network is a brilliant introduction to the psychology of facial expressions and its subtle mastery by the actors. A very fun and clever exercise in how we can use cognitive psychology to inform our understanding and analysis of film.
By providing an introduction to eye movements and summarising Yarbus' work on the influence of task on eye movements during picture viewing he provides the foundations from which we can start hypothesising about the relationship between film form and viewer attention. In next week's post, David has given me the honour of building on this foundation by explaining first-hand some of my empirical research into film viewing utilising my eyetracking methods. Watch this space to find out more!
BBC TWO in the UK broadcast a wonderful factual programme on sensory illusions (Horizon, 9pm, 18/10/10). The programme covered many well known sensory illusions such as the McGurk effect, multimodal illusions such as the effect of sound and colour on taste perception, magic tricks such as Gustav Kuhn's disappearing ball illusion, and the perceptual completion of impossible figures. If you are interested in sensory illusions the programme provides a nice, succinct introduction. The programme also goes on to talk about the science of illusions and how illusions allow us to understand the cognitive processes and neurology of perception.
The programme also introduced the phenomenon of Synaesthesia in which perception through one modality (e.g. vision) triggers perceptual experiences in another modality (e.g. taste). We all have some synaesthetic experiences such as smells triggering specific visual memories but for people with more developed synaesthesia these cross-modal percepts are richer and more complex. Synaesthesia is a fascinating research area which up until a few years ago was dismissed by the Psychological research community as hokum. Thankfully a new generation of researchers such as Jules Simner (Edinburgh) have used clever behavioural measures combined with neuroimaging to show that synaeasthesia is a real phenomena and one which can seriously advance our understanding of sensory perception.
Viewers in the UK can catch the programme on the BBC iPlayer:
After 13 years in Edinburgh, a BSc. in Artificial Intelligence and Psychology, a PhD in Cognitive Science, and a couple of post-docs I have finally left! 13 years of continuity have come to an end. This sudden discontinuity has been brought about by my move to Birkbeck, University of London to take up my first lectureship in Psychology. Thank you to everybody who made my time in Edinburgh so enjoyable, my friends, colleagues, collaborators, and students. And thank you Birkbeck for offering me this big break just when I needed it.
My new contact details are :
Tim J. Smith, Ph.D. Lecturer Rm 501d Department of Psychological Sciences Birkbeck, University of London Malet Street, London WC1E 7HX
Don't worry, this does not signify the end of 'Continuity Boy'. Far from it. I will continue updating this research blog and I look forward to regaling you with details of the research coming out of my lab.
A great opportunity to work with/learn from one of the great minds in Cognitive Film Theory:
>>> The Amsterdam School for Communications Research currently have a four year PhD vacancy.
We are looking for a researcher who has knowledge of (cognitive) theories of narrative and film and wants to study experiential states in film viewers. Experience in psychological experimentation is also absolutely necessary. Interest in manufacturing film materials for the experiments is a recommendation.
A scholarship sufficient for costs of living and tuition is related to the position.
Please pass this announcement on to potential candidates. Deadline for applications is Sept. 12 2010. For details, applications and all correspondence see:
Along with cinema there is another medium that is obsessed with continuity: comics. The universes created by comic giants DC and Marvel have strictly enforced continuity so that super heroes like Superman can pop up in other characters plot arcs such as Batman and all actions fit within their own overarching stories.
The lengths that fans, writers, artists, and licence holders go to to maintain continuity within comic universes (along with the related Sci-Fi and fantasy universes such as Star Trek, Star Wars, and Doctor Who) is remarkable. The recent spate of Marvel movies including Iron Man, The Hulk, and next year's Thor have shown how universe continuity can be maintained in cinema and create innovative sequels/crossovers. Whether or not they work for the general public is yet to be seen. I personally love the Avengers super-hero team but is the world really ready for cinematic versions of AntMan or the Wasp?
I guess we'll find out over the next few years but for now lets enjoy a musical ribbing of the whole phenomenon.
Greetings from the Cannes Film Festival 2010! After years of eagerly reading about all the gossip, film news, and screenings at Cannes I finally got the opportunity to attend the festival. Friends Calum Waddell and Naomi Holwill from new Edinburgh based production company, High Rising Productions (http://www.highrisingproductions.com/) asked me to join them on their Cannes experience and help out promoting some new film projects. How could I resist!
My first impressions of Cannes have been surprise and awe. You really get a sense of how much film is an industry when you are wandering around the film market and attending the market screenings. The glossy public face of the festival, the Official Competition is really just the polished face of what is really a massive buying and selling market for films. Cannes is completely mis-labelled as a festival as only industry representatives or press can get access to the 'festival' and unless you are pretty high up in the industry most of the competition and red carpet screenings will be out of bounds. Most people attending the festival spend their time in the many tiny impromptu screening rooms scattered around hotels or the palais. This year there are over 1400 screenings during the two weeks with 40 cinemas showing films simultaneously!!! Most of the films will be receiving their world premiers here at Cannes and almost all of them are completely unknown commodities. There is a great chance of stumbling across some real gems that may not make it to your local multiplex for a couple more years. Unfortunately, there is a much greater chance of sitting through some absolute dross :)
Whilst I'm here I'll try to post some reviews as I go along. Lets kick off with a couple of British gems and a few stinkers.
2B, directed by Richard Koehling.
I have to confess to being a bit of a Sci-Fi geek (was it not evident from this blog already!) and loving all things AI. The synopsis for this film got me excited: set in a near future New York, a brilliant scientist is killed by the first post-human that he personally created and attempts to transfer his mind into a new body so that he can live forever. The premise is good, if rather familiar and could have made an intriguing, contemplative sci-fi film. Sadly the budget appeared to be micro and while containing some recognisable faces from US TV, Kevin Corrigan (Fringe) and James Remar (Sex and the City) there was either too little budget to afford action sequences of no inclination to include them. Instead the film consists of a series of philosophical debates on the nature of human life, mind and soul. The ideas are good but the execution is without vision and ultimately tedious.
Edinburgh Informatics department, where I studied Artificial Intelligence has a DVD library devoted to all things vaguely related to AI, cybernetics and robots. 2B would definitely make it into their DVD library. I doubt it will make it into anybody elses though.
Redline by Takeshi Koike
Redline is for those of you who love your Japanese animation loud, fast, and intense. After watching this cinematic hit of amphetamines I felt like I needed a lie down! Koike, the director of the Animatrix section 'World Record' has crafted future universe populated by bizarre humanoid characters and an intense form of car racing where anything goes. The story follows a racer, Sweet JP who is notorious for race fixing. After almost dying in a race he unexpectedly finds himself called up for the Universe's most intense race, the Redline which is to be held illegally on the hostile Roboworld. As he prepares for the race a romance blossoms between JP and fellow racer, Sonosee a typical manga heroin with massive eyes, disproportionate breasts, a school girl innocence and a huge car. The action culminates in the spectacular Redline race in which Wacky Races-esque competitors jostle for position whilst also fighting to stay alive against missile attacks from the Roboworld denizens, mecha-suited warriors, giant gelatinous monsters and huge space lasers.
This film is an intense shot of traditional cell animation with a huge dose of Japanese surrealism. Highly fun if you have ADHD but possibly a bit too intense for the general viewing public. Not to be recommended if you suffer from an existing heart condition.
My research on eye movement behaviour during film viewing and its relationship to editing style is mentioned (briefly) in the print edition of New Scientist this week (issue 2748) and on-line here
The New Scientist article is in response to James Cutting's (Cornell) recent Psychological Science article on the 1/f structure in film editing. The scale of James' study is unbelievable. He and his collaborators basically created their own handcoded cinemetrics (http://www.cinemetrics.lv) database identifying every cut in 150 films! They then used this database to perform incredibly clever pattern analysis to look for repeating structures in adjacent shot lengths. They found that over the last 70 years Hollywood has been evolving towards a 1/f structure. I'll leave the technical explanation of what this means to James (below) but the basic gist is that shot lengths have begun to cluster in sequences of localised repeating lengths. Similar patterns have been observed in all aspects of nature such as tides, music, dance, the spatial frequency of the visual world, and neural firing rates. 1/f is thought by some to be a primitive pattern of processing in the human brain and influences attention fluctuation. James hypothesises that the adoption of a 1/f structure in editing structure may enable film to synchronise with viewer attention and create a more harmonious viewing experience. Of course, given that this was also the hypothesis I put forward in my Ph.D. thesis I am overjoyed by James' finding :) Let's hope this ushers in a new era of mathematically advanced cinemetrics and introduces more people to the area of Cognitive Film Theory.
Attention and the Evolution of Hollywood Film
James E. Cutting, Jordan E. DeLong, and Christine E. Nothelfer
Reaction times exhibit a spectral patterning known as 1/f, and these patterns can be thought of as reflecting time-varying changes in attention. We investigated the shot structure of Hollywood films to determine if these same patterns are found. We parsed 150 films with release dates from 1935 to 2005 into their sequences of shots and then analyzed the pattern of shot lengths in each film. Autoregressive and power analyses showed that, across that span of 70 years, shots became increasingly more correlated in length with their neighbors and created power spectra approaching 1/f. We suggest, as have others, that 1/f patterns reflect world structure and mental process. Moreover, a 1/f temporal shot structure may help harness observers’ attention to the narrative of a film.
In our segment on Bang Goes the Theory (Mon 28th Sept 7:30pm BBC 1; catch it on BBC iPlayer if you missed it) Peter Lamont and I discussed some of the everyday psychological phenomenon used by magicians to fool our perception and create illusions.There were two main phenomena demonstrated in the piece: Inattentional Blindness and Change Blindness. If you want to try the change blindness test used in the show for yourself skip down to the 'Now its your turn!' section below.
Inattentional Blindness
In the piece, Peter demonstrated how a magician can make an object seem to disappear by controlling where you are looking. We have the impression that we see all of the visual world in great detail at the same time because as soon as we turn our attention to an object in the world we can see it. However, this is an illusion created by our brains to overcome the limitations of our eyes. Our eyes are actually only able to pick out visual detail from a very small portion of the world at any one moment. This is because the light sensitive surface at the back of the eye, known as the retina, has the greatest concentration of cells in a small region around its centre, known as the fovea. The light landing on the fovea comes from a region out in the world roughly the size of your thumb nail held out at arm’s length. When we look at an object we move our eyes so that the light reflected off an object lands on the fovea. The light reflected off objects away from the centre of our attention land on less sensitive parts of the eye resulting in a lower quality image. We may have the impression that we can see everything but in reality if our eyes are not pointing at an object all we can see is a blurry image with poor definition.
However, because we move our eyes on average 3-5 times every second our brains use all the information we get across this sequence of fixations (when the eyes are still) to piece together a detailed impression of the world. Our perception of the world is constructed over time from minimal detail and we assume that we see more than we actually do. This assumption is used by magicians to make you think that if you didn’t see something happen, such as a ball being pocketed, then it didn’t happen!
In the Bang! piece, Peter uses misdirection to ensure that you are not looking at the hand which is hiding the object. He does this by using cues to encourage you to look elsewhere. These cues may involve a sudden flourish of a hand, the waving of a wand, directing his eyes to an object, his posture and referring to an object by name. All of these cues direct our attention to one object whilst misdirecting us from the hand actually performing the trick. Social cues such as these are so powerful and we respond to them so consistently that magicians are able to reliably influence the attention of individuals or entire audiences. The result is that we are blind to the method of the trick because we failed to attend to it. The most famous inattentional blindness demonstration is Simon’s & Chabris ‘Gorillas in our Midst' experiment in which they made viewers fail to notice a person in a gorilla suit by focussing viewer attention on basketball players in the same scene. For a demonstration of the Simon’s & Chabris demonstration and a similar magic trick utilising inattentional blindness see the Colour Changing Card Trick we previously described on this blog.
For further discussion of the use of natural inattentional blindness phenomenon used by magicians see Peter’s book and article below.
Lamont, P., Henderson, J. M., & Smith, T. (in press). Where science and magic meet: the illusion of a ‘science of magic’.Review of General Psychology. (* e-mail Peter for a pre-print)
Change Blindness
Another everyday phenomenon used by magicians to create illusions is Change Blindness. Change Blindness refers to when a detail of the visual world changes without us noticing. For example, the most famous change blindness demonstration was performed by Simons and Levin (1997). In this demonstration they made strangers in the street fail to notice when the person they were talking to changed into a different person. They achieved this by hiding the change behind a door which was rudely carried between the stranger and the experimenter during the conversation. See a demonstration of the effect here. And the original article here.
Change Blindness occurs for three reasons: 1) either we don’t perceive the object before it changes, 2) our attention isn’t attracted to the change, or 3) we don’t compare the changed object to our memory of the original object. The first reason is similar to inattention blindness in that it is a failure to take in enough detail of the object. This is probably due to not looking at the object and only taking in sketchy information about the object from our peripheral vision (away from the fovea). The second reason is essential for us to experience change blindness. In the Simons and Levin example the change was hidden behind the door. If the door hadn’t been present and somehow one person had spontaneously changed into another person the change itself would have captured our attention. This is because the change creates visual transients: sudden unnatural changes in the light landing on our retina. In nature visual transients are often caused by dangerous events such as rapidly approaching predators or unexpected object suddenly lunging towards us. As a consequence our visual system is tuned to transients and responds by directing our eyes towards them. Our faith in the our ability to automatically response to such changes is so great that if these transient are hidden we often fail to check whether a change has happened and fail to notice them.
In the Bang! piece we used a change blindness demonstration to show how we rely on the visual world to tell us what has changed. The method we used to hide the change was more subtle than the door example: we used the viewer’s own eye movements. Every 200-300ms we rapidly shift our eyes in order to look at a new object. These eye movements are called saccades. We generally do not blink during a saccade so light continues to land on our retinas. Saccades are so rapid that the light projected into our eyes blurs across our retina. Clearly we do not perceive this blur as otherwise every time you moved your eyes – such as right now, as you are reading this text – we would experience flashes of blurring. To ensure we don’t see the blur our visual system stops processing the light during a saccade. We are effectively blind for the 20-50ms it takes to shift the eyes to a new stable position! If a change is timed to coincide with a saccade the visual transients associated with the change are hidden and our attention is not drawn to it. The change is hidden and the viewer is completely unaware that anything has happened.
In our lab we use an eye tracker to monitor people’s eye movements and detect when a saccade occurs. An eye tracker uses a high-speed infra red camera to locate the viewer’s pupil and record its movements. From the movement of the pupil we can work out where the viewer is looking on a computer screen. In the Bang! piece we showed Dallas a photograph of a real scene and changed objects in the scene as he moved his eyes. For anybody else watching the scene, the changes are immediately apparent because the changes are not timed to coincide with our eye movements. The changes create visual transients which capture out attention. But for Dallas it is much harder for him to detect the changes as the transients are hidden and instead he has to rely on his memory to check what has changed. As our memory for a visual scene is not perfect we often fail to detect changes either because we didn’t store enough information about the objects in memory before the change or we fail to compare the object to our memory. This demonstration shows how minimal our perception of the visual world is, how little information we store in our memory, and how we rely on the visual world itself to tell us when something important has happened.
Now its your turn!
Without an eye tracker we can create similar change blindness effects by hiding the visual transients behind a flicker. I have recreated the change blindness test performed by Dallas by repeatedly flickering the photograph. Every so often an object in the scene will change. Your task is to watch the first video below and detect how many changes occur.
Did you get the right answer? Now watch what happens when we make the changes without the flickers:
The changes seem to ‘pop out’ because the visual transients capture our attention.
The list of changes and their times in the video are written below in invisi-text. Uncover the answer by highlighting the white text.
*Hold down the left mouse button here and drag it to the bottom Ã
9 sec = Plane in centre appears
10 sec = VUE sign disappears
12 sec = Top floors of the tower block on the right disappear.
14 sec = Safety barriers change colour
16 sec = Man and child change to a woman
18 sec = Rear wheel of left-most bike disappears
20 sec = lowest branch of the tree disappears
21 sec = Man appears next to dustbin
23 sec = Bus disappears
25 sec = Left tower block disappears
*ß finish highlighting text here.
The first time you viewed the scene you may have found it easier to detect some changes rather than others. Dallas had the same experience during his version of the demonstration. The variability in change detection is due to where your eyes are at the time of the change and where they have been up to that point. Work in our lab has shown that you are more likely to detect changes to objects that you are looking at or moving your eyes to compared to objects you haven’t looked at (Henderson & Hollingworth, 1999). When the changes happen without being hidden by a flicker or an eye movement these factors don’t matter. All changes are equally likely to capture your attention. This difference tells us that what you perceive in a visual scene and how the details are stored in memory is very closely related to what you look at. You may think you are aware of peripheral details in a scene but if you do not look at them you will not be able to detect when they change.
These phenomena, inattentional blindness and change blindness are key tools used by magicians to control our visual experience and create fantastical effects. However, they are not exclusive to magic as they are a natural product of how our visual system works. We fail to detect changes or see details of our visual world during every day life. It is only when this failure is brought to our attention during demonstrations like the one above or during magic tricks that we are experience surprise. By looking at these phenomena and how magicians utilise them we can learn more about how these operate in every day life and how much our experience of the visual world is an illusion.
For further information on using eye tracking to investigate change blindness look at the papers below and the rest of the research coming out of our lab:
A few weeks ago I had the great privilege of hosting the BBC in our Visual Cognition lab. The BBC filmed a piece for the new science show, Bang! Goes the Theory (Mondays 7:30pm, BBC1). In the piece my colleague Pater Lamont, a psychologist and talented magician explained some of the psychological phenomenon utilised by magicians during close-up magic and then I demonstrated how these phenomenon can be investigated using eye tracking. The whole piece was filmed in and around Prof. John Henderson's Visual Cognition lab.
The piece will be broadcast on Monday 28th September BBC1 at 7:30pm (10:35pm, BBC1Wales). If you miss it you can view it on the BBC iPlayer (UK only). Catch it, I'm sure it will be fun and informative.
A colleague of mine, Dorota Ostrowska from Birkbeck has written an article on the potential for interdisciplinary film research. As a case study Dorota has used my PhD thesis. Her summary of my research and its implications for broader film studies is very insightful and complimentary. I especially enjoyed her discussion of the implications of my cognitive theory of continuity editing on the classical socio-political interpretation of the 'Hollywood Style' of film editing:
(pg. 108) "Smith’s research offers us a new insight into the view of continuity
editing as a basis of bourgeois and capitalist ideology. His findings show a
psychological basis for continuity editing, revealing its reliance on some
powerful cognitive mechanisms. Such an understanding of continuity
editing does not make the ideological tenets of the cinema that it underpins
right. Instead, it presents us with a different paradigm (scientific and empirical)
to explain why, even today, and in spite of vehement criticism, continuity
editing is still a predominant mode in audiovisual production across
the world. The reasons for the lasting success of this type of editing may be
as much cognitive and perceptive as they are ideological. Furthermore,
Smith’s conclusions change the status of certain editing practices, such as
continuity editing, from a series of arbitrary choices imposed on audiences
by editors working under pressure from money-hungry studios, into
choices based on the cognitive expectations of these audiences. Smith’s
research gives us better understanding why continuity editing has been so
pervasive and the cognitive basis of its ‘naturalness’ to spectators."
This is indeed the implications of my theory and empirical evidence. Although, as a cognitive scientist I would never have stated these implications so boldly as Dorota :) I do not dismiss the political, ideological, or practical forces influencing the path of film form evolution. The intention of my research is to add an extra influence to the evolution, namely cognitive compatibility. I will leave it up to film theorists such as Dorota to decide what this means for classic film theory.
I do not agree with all of Dorota's interpretations of my research but I think the article is very thought provoking and it is nice to see my research influencing others.
Film as a source of visual knowledge in informatics, architecture and music
Ostrowska, D. (2009), Studies in European Cinema, Vol. 5 (2), pgs. 105-116.
This article is a series of questions and reflections, which grow out of my recent
research projects which centred on the issue of interdisciplinarity in relation to
film studies and cinema. Through my work at the Cinema Interdisciplinary
Network (CINET), which we have been running at the University of Edinburgh
since 2005, I have come across a number of academics and researchers who work
with cinema or use moving images in their research but who are not film studies
scholars and are not working in the areas more traditionally associated with film,
such as literature or cultural studies. Could the CINET members’ engagement with
cinema yield some new insights about the object of their study for film scholars?
What does it mean that cinema is becoming a repository of visual knowledge?
How is this knowledge to be treated and organised? What does it mean that film is
becoming not only an aesthetic object but also a tool to conduct intellectual analysis?
What are the difficulties and pitfalls of such research projects? In order to
address these questions, I will discuss two projects presented in the context of
CINET – ‘An Attentional Theory of Continuity Editing’ which is Tim Smith’s doctoral
project completed in the department of Informatics (Computer Science) and
‘Inflecting Space: Correlating the attributes of voice with the character of urban
spaces’, which is a collaboration between architecture and music.
A new journal article published this week in Psychological Science asks whether watching popular Hollywood films depicting historical events can enhance learning of historical facts. This is a question that has been asked many times in the past and educators often assume that there is a benefit to watching video reenactments, not least of all because they are entertaining. This study aimed to test whether there can be both benefits (when the hostorical informationd epicted in the film is accurate) and costs (when the details are inaccurate) to using historical films in a history lesson. They used selection of excerpts from well-known feature films including The Last Samurai (Ed Zwick, 2003), Amistad (Steven Spielberg, 1997) and Glory (Ed Zwick, 1989). They recorded better memory for historical facts when the students watched a film version but only when the film depicted the facts accurately. When the facts were inaccurate the false details in the film sometimes take precendence over the accurate details taught to the students using traditional means.
This is a very interesting study as it shows both the strengths of using films to teach history but also the weaknesses. Any educator considering using a feature film in their lessons should consider that the primary purpose of a feature film is to entertain, not be historically accurate. The most entertaining way of telling a story is rarely the most authentic. Film makers often distort the truth to increase drama and excitement. As outlined in the study below, educators should make their students aware of this so that they don't believe everything they see on the screen.
I personally learned this lesson the hard way by very confidentally answering "Ben Hur" to the question "Who helped Jesus with the cross?" in a Religious Education class. Damn artistic licence!
(*according to the New Testament the answer is Simon The Cyrene...... not Charlton Heston)
Using Popular Films to Enhance Classroom Learning: The Good, the Bad, and the Interesting
Butler, A. C., Zaromb, F. Z.,Lyle, K.B. &Roediger, H. L. III (in press) Psychological Science
ABSTRACT—Popular history films sometimes contain major historical inaccuracies. Two experiments investigated how watching such films influences people's ability to remember associated texts. Subjects watched film clips and studied texts about various historical topics. Whereas the texts contained only correct information, the film clips contained both correct information (consistent with the text) and misinformation (contradicted by the text). Before watching each clip, subjects received a specific warning, a general warning, or no warning about the misinformation. One week later, they returned for a cued-recall test about the texts. Watching a film clip increased correct recall of consistent information relative to recall of the same information when subjects did not see the clip. However, when the information in the film contradicted the text, subjects often (falsely) recalled misinformation from the film. The specific warning substantially reduced this misinformation effect. Teachers should use popular history films with caution and should warn students about major inaccuracies in the films.
Lars von Trier’s Antichrist is an unsettling film for many reasons. It will be discussed in depth for its more obvious unsettling traits such as its intense graphic violence and self-mutilation and lack of a clear message especially with regard to gender relationships and the nature of evil. Some of these traits, such as the horror imagery, settings and plot-devices may seem unique within Trier’swork to date. However, there are other, more subtle methods used to unsettle the viewer which Trier has been refining since Dogme 95. I’m speaking, of course about his use of discontinuous editing.
Antichrist is a two-header centring around the grief of a married couple (Charlotte Gainsborough and Willem Dafoe) after the loss of their young son during the film’s prologue. The majority of the film depicts the two main actors whilst they try to come to terms with the loss and confront the wife’s fears about their holiday cottage in the woods, Eden.The film employs stunning high-speed black and white cinematography to add incredible beauty to some brutal scenes including the son’s fall to his death and the copulation of the couple during the prologue. These scenes are offset by, uncharacteristically glossy scenes of the couple interacting initially at home and then at the cottage.These scenes have a “Hollwood”-esque appearance: very beautifully lit, shallowly focussed with frequent focus-pulls into the shot, and shot from mostly stable camera positions. This is in stark contrast to the compositional style developed by Trier and his cinematographer Anthony Dod Mantle over their last few films culminating in computer-controlled camera and lighting system (Automavision) employed by Trier in his last film, The Boss Of It All. In a previous blog post I discussed how this system randomly shifted the camera position, zoom and focus of the camera. This resulted in the focal object (e.g. a talking character) drifting out of shot or being framed in such an obtuse manner that the viewer had to struggle to fixate the region of interest such as the characters eyes or mouth.The unpredictable camera movements also force the viewer to actively pursue the focal object with their eyes rather than stablise their gaze (i.e. fixate) the object. Even before Automavision, Trier’s use of hand-held cameras and improvised staging and acting in earlier films (see Idioterne and Dancer in the Dark) meant that the viewer always had to be active in their viewing of his films. Unlike Hollywood films edited according to the Continuity Editing Style (see my thesis) which keep focal objects relatively stationary on the screen and always establish predictable screen locations or the modern Intensified Continuity style (see David Bordwell’s discussion ) which use rapidly edited close shots , Trier’s films make no concessions to the viewer’s attention, actively engaging them in the staging of a scene by forcing them to search the screen for the most relevant details. In Dancer In The Dark the hand-held camera roams around the main characters often failing to keep up with changes in action. Trier compensates by cutting to different takes of the same scene with a slightly different camera position on the same actors. This creates, what is traditionally referred to as a Jump Cut: a sudden displacement of an object or actor on the screen caused by a cut to a camera position less than 30 degrees away from the original position (for an in-depth discussion of perceptual reasons for Jump Cuts see Joseph Anderson’s The Reality of Illusion). Prior to the cut, the viewer is trying to compensate for the actor’s movement relative to the screen by pursuing the actor or saccading. A sudden unexpected Jump Cut then displaces the actor on the screen requiring the viewer to reorient to the new shot. The consequence is much more active eye movement behaviour than is usually required when viewing a film edited in the Continuity style. This level of active viewing is so alien to a cinema audience that the experience can be exhausting and unsettling. Such affective response is utilised by Trier to add to the overall experience of watching his films.
At the recent Society for Cognitive Studies of the Moving Images annual meeting in Copenhagen we were treated to an advanced screening of Antichrist and Q&A with Lars von Trier. I asked him how his discontinuous editing style had come about and whether he used it deliberately to unsettle his viewer. Trier enthusiastically described the way that he and his long-term editor Molly Marlene Stensgaard had experimented in the editing suite and devised a style of cut that, whilst violating classical editing conventions, they believed accurately portrayed the mood of a scene. I referred to this cut as a Jump Cut but Trier seemed uncomfortable with this name and possibly its negative connotations. Trier explained that he didn’t consider the cut to be discontinuous because he ensured that the audio track was continuous across the cut. The camera position would suddenly shift across the cut as if it had been moving smoothly around the principal object but a portion of the movement had been omitted by the edit. However, because the camera positions were actually filmed at different times the shots either side of the cut could depict a (somewhat) continuous action and the audio track could be stitched together to make it sound as if the scene were continuous. Trier stated that he thought the most important element for creating the illusion of continuity was the audio track. As long as the audio implied that the scene was continuous the audience would perceive the scene as continuous. A Jump Cut traditionally implies an omission of time. Trier’s cut overrules this implication by using continuous audio. Not perceptual time is omitted. As such we will refer to Trier’s type of cut as an Anti Cut.
The effect of an Anti Cut on the viewer is cognitive dissonance. Cognitive dissonance comes about when a person holds two (or more) conflicting concepts in their mind at the same time. The experience of cognitive dissonance is unpleasant. We endeavour to find order in our experiences and ideas and when conflicting concepts cannot be resolved we are uncertain how to act. If these concepts come from two different sensory modalities the brain tries to reconcile the two sources and come up with an amalgamation. A classic example of this is the Mc Gurk Effect (see video below). The McGurk effect is a demonstration of how we did not evolve to process our senses in isolation. When watching someone say a word our perception of the sounds they make is informed by both the audio and way their lips mouth the word. If these two sources are in conflict e.g. the mouth says “ba” but the audio track says “ga”, we perceive something in between, e.g. “da” . This demonstrates how sensory perception is multimodal and the brain will endeavour to resolve cognitive dissonance even if it means distorting reality and creating an illusory perception.
In the case of Trier’s Anti Cut it appears that the audio track takes precedence over the visuals to create the perception of temporal continuity. This may only be possible as long as the visuals do not conflict with the audio. If the cut involved a shift in camera position and a repetition of action we would expect the viewer to perceive the repetition irrespective of how temporally continuous the soundtrack is. However, Trier typically cuts in between actions when no visual events are occurring and, therefore cannot be used to perceive the omission of time. The sudden shift in camera position and changes in actor posture, position, lighting, etc all imply a temporal omission but it appears that these are overridden by the temporal continuity implied by the audio track. Scenes edited using Anti Cuts, such as the bedroom scenes in the first act of Antichrist or the scene in which Bjork’s character’s son receives a new bike in Dancer in the Dark, create a strange, almost abstract sense of time. Many aspects of the scene imply temporal continuity such as the continuous audio and the depicted events. However, the sudden shifts in camera position caused by the Anti Cut imply discontinuity. The resulting perception may be of general temporal continuity but with residual conflicting cues which, if reasoned about may confuse the viewer and undermine the general perception of continuity.
Trier’s last film, The Boss of it All contains many examples of such conflicting cues. In order to compensate for the frequent, inexplicable camera moves caused the computer controlled camera system (Automavision) Trier had to use frequent Anti Cuts to stitch together different takes. Often the lighting and sound balance would also change wildly across such cuts. Unlike in Antichrist when the audio was carefully controlled to imply temporal continuity, the sudden changes in The Boss of it All implied temporal discontinuity. The result was the perception of a scene as a construction, clearly stitched together from takes filmed at different times. The depicted actions implied temporal continuity and if the viewer chose to engage only with this dimension and ignore the other conflicting dimensions they could perceive scenes as temporally continuous but my feeling of watching the film was that such active reconstruction of the scene required much more effort than watching a film edited according to the Continuity Style or even some of Trier’s other films.
As always, Lars Von Trier is pushing the boundaries or editing in order to experiment with our perception of film. As a film viewer I adore Trier’s introspective experiments and as a scientist I would love to see empirical investigations of these issues. What is the relationship between the different sensory modalities in film and which modality takes precedence and leads to the resulting perception of continuity. In Chapter 5 of my thesis I laid out a theory about how continuity perception in film may operate and I went on to empirically investigate time perception across cuts. This investigation was purely in the visual dimension although I acknowledged how powerful the audio dimension appeared to be and how editors placed such emphasis on audio in the creation of continuity. Direct empirical investigation of the role of audio in continuity perception is required but for now we can occupy ourselves with marvelling the masterful way Trier manipulates our experience of time and space in order to create an unsettling viewing experience.
Would you like to come and work in Visual Cognition? Would you like to be part of John Henderson's great research lab and learn how to investigate static, dynamic, and film cognition using eye tracking?
Well we now have funded PhD studentships available.
Details below:
UNIVERSITY OF EDINBURGH SCHOOL OF PHILOSOPHY, PSYCHOLOGY AND LANGUAGE SCIENCES
PPLS (PSYCHOLOGY) CAREER DEVELOPMENT STUDENTSHIPS Deadline for Application: Friday 10th July 2009
A minimum of four Career Development Studentships are offered by Psychology within the School of Philosophy, Psychology and Language Sciences
Career Development Studentships are designed to provide students with teaching experience, training, and other career development opportunities. A stipend of £10,000 per annum will be provided in addition to tuition fees at the Home/EU level and additional programme costs of £600 per annum. The studentships will start in September 2009.
Further information on the topics/research areas are available online at: http://www.ppls.ed.ac.uk/postgraduate/CareerDevStudentship.html Eligibility Candidates should have an academic background in Psychology holding (or expecting to hold by September 09) a postgraduate Masters level qualification or equivalent and be eligible to apply for PhD level study at the University of Edinburgh. Students entering the second year of their PhD Psychology study at the University of Edinburgh are also encouraged to apply.
Both International and Home/EU students are eligible to apply although please note that the tuition fees provided in the studentship are at a Home/EU level only.
Application Procedure Candidates are invited to apply through the University of Edinburgh online application system (EUCLID); please ensure that you note your intention to apply for this studentship under the funding section of the application form. When you submit your application, please also email pplspg@ed.ac.uk to indicate that you have done so.
Deadline for application: Friday 10th July 2009 Selection Process Short listed candidates will be invited to attend an interview at the University of Edinburgh. Successful candidates will be informed by Friday 14th August 2009
I am a lecturer in the Department of Psychological Sciences, Birkbeck, University of London. I study all aspects of Visual Cognition including attention, perception, and memory in static and dynamic visual scenes. I have a passion for film and combine this with my methods in empirical psychology to investigate Film Cognition. This blog chronicles my research including my ideas, experiments, insights, methodologies, references, inspirations, and rambling digressions. Enjoy!
Follow me on twitter @timothy_j_smith
Posts
