ECHOES

I am proud to announce a very exciting new project I am involved in, ECHOES II: Improving Children's Social Interaction through Exploratory Learning in a Multimodal Environment.

The intention of the ECHOES project is to develop a safe, motivating, and inclusive multimodal learning environment allowing Aspergers and Typically Developing children to explore different social situations to improve their social interaction and communication skills. ECHOES II will provide a unique opportunity to explore a place in which learning happens and in which learning can be studied.

The ECHOES project is a multi-site (Edinburgh, London, Sussex, Birmingham, Dundee, Glasgow, and Cardiff), three year project funded jointly by ESRC and EPSRC via the TLRP TEL call. This project which will combine the latest and most innovative methods in participatory design of education technology with cutting-edge multimodal interfaces including motion, gesture, and facial expression capture, audio reactivity and gaze tracking within an augmented reality 3D environment. Further details on the project can be found here:

http://www.lkl.ac.uk/cms/index.php?option=com_content&task=view&id=287&Itemid=91

The ECHOES project is currently at the recruitment stage. We are looking for a range of full-time, part-time RAs and PhD students. These posts are spread across the various sites. Details are below. If you are interested in any of these posts please follow the links below and contact the people named in each advert.

*Check each advert for closing date as it varies across posts*

Full-Time Research Fellow (School of Informatics, University of Edinburgh)

The successful applicant will contribute to the design and development of an intelligent multimodal interface, using a combination of an interactive whiteboard, 3D graphics, speech synthesis, and video input technology. The post will involve development of the ECHOES software architecture and organisation of software modules developed at the other sites. Experience in software engineering and multi-modal interfaces is highly desirable.

http://www.jobs.ed.ac.uk/vacancies/index.cfm?fuseaction=vacancies.detail&vacancy_ref=3009321

Part-Time Research Assistant (School of Informatics, University of Edinburgh)

Job description same as above except RA will focus more on combining existing interface technologies, e.g. motion tracking + gaze tracking, into the ECHOES architecture.

http://www.jobs.ed.ac.uk/vacancies/index.cfm?fuseaction=vacancies.detail&vacancy_ref=3009321


Full-Time Research Fellow (London Knowledge Lab, University of London)

The successful applicant will collaborate in the development of the ECHOES II environment with other researchers, focusing particularly on the design and implementation of the learning activities and interventions along with the ‘action engine’ responsible for the selection of the activities based on user input .

http://jobs.ioe.ac.uk/Vacancy.aspx?ref=7AC-CPLKL-4665

Part-Time Research Assistant (London Knowledge Lab, University of London)

The successful applicant will collaborate in the design of the ECHOES II environment with other researchers, focusing particularly on the design and testing of the learning activities and the evaluation of the educational impact of the environment.

http://jobs.ioe.ac.uk/Vacancy.aspx?ref=7AC-CPLKL-4664

Full-Time Research Fellow (School of Informatics, University of Sussex)

The successful candidate will use participatory design methods with children in order to design and implement the interface for the ECHOES II environment, and collaborate in the develompent of the environment with the other researchers, focusing particularly on the development of 3D characters and avatars and “mirroring” the user’s actions through video input.

Enquiries should be address to Dr. Judith Good (J.Good@sussex.ac.uk).

Full-Time PhD. (School of Computing, University of Dundee)

The School of Computing invites applications for a PhD student to work on the functional communication of children with autistic spectrum disorder (ASD) and similar disabilities within a wider project to develop and evaluate a technology enhanced multimodal learning environment designed to scaffold children’s social interaction and communication skills.

The successful candidate will collaborate with other researchers who will be designing a system for the exploration and learning of social interaction skills, and will be focussing of how the ECHOES environment impacts on the interactive communication of children with ASD, specifically Asperger Syndrome.

A first class or good 2:1 Honours or Masters degree, or equivalent, in Applied Computing, Computer Science, Psychology, Cognitive Science or related discipline is essential, as are knowledge of user-centred design methods, computer vision, and an ability to organise and run studies involving children, parents and professionals. Good written and communication skills are essential.

Enquiries and application requests should be addressed to Dr Annalu Waller (awaller@computing.dundee.ac.uk ).

Closing date for applications: 15 September 2008

Completing the Circle

The Lansdown Centre for Electronic Arts (Middlesex University) is hosting a one-day symposium focussing on novel methods, or methods newly borrowed from other disciplines, in evaluating the user's or audience's response to media such as websites, portable media (such as iPods, PSPs), pervasive games, film, videogames, technology-rich performance, interactive art. The organising committee have put out a special call for researchers using eye tracking to evaluate film or media.

For further information see:

http://www.cea.mdx.ac.uk/?location_id=59&item=31

Research Assistant in Active Viewing of Dynamic Scenes

John Henderson and I have recently been awarded a 2-year Leverhulme research grant investigating Active Viewing of Dynamic Scenes. We are very excited about the project as it will enable us to explore issues of attentional control, eye movements, and scene processing using dynamic scenes such as naturalistic social scenes, film and television. This will allow us to build upon the initial research I have been doing in this area and provide the foundations for other research groups to follow.

The project will begin later this year and we are currently advertising for a part-time Research Assistant to join the project. Details copied below. If you are interested please contact me or John asap as the deadline is July 11th.

"Research Assistant, Edinburgh University

Active Viewing of Dynamic Scenes: Eye Movements in Video

The University of Edinburgh invites applications for a two-year, part-time (20 hours per week), fixed term Research Assistant related to a new project funded by the Leverhulme Trust titled: Active Viewing of Dynamic Scenes: Eye Movements in Video. The post requires technical expertise with computational methods in computer graphics with particular application to video. Knowledge of human visual perception and eyetracking methods is desirable. The research assistant will interact with cognitive psychologists and cognitive scientists in the research team, and should have excellent interpersonal and communication skills. Applications are welcomed from individuals with a good first degree in computer science, informatics, or related computational discipline.

Informal enquiries: Prof John M. Henderson (john.m.henderson@ed.ac.uk).

Apply online (www.jobs.ed.ac.uk, Vacancy Reference: 3009312) including a CV and statement of relevant experience. Alternatively, telephone the recruitment line on 0131 650 2511. "

MSC in Visual Cognition

I am happy to announce that as of the academic year 2008-2009 Edinburgh University Psychology department will be hosting an MSc programme in Visual Cognition. This one-year taught MSc will give students the opportunity of learning from leading researchers in the field of Visual Cognition and Percention in Action: Prof. John M Henderson, Dr. James Brockmole, Dr. Robert McIntosh, and Dr. Antje Nuthman. Students will also benefit from the broad and varied visual cognition and eye movement research groups across the University. The taught course will culminate in a Summer research project in which the students can explore their own research ideas under the tutelage of researchers such as myself. This provides a framework under which students interested in Psychology and Cognitive Science can improve their empirical skills in the area of eye tracking and apply it to new and innovative areas. I would specifically welcome applicants who are interested in eye movement and attentional control during static and dynamic scene viewing and during film and media viewing.

Funding may be available for this MSc.

For further information follow this link:
http://www.psy.ed.ac.uk/postgrad/msc/vc

If you have any specific questions about the course or want to discuss potential research ideas contact me directly (tim [dot] smith [at] ed.ac.uk).

Attention: experimental film project

How do we watch film? How does the way that I watch a film differ from the way you watch a film? If we were to make a film of the way I watch a film, what would it look like?

These are some of the questions a group of Design and Digital Media MSc. students under my mentoring are exploring in an art installation entitled Attention: experimental film project. Parag Mital, Stefanie Tan Su Ann, and Dave Stewart have create an interactive film experience for their Digital Media Studio Project that will be premiered this Friday (21^st March), 12-5pm in Teviot Row House, Edinburgh University.

The installation is inspired by my research into how we attend to film, how this shapes our experience of a film, and how film manipulates this attention (see my thesis). Film can be interpreted as an analogue of our experience of the visual world. Snapshots of fragmented but related visual information is presented to us in a way that we can comprehend by forming conceptual linkages between them. This is similar to the way we acquire information from the world by moving our eyes. If we take this analogy to its logical conclusion, a visual recreation of the information acquired from a scene as a person shifts their eyes may resemble an edited filmic representation of that scene.

Creating such a filmic recreation of a viewers experience is the intention of this project.

How is this accomplished and what is the result? Check out the project website or come along to the premier to find out

http://www.theexperimentalfilm.com/

Rank Prize Funds Mini-Syposium

Last week (3-6th December, 2007) I attended a Mini-Symposium on Representations of the Visual World in the Brain organised by the Rank Prize Funds. The symposium was a small gathering of highly respected established researchers and young researchers across the field of Visual Science. Attendees included Ron Rensink (UBC), Nancy Kanwisher (MIT), Mike Land (Sussex), Ben Tatler and Ben Vincent (Dundee), Jens Helmert (Dresden), Jim Brockmole (Edinburgh), Melissa Vo (Munich), and many others from across Europe and North America.

The symposium was an incredibly stimulating, intense experience and I have to express my immense gratitude to the Rank Prize Funds for organising it (aside: the Fund was established by the late Lord Rank to support scientific research in his two main interests Nutrition and OptoElectronics; Rank was also the founder of Rank Film....a rather coincidental overlap with my interests). The symposium was held in the wonderfully picaresque Wordsworth Hotel, Grasmere, Cumbria. Not that I got to appreciate much of it as I was too busy being intellectually stimulated.

My enjoyment of the symposium was rounded off by my being awarded the Prize for the Best Presentation by a Young Researcher for my presentation entitled 'Facilitation of Return'. It is a great compliment for the work I am doing with John Henderson to be acknowledge by such a distinguished group of researchers.

.....and Lord Rank

C.V. updated

Just a quick note to say that I have updated my C.V. and my homepage. Both were old and rather poor representations of what I have been up to. Much better now :)

The Famous Colour Changing Card Trick

Richard Wiseman, the magician, psychologist, science communicator and author of Quirkology: the Curious Science of Everyday Lives asked me and John Henderson to measure viewers’ eye movements whilst they watched one of his magic tricks. The Colour Changing Card Trick is a very clever use of a perceptual phenomenon known as ‘inattentional blindness’. Check out the trick before reading on:

Inattentional Blindness is an absence of awareness of some detail or event in the visual world due to a failure to attend to it. This absence can often be strikingly large (as in Richard’s card trick) or, most famously in Simons & Chabris (1999) ‘Gorillas in our Midst' experiment. In the Simons & Chabris experiment subjects were told to watch a video of two teams, one wearing white, the other wearing black pass basketballs within their teams. Half of the subjects were told to count the number or passes made by the team wearing white. The other half were told to count the passes made by the team wearing black. Half way through the video a man wearing a black Gorilla suit walked through the scene, stopped in the middle of the scene, waved at the camera and then walked out of shot. When asked after the video if they had noticed anything odd during the video, the majority of subjects failed to report the Gorilla! The probability of noticing the Gorilla was greater when the subjects had been instructed to attend to the black team (58% detection) compared with the white team (27%) indicating that the task had biased the subjects attention either towards black or white objects. The subject’s selective attention shapes the details of the scene that reach the level of conscious awareness and subsequent memory but, importantly the subject is not aware that their awareness is in any way partial. This mismatch between what the subject think they see and what they actually see is what creates the shock at the end of the Gorilla experiment or the Colour Changing Card Trick.

The Colour Changing Card Trick uses a simple card trick to distract viewer attention from what is actually going on, namely the changing of both presenters’ T-shirts, the backdrop, and the table cloth. Such misdirection is a classic tool of any magic performance. All changes are made off camera when the continuous camera shot zooms in to a close-up. This removes the actual change itself from view, leaving only the result. In order for viewers to notice the change they must have previously attended to the object that has changed and have sufficient memory of that object to notice that its current form is different. By measuring viewer eye movements during the trick we can see whether viewers attend to the objects before the changes and whether there is any increase in attention to the objects after the change. Such an increase may indicate the precise moment at which the viewer notices the change.

The results are still being analysed but for now Richard has posted a video illustrating the eye movements of 9 subjects whilst they watched the trick (5 men and 4 women). The one red spot is the gaze location of a woman who detected the Female presenter’s T-shirt change.

This video was created using my own Gazeatron software.

As can be clearly seen from the video most viewers look in roughly the same parts of the scene at the same time. This close control over where viewers are looking is exactly the intention of the magician. By ensuring such systematic viewing the magician can hide changes/manipulations in the unattended spaces. It is only once one of the viewers notices the change (the red spot) that their gaze location begins to differ from everyone else’s.

If you want to know more about the psychology of misdirection and its relationship to eye movements check out Gustav Kuhn’s research at the University of Durham.

Eyetracking and Dynamic Scenes Interest Group

The technology available for recording the direction of a person’s gaze has taken massive leaps forward in the last few years. The cost, precision, usability, and, most importantly discomfort of using an eye tracker has now reached a level where any psychology research lab or usability/HCI assessment centre can use eye tracking. Recording the focus of a person’s overt attention (as well as other measures such as pupil dilation, blink rate and eyelid closure) can provide a real-time measure of their experience and an indication of their cognitive processes. Such an insight can complement existing methodologies and allow researchers to understand the dynamics of human experience.

At the recent Experimental Psychology Society conference held here in Edinburgh, eye tracking was clearly rising in popularity as many different researchers applied it to areas such as reading studies, lexical processing of speech, facial expression recognition, social attention, object perception, working memory, visual search, scene perception, and, of course, attention research. The current popularity of eye tracking can be directly related to the increases in technology. Most of the researchers currently using eye tracking are not, primarily attention researchers. The tools provided by the eye trackers allow them to use eye movements as an index of other phenomenon such as real-time cognitive processes. Previously, a research lab would be required to build their own display and analysis tools from scratch (using programming environments such as C or Matlab). This placed eye tracking research clearly out of reach for most people and required a considerable understanding of basic oculomotor control and attention. I believe the current renaissance of eye tracking (of which Edinburgh is part of with our multiple research groups using eye tracking) is a hugely positive step-forward and I hope it continues into other areas of psychology.

One big obstruction to the advancement of eyetracking in to other areas of psychology is its current incompatibility with dynamic scenes. Many areas of psychology are interested in understanding human behaviour in realistic settings such as social interactions, conversations in the real-world, moving through the world, performing actions, complex tasks such as driving, and even watching TV (or is that just me :). As anybody who has ever tried to record eye movements in one of these settings will know, it is phenomenally complicated and time consuming. For example, Mike Land’s influential research on goal-oriented attention during tea making required the construction of novel eye movement technology and the hand coding of every frame of the resulting video! It is no wonder that eye tracking of dynamic scenes is so uncommon when the analysis is so laborious.

The situation has recently got a lot better with the introduction of new eye tracking software. The Tobii and Eyelink (SR Research) eye tracking systems now come with software for displaying dynamic scenes (e.g. videos, animations, etc). However, being able to display the videos is pointless if there is no easy way to analyse the resulting data. No systems, that I am currently aware of assist in the analysis of eye movements in dynamic scenes.

This lack of support for eye movement researchers interested in using dynamic stimuli has motivated me to make a call-to-arms. I know of a growing number of researchers, both in academia and industry who are struggling with the problems, both practical and theoretical associated with recording eye movements in dynamic scenes. No support structure exists for these researchers, no common source of knowledge or tools, and no where they can go to ask for help. Because of this I’ve decided to put out a call to all researchers using or interested in performing eyetracking in dynamic scenes. The Eyetracking and Dynamic Scenes [EDS] Interest Group will comprise a mailing list to which members can post information relating to their research, ask for help, and post useful resources (e.g. software) and references.

If you are interested in signing up for the Eyetracking and Dynamic Scenes [EDS] mailing list, please e-mail me at tim.smith [at] ed.ac.uk with the subject “[EDS] registration” and include the following content in the body of your e-mail:

Name
Position e.g. Research Fellow
Affiliation e.g. name of University or business
Location
Short summary of your research interests.
Existing eye tracking equipment you use.

Together we can make the experience of researching eye movements in dynamic scenes pleasurable, practical, and painless.

Kent Presentation

Quick plug.

I’ve been invited to give a presentation to the Centre of Cognitive Neuroscience and Cognitive Systems at the University of Kent next Wednesday (21^st March, 2007). I’ll be presenting the results from my eye movement and film study that I discussed at SCMS and Madison but with the focus shifted to the science. If you happen to be a University of Kent student or in the neighbourhood you should com and check it out (details).

The following morning I’m giving a guest lecture as part of Murray Smith’s Cognition and Emotion in Film course. This sounds like a fantastic course. I wish there had been a similar course when I was an undergraduate. Murray Smith is a very active member of the cognitive film theory community and particularly the Society for Cognitive Studies of Moving-Images. His work on emotion and empathy in film viewing is very influential.

I’m looking forward ton bouncing ideas around with his students.

Seeing Spots

Thanks to David Bordwell for his very flattering comments about my presentation on his blog. David, and the rest of the Madison Communication Arts department were incredibly welcoming when I presented there last Monday (12^th March) and the session was an absolute pleasure.

Now that David has alluded to my findings and given a brief description of my presentation you’re probably interested in finding out more. Sadly, I’m going to have to ask you to watch this space just a while longer. As is the way in academia, the publication of academic papers takes a long time and the paper that describes my findings is not yet ready for public distribution…..I know, I know: I’m a big tease. I promise you it will be worth the wait and I’ll publish the paper on my blog as soon as it is available.

In the meantime I can give you a glimpse of the “little yellow dots” David refers to.

This image is a screengrab of a software tool I created called (rather cheesily) Gazeatron. The tool allows me to plot the gaze position of multiple viewers on to the video they were viewing. The image above illustrates where 17 people were looking during this frame of the film (the yellow spots were viewers who could hear the audio and the pink were viewers who could not). Gazeatron allows you to see the same data in real-time as the video plays. By observing the swarming behaviour of the gaze positions whilst multiple viewers watch a film you gain an incredibly detailed insight into the viewing experience. Gazeatron also provides automated analysis for features of the eye movements to provide objective measures in supplement to the subjective observations.

Existing eye tracking tools do not allow you to analyse film viewing in this way and, I would argue reducing viewer attention to a film to static screenshots or data points does not give you a feel for the dynamics of the viewing experience. I’ll work on posting a video of Gazeatron so you can all see what I mean.

A bit of background on eye tracking. Each spot in the image above represents the point where a single viewer is looking. This is important as it tells us, roughly the part of the visual field they are attending to and, therefore processing at any moment in time. You may think you are aware of the whole visual field but in fact you are only able to process a very small portion to a high degree of accuracy at any one time. When you want to process a new part of the visual field you shift your eyes (perform a saccadic eye movement) so that the light from the new target is projected into the region of highest sensitivity in the eye, referred to as the fovea. These saccadic eye movements are very quick and we are not aware of them as our brains “stitch” the images either side together to create the impression of a stable visual world. By recording these eye movements we can infer the moment-by-moment experience of a viewer.

Eye movements can be recorded using a technique known as Eye tracking. There a variety of ways to track somebody’s eyes such as sclera coil and dual-purkinje (some clearly more scary than others). The most common technique used today, and the one I use is Corneal-Reflection. These trackers shine infrared lights onto the eye and film the reflected image using an infrared camera. By locating the iris and the infrared light reflected off the cornea the gaze of the viewer can be calculated. The gaze is simply a vector pointing out from the viewer’s eye into space. Therefore, eye trackers can be used to tell us where people are looking on a computer screen, table top, real-world interaction or….whilst watching a film.

The eye trackers I use are the Eyelink II head-mounted and the Eyelink 1000 tower mounted tracker, both from SR Research. These trackers are located around the University of Edinburgh, mostly in John Henderson’s lab which I am part of.

Tracking a viewer’s eyes whilst they watch a film is not as simple as you might think. The Eyelink trackers all come with software that allows you to present videos but they do not, currently have accompanying tools for analysing the eye movement data in the way I’ve described above. Most other trackers do not provide assistance in presenting films and a lot of previous researchers have resorted to tracking viewers using a head-mounted real-world tracker and recording a video to see what they are looking at (a similar technique is used in driving studies). The only other tracker I have used that is suitable for presenting films is Tobii. This system is incredibly easy to use as it is focussed at usability studies and as a hands-free interface for disabled users. The Tobii eye trackers are incredibly well designed but their price, ~£17,000 puts them out of the reach of most users (the price issue is the same with all eye trackers). Their accuracy is also not as good as the Eyelink systems which is why most vision researchers don’t use them.

If you’re looking for a cheaper option there is the option of building your own eye tracker. Derrick Parkhurst has developed open-source software and instructions for how to construct the necessary hardware to build your own eye tracker. The openEyes project is a great idea although I’m yet to have a go. If you have a go, best of luck and tell me how it goes.

If anybody has any further questions about eye tracking and film please either post a comment below or e-mail me.

As for what I have found by eye tracking film viewers well……that’ll still have to wait. Sorry. For the time being I hope you enjoy the picture of little yellow and pink spots. Who’d have thought seeing spots could be so useful!

SCMS and State Street Brats

I am writing this blog post from a hotel room in Madison, Wisconsin (note: the picture above is not the hotel I’m staying in. That’s the Chicago Hilton; more later). Why am I in Madison? Other than the fact they have great frozen custard, Afghan, Mexican food and hotdogs (http://www.statestreetbrats.com/), I am here to present my eye tracking research to the Department of Communication Arts at the University of Wisconsin-Madison. If you are at all interested in Cognitive Film Theory you will be aware of Madison’s Comm. Arts department. They began teaching cognitive film theory before most of us were even aware that such an approach existed. People like Ben Singer, Jonathan Frome (now of Uni. Central Florida) and, of course the hugely influential David Bordwell (yes that guy who’s books you are always being told to read by your film teacher/lecturer/film geek friend/me). The department is so infused by cognitivist ideas that it was an absolute pleasure to present my research there. The intense discussion that my presentation invoked was incredibly invigorating. Its great to get such an intrigued and welcome reception of, what is essentially cognitive science research (although, research focussed on film). I look forward to many future exchanges with this group. I’m probably not supposed to mention this as I’m not sure if the details are finalised but, David Bordwell and the Department of Communication Arts will be hosting the Society for Cognitive Studies of the Moving-Images conference next Summer (2008) so I shall be eagerly returning to Madison next year. I also want to encourage as many other people interested in issues related to cognition and film, whether from arts, humanities, social sciences, sciences, or anywhere to attend this conference. It is going to be a hoot J.

Thanks to The Comm. Arts department for being so incredible hospitable, special thanks to Jeff Smith for being my guide and David Bordwell for being so receptive to my imposition.

Now some background: the reason why this visit to Madison was possible was because I was presenting at the Society of Cinema and Media Studies (SCMS) conference in Chicago last week. The conference is one of the foremost gatherings of Cinema, Television, and Media theorists from around the world. I attended two years ago when it was held in London and my appetite was whetted. This year I presented a paper about my recent eye tracking examinations of different editing techniques across a range of films. The paper was really well received and a lot of people were very intrigued by the potential for eye tracking as a tool in their analysis of film. The technology is getting close to the point that most researchers would be able to perform the kind of analysis I do on films. However, there are a couple of key components such as analysis and visualisation tools that are currently missing from most commercial eye tracking systems that would be required for the technique to really take off. I have developed my own tools that allow me to fill these blanks but most researchers would not be able to do this. And, of course the cost of most easy to use eye tracking systems is still prohibitive.

Who knows, maybe the technology will suddenly take both a cost and technological leap forward and it’ll become accessible to all. Watch this space……

Returning to SCMS, the conference was held in the Chicago Hilton (very swish….well the lobby is anyway; the conference presentation rooms/bedrooms are a tad odd). I was a complete conference geek, attending almost every session. Considering that the days ran 8:15am-8pm this is quite an achievement. The reason I attended so many sessions was because of the incredible range of interesting presentations. Everything from a bit of Cognitive Film Theory (Jonathan Frome, Joe Kickasola, Mark Minett), masses on New Media, Interactive Media, and Videogames, emotions and film, including discussion of automatic facial expression recognition (Kelly Gates), and even a presentation on the Queering of Kevin Smith (it doesn’t take much ;)…Carter Soles). This year there seemed to be a lot of presentations on the impact on-line distribution, web video, and interactive TV and media such as videogames were having on our classical theories of film and television. Fascinating stuff. One of my most satisfying panels was debating the implications of interfaces for interactive TV content e.g. TIVO and PVRs, and their effect on our relationship to the film/TV content. Does the interface, which is meant to empower the viewer by allowing them access to the content actually compete with the content itself?

So, all-in-all a great conference and trip to Madison. I look forward to coming back in 2008.

Examples of Automavision and a proposal for Automavision 2.0

Following on from my discussion of Lars Von Trier’s Automavision and Lookey I thought you would like to see some examples of Automavision. David Bordwell, the author of a phenomenal number of outstanding books on the subject of film has written a very interesting blog post on the subject here. He has had the good fortune of viewing The Boss Of It All, unlike myself and capturing some screenshots. The effect is intriguing. Automavision appears to create unmotivated, and classically imperfect framings which Von Trier accentuates by cutting rapidly between very similar shots. Bordwell notes that the result is that almost every cut is a Jump Cut, a violation of the 30 degree rule that causes the image to jump uncomfortably and creates ambiguous temporal relationships between the shots. The same effect occurred in Dancer in the Dark (see a video example here) and can be said to have contributed to the overall discomfort felt by viewers of the film.

What is most interesting about Von Trier’s use of Jump Cuts is that, whilst they abandon the classic continuity style’s preservation of temporal continuity within scenes they still retain a clear cohesion that allows the viewer to understand the action represented. It is almost as if Von Trier, in consciously violating the dimensions of continuity prescribed by the classic continuity style he is revealing extra dimensions of continuity that he uses to create extra significances within his films.

Also, as noted by Antithesis Boy in his comment on my last post the Automavision technique shares a lot of similarities with virtual camera control in videogames. Automavision uses a computer to randomly generate framings for the camera and the result is non-classical framings. Videogames often take place in a 3D virtual environment and require a virtual camera to follow the action within this space. This camera is computer controlled and the objective is to create the best framings possible (e.g. Jhala and Young, 2006) but the result is often unacceptable and bizarre framings. The difference between the two systems is that a Virtual Camera is positioned relative to the objects within a scene where as Automavision (as far as I can make out) does not care what the scene is. This is why Automavision loses significant objects off the edge of the screen or frames them oddly. In his blog post, Bordwell notes that Von Trier does not always choose the most outrageous framing generated by Automavision indicating that he realises that the unconventional framings have a particular effect on the viewer and should be mixed with more conventional compositions to create the intended viewing experience. If Automavision is truly random, as implied by Von Trier then there is no way to control the degree to which the resulting framings are unconventional. This means that Von Trier must keep refilming and regenerating framings with Automavision in order to get suitable shots.

The next generation Automavision system could improve its ability to generate unconventional framings by incorporating some of the intelligence of Virtual Cameras. If it is able to apply the classical framing conventions then it can knowingly violate them. This would also allow it to modify the conventionality of its framings by varying the relative influence of chance and the framing conventions. To do this it would need to begin processing the visual scene, which is no easy task and deciding where the significant objects should be positioned in the frame. An interesting reverse application of this technique would then be to apply Automavision 2.0 to a videogame. Anybody up for a game of Halo directed by Lars Von Trier?

It Lookey like Lars Von Trier is at it again.

Not content with creating a revolution in filmmaking by spearheading the Dogme 95 movement, Lars Von Trier is now experimenting with how his films are shot and how viewers engage with them. His new film, The Boss Of It All, is filmed using a new camera control technique developed by Von Trier called Automavision. The system removes the need for a human camera operator, replacing them with a computer which randomly selects camera shots and movements based around an initial camera position selected by the director. The system removes the usual control the director and cinematographer have over the composition of each shot and, specifically it’s framing. Lars Von Trier explains his desire for developing this technique:

“I am a man who likes to control things, and if I can't control them totally I will not control them at all. After doing Europa with very very fixed shots and camera movements, I was tempted to do something totally different. I started using a handheld camera and we invented a form of framing, or non-framing, called pointing of the camera, because I hate framing.” (guardian.co.uk)

This hand-held, almost haphazard method of framing combined with non-continuity editing had recently become Von Trier’s trade mark. Even after he deviated from the original strict edict of the Dogme 95 manifesto with Dancer in the Dark, Dogville and Manderlay he retained this style of framing. The style deliberately breaks with the traditions of the Hollywood continuity style by forcing the viewer to actively search each shot for the most significant elements. In the Hollywood style these elements, for instance a protagonist would be centred within the frame and lit in a way which made them highly salient even in busy scenes. In Von Trier’s recent films the protagonist is often cut off by the frame, visually diminished by other less important elements, or moving at odds with the hand-held camera movement. In combination with his often uncomfortable subject matter, this unpredictable framing technique often leads viewers to describe the experience of watching a Von Trier film as “hard work”.

In a series of recent eye tracking experiments I have shown that this difficulty in following the action of a Von Trier film can be very clearly seen in viewers’ eye movements. When watching a film composed according to the classic continuity style, all viewers will focus their attention on a small number of objects within a shot. In most shots there will only be one clear centre of attention, usually the face of a principle actor and it will be this that all viewers track within the shot and across cuts. By comparison, viewers watching Dancer in the Dark or Dogville distribute their attention across more of the screen and show less agreement of what they believe to be the most significant object. When a cut then happens (which they often do at unexpected moments in Von Trier’s films) viewers are not guided to the new centre of attention by the director so they have to actively search the scene. This active engagement with the visual constituents of the film creates a viewing experience that is completely counter to the normal smooth, direct, almost passive viewing experience of a classical continuity film.

Von Trier’s desire to create films that actively engage viewers can also be seen in his use of Brechtian theatrical technique. In Dogville and Manderlay the use of black backdrops, minimal props, and transparent scenery expose the artificiality of the film. Bertolt Brecht developed these techniques, amongst others, as a way of encouraging his theatre audiences to adopt a critical mindset. By actively counteracting the “suspension of disbelief” encouraged in classical Artistotelian theatre, Brecht was trying to engage his audience in the critical interpretation of the depicted action, the act of its construction, and its place within real-life. The application of these techniques to cinema by Von Trier also resulted in an extra level of visual engagement beyond that created by the non-framing camerawork. By populating the set with transparent scenery the director is unable to hide insignificant actions. All actors must be present on set at the same time and act even though their actions are not important to the current shot. These peripheral actions crowd in on the main action, drawing the viewer’s attention away from the main action and creating further disagreement between viewers.

However, the natural instinct for framing is hard to overcome and it appears that Von Trier realised that his desire for completely un-framed shots would not be possible so long as he or his camera operator were controlling the camera. A hand-held camera is often described as a visual prosthesis: an extension of the camera operator’s eye; seeing what they see. The movement of a hand-held camera may be rough and the framing imperfect but, like the human eye it will always eventually settle on the most important parts of a scene. If Von Trier is to create shots in which viewers are unable to predict what is the most significant part of a scene or how the camera is going to move he needs to take the human camera operator out of the equation. Hence, Automavision.

Automavision is not the only innovation in Von Trier’s new film, The Boss Of It All. Von Trier recently announced that he has embedded five to seven “Lookeys” in the film:

"For the casual observer, it's just a glitch or a mistake….For the initiated, it's a riddle to be solved. All Lookeys can be decoded by a system that is unique." (Von Trier quoted on Screendaily.com)

Von Trier is offering 30,000 Danish kroner (£2,700) to the first Danish viewer that identifies all the Lookeys. The Lookeys are described as “visual elements that are out of place” (www.lookey.dk) and are intended to turn the film into a “mind game”. By informing his viewers of the presence of these Lookeys Von Trier is again encouraging his viewers to actively engage with his films in a way in conflict with the normal film viewing. Spotting continuity errors, which is how these Lookey’s would be described if they were unintentional, has been an occupation of film viewers throughout the history of film. The pastime has escalated to such a level that there are even books and websites devoted to it. Continuity errors are typically mistakes made during production that are spotted by viewers on repeated viewings of a film. The most common errors are unintentional costume changes across shots or cigarettes and drinks that disappear or refill.

Finding errors is always good fun although it can often be very difficult. In my Ph.D. thesis I created a taxonomy of continuity errors that classifies them according to when they are made during the film’s production process and what is required to spot them (page 186 of my thesis). The detection of errors is very dependent on how the viewers watch the film. If the viewer looks at the parts of the screen the director wants them to look at they should never be aware of any continuity errors. The focal objects (those at the centre of attention) should never have errors as they would have been spotted by the director, cinematographer, or editor. Errors are more likely to be located in the periphery of the screen, areas where the production crew and the average viewer are unlikely to look. As film is a dynamic medium and a viewer can only focus their attention on one small part of the screen at any one time, peripheral errors should be missed as viewers are rationing their attention to the most significant parts of the screen.

In a traditional Hollywood film composed for continuity, errors are hard to spot as viewer attention is so carefully influenced by the director and editor. In a Von Trier film, as already discussed, the intention appears to be to encourage active search of the screen and disagreement between where different viewers look. With the addition of Automavision the likelihood that a viewer looks at a part of the scene that would traditionally be insignificant probably increases. This may mean that viewers of The Boss Of It All may be more likely to spot the Lookeys than if they had been included in a traditionally composed film.

However, without knowing exactly what form these Lookeys take (see page 186 of my thesis for a taxonomy of errors) or what expectations the viewer will have to have to realise that the Lookeys are errors we cannot know whether they are easier or harder to detect than traditional continuity errors. What we do know is that the experience of watching The Boss Of It All is, like in all Lars Von Trier films going to be unlike watching any other film.

Re-viewed

Last Monday (23/10/06) I had the good fortune of attending the opening of a new interactive art exhibit at the Leeds Metropolitan Gallery. The exhibit is called Re-possessed and presents a collection of interactive art installations inspired by Alfred Hitchcock’s Vertigo. Each installation takes a section of the film and provides the participants with novel ways of exploring the film, their experience of it, and the issues of power, gender, and voyeurism that are so prominent in the film. The exhibit runs until 18^th November and I highly recommend it if you are interested in film or just want to play with some highly innovative interactive exhibits.

The main reason I was invited down for the opening was because one of the artists, Richard Stevens had stumbled across my research whilst looking for references about applying eye tracking to film. His installation, Re-viewed uses a Tobii eyetracker to record viewer’s eye movements whilst watching scenes from Vertigo and then represents the eye movements using some very interesting visualisations that Richard and his students have created. These visualisations give the visitors an insight into their perceptual experience of film that they have never previously had. It also emphasises group differences such as between Male and Female viewers. Given that gender politics and the resulting power relationships are very prominent themes in Hitchcock’s work (see Laura Mulvey’s article on the Male Gaze in her 1975 book “Visual Pleasure in Narrative Cinema”) this method of using eyetracking to extract the gender differences in the viewing experience could prove very informative.

The reason why Richard contacted me is because he and his collaborator, Tony Renshaw were uncertain how to interpret the eye movement data given that so little work has been done in relation to film. They also needed some clear way of expressing any group differences that they observed. Currently, Richard’s visualisations can display the moment by moment differences but in order to report and quantify these differences precise measurements are required. I have encountered similar problems with my own eye tracking work and have been developing suitable measurements. Hopefully I will be able to aid Richard and Tony in the analysis of their data and we will find some interesting group differences. We’ll then work towards publications based on the results. I’ll keep you posted if anything emerges.

I am very excited about this exhibit as, to my knowledge it is the first time eye tracking of film has been performed on such a large scale with a large cross-section of the population. Most eye tracking studies (all Psychology studies for that matter) are usually conducted on University undergraduates; the most easily accessible subject pool for university researchers. The number of subjects eye tracked is also usually less than 20 so there is no scope for exploring group differences. Hopefully, through the development of new measurement techniques and automation of the analysis we can extract some interesting results from this study which will further our understanding of the viewing experience.

So, if you are any where near Leeds in the next two weeks head down to the Leeds Met Gallery. You’ll be furthering scientific knowledge whilst having a fun and informative time.

End of an era + Catch up

Today I moved out of my PhD office. Even though I finished my thesis in January, graduated in June, and have been working on a completely different project (Le Active Math) since last October I had somehow managed to remain at my old desk. Today marks the start of Edinburgh University Fresher’s so new PhD students were coming to inhabit my office which meant that I finally had to cut my ties with my old desk. So sad….so many memories….and too many long hours spent at that desk.

My departure from the office was eased by the fact that most of the office mates I had shared it with during my Ph.D. had also left. Scott Nowson started his new life at Macquarie University in Sydney three weeks ago (you can check on his progress at http://nowsonexitmusic.wordpress.com/). Sarah Gingell finally finished her Ph.D. (after too long to record here) last year and has moved on to a real life. Colin Fraser went part-time on his Ph.D. and is now touring Europe promoting e-Democracy (you’ll have to ask Colin to explain what this entails). Zoe Bruce (and Morgan the Dog) left a couple of years ago to pursue a career in academic publishing (Zoe that is, not Morgan). We all had some great times together. In fact we may have had too many great times considering how long our research ended up taking J. Still, I wouldn’t have traded you guys in for anything. Thanks for keeping me company.

Now I find myself divided, literally. I’ve got a “hot desk” (what a cool name for what is essentially a desk that anybody can sit at) in the same building in Informatics which I shall be using most days for my Le Active Math work. I also have a very nice office all to myself in John Henderson’s lab in Psychology. That is where all the interesting blog-relevant work will take place. I’m really excited about starting full time over there but this won’t happen until January. In the meantime I have to run a MASSIVE evaluation of Le Active Math….. Wish me look.

Blog posts I should have written and still might:

• Cognitive Studies of the Moving Images Conference, Potsdam, Germany (20/7-23/7/06). Fantastic conference, great people, interesting research, and fabulous reception for my work. The future of Cognitive Film Theory is bright. See Bruce Hutchinson’s account of it at bdhutch.blogspot.com.

• Narrative and Interactive Learning Environments conference, Edinburgh (8/08-11/08/06). I really enjoy this conference as it attracts such a small, friendly, yet hugely intelligent group of people. I was one of the organisers this year and found myself stage managing and being sound and lighting engineer for a multimedia performance group called Palindrome. These guys do visionary things with motion capture and real-time music production. We were really lucky to have them take part in NILE.

• Active Vision workshop, Dundee (1/09/06). This is a great little workshop organised by Ben Tatler. Lots of interesting presentations about eye movement research from a diverse range of disciplines. The atmosphere was really relaxed which resulted in a lot of discussion and humour. This workshop was a great example of how many interesting eye movement researchers there are across the UK. I’m really looking forward to being more “active” in this community (Sorry, I couldn’t help it)

• Edinburgh International Film Festival (14/08-25/08/06). God I love living in Edinburgh! August is a month of cultural, artistic and pleasurable excess that always amazes me. The high point for me is always the film festival. I see an unhealthy number of films each year and 2006 was no difference. Highlights include Brothers of the Head, Black Sheep, Host (Gwoemul), Snow Cake, Clerks 2 (and attending a Q&A with Kevin Smith), and Looking for Cheyenne (Oublier Cheyenne). Thanks to Shane Danielsen the festival director who programmed a fantastic final festival. You shall be missed. Especially your bizarre digressional Q&A style and uncontrollable flirting. Fantastic.

Actually I think I just ended up writing those blog posts!

You can’t get rid of me that easily.....

I am very happy to announce that I shall be continuing my research for at least another year. I have been accepted for a post-doctoral research position in John M. Henderson’s Visual Cognition lab in the Edinburgh University Psychology department. Now I know what you’re all thinking: “I thought he was already in Edinburgh Uni”. You are correct. My current status is as a Research Associate for the Le Active Math project employed by Moray House School of Education at Edinburgh University. However, my move to Psychology will be a very fortuitous one as it will allow me to conduct my research without distraction and within a group of likeminded people.

This move to psychology has only been made possible by the uncannily well-timed move of John Henderson’s visual cognition lab from Michigan State University to Edinburgh. John and his partner Fernanda Ferreira were offered chairs in our psychology department and they were wise enough to accept. In the simple signing of a contract Edinburgh has suddenly become a hot-bed of eyetracking and visual cognition research. John, for those of you who don’t know, is one of the foremost researchers in the field of visual search, scene perception, and visual cognition in general. If you read my thesis you will find that he is one of my most cited people. His work is highly influential and I am sure that it will continue to be so from his base in Edinburgh.

John has been very encouraging of my research and interested in the theories I have been developing. As a post-doc under John I shall be continuing my research into the use of attentional cues in film to create the perception of continuity. I will also be expanding my theoretical, methodological, and writing skills to begin establishing myself as a visual cognition researcher.

Thank you, John for this opportunity.

So it looks as if you (reader) won’t be getting rid of me or this blog for a few years to come :)

Rock ‘n’ Roll Research

or “How to do an experiment in two days”.

I was contacted at the start of July by a Gustav Kuhn from Durham University. Gustav is doing some really interesting research looking at how magicians use misdirection and perceptual expectations to make you perceive something that actually isn’t there (Gustav is a highly talented magician himself. He’s available for children’s parties, corporate events, bar mitzvah’s …..). He’s done a series of eyetracking experiments to examine where people look whilst watching magic tricks. To satisfy a reviewer of one of his papers he needed to run an eyetracking study and he needed to do it fast. *Cue the Mission Impossible music*

Unfortunately, Gustav’s eyelink II system was not setup yet so he asked if he could run his study using the equipment we have here. I agreed with three small provisos:

1) the study had to take place after the 11^th July as I was at a project meeting in Germany before then;

2) I would have to run some of my own stimuli on the subjects whilst they were being eyetracked;

3) the study had to be finished by the 14^th July as I had to fly to Germany (again) to present at a conference (CCSMI, more later).

In the course of 3 days I managed to prepare two sets of stimuli for my part of the study, design and implement the experiment using the Experiment Builder software which I had never used before, test, pilot, and run 13 subjects through the experiment, and process the data so that Gustav could do his analysis. Phew! Gustav was up in Edinburgh for a day and a half and in that time we managed to get all the data we needed and, through subsequent analysis, Gustav got the result he needed to appease his reviewer. Now that’s how science should be :)

Unfortunately, the pace for my part of the experiment has slowed down somewhat. I tested two types of stimuli. The first were the videos I used in the editing memory experiment. I wanted to see which details of the videos viewers were using to detect the editing discontinuities. I also had some hypotheses about how the discontinuities would effect their eye movements (see d'Ydewalle, G., Desmet, G., & Van Rensbergen, J., 1998 for similar effects). To extract these effects I need to examine each video by hand and then perform some complex statistics on the eye movement data. Sadly this can not be done in a day L I’ll publish these results in the Editing Memory sometime soon.

The second stimuli I eyetracked were feature films. I have been desperate to get eyetracking data for films with different degrees of continuity for ages and Gustav’s urgency finally motivated me to do it. In my thesis I develop a series of hypotheses about how eye movements should be controlled by an editor in order to create the perception of continuity. This data should provide direct evidence of these techniques (if they exist) and motivate further, in depth studies. The films I eyetracked are also interesting to film theorists as they cover the most significant styles of films: Blade Runner (Continuity), Citizen Kane (Deep Focus), Koyaaniqsatsi (Non-narrative), Dancer in the dark (Dogme-esque), Eisenstein’s October (Dialectical Montage), Hulk (Digital Composition/Collage), Requiem for a Dream (Quick Cutting/MTV style), and a few more.

I’m really excited about the findings of this study and I’ll keep this blog updated as to what I find. Unfortunately, analysis of eyetracking data for long sequences of feature films is not easy and there exists no tools to assist. I’m currently developing my own methodologies, tools, and analyses to make this possible. Fingers crossed everything goes to plan and I have some interesting results to present asap.

So in conclusion, high speed experimentation is by far the most Rock ‘n’ Roll way to run experiments. Unfortunately, there is nothing Rock ‘n’ Roll about analysis. (Any suggestions on how I can make my analysis “rock” are highly appreciated)

Corrected version of my Ph.D. thesis submitted

I have just resubmitted my Ph.D. thesis in its final hard-bound version. When I had my viva (see Viva! Viva) back in February I was awarded a pass with minor corrections. The corrections amounted to the modification of a few graphs, fixing of typos, rephrasing of a few sentences to make things clear and general reformatting. Not much really. I finally got round to making the changes to my thesis and I formally resubmitted the thesis last Monday (9^th May). I have also registered my thesis on-line with Edinburgh University’s Research Archive. The archive is a wonderful, publically accessible repository of research documents produced at Edinburgh. You can find my thesis here.

I have also updated the version linked from my website. Now everybody can read what I have been up to for the last 4.5 years and be confident that reputable researchers have given it their seal of approval.

Now all I need to do is channel some of the ideas stemming from my thesis into research proposals for my future career. Too many ideas, too little research funding/job offers J

Science Festival Fun

This is just a quick note to say that I am currently running activities at the Edinburgh International Science festival (April 11^th – 15^th, 2006) at the National Museum of Scotland. There are two activities: Editing Memory and Directing Reality.

Editing Memory is a drop in activity in which you get to test your memory for films by reconstructing the film using editing. It is proving to be very VERY popular (my helpers can barely keep up!). Come and have a go (its free) if you want to test you memory or try editing a film. You can visit the website for more information and the eventual results: http://scifest.inf.ed.ac.uk/editing-memory/

Directing Reality is a 1-hour workshop in which kids (and big kids) can come and make Trick Films using really simple filming techniques. The films the kids are making are just like those produced by George Melies over a hundred years ago. The intention of the workshop is to get the kids thinking about the expectations and assumptions we make when perceiving the visual world and how these can be violated to create funny illusions using editing. The kids pick it up really quickly and the tricks they come up with are marvellous. If you want to come and have a go get down to the museum before Saturday although you might want to book on-line before hand (http://www.sciencefestival.co.uk/) as tickets are selling out fast (they cost £3).

Hope to see you there :)