Samantha Moore – Animating unique brain states

The animated documentary and ‘psychorealism’

The starting point for this paper is ‘psychorealism’, a term which Chris Landreth coined for the way in which animation can depict internal realities; I will discuss my interpretation of psychorealism, and apply it to the making of the 2010 film An Eyeful of Sound.

In 2004 Chris Landreth won the animated short film Oscar® for his film Ryan, an animated documentary about his interviews with the animator Ryan Larkin. Barbara Robertson (2004) notes in an interview with Landreth, that at first he began taping interviews with Larkin in a fairly standard documentary fashion with a view to making something along the lines of Aardman’s Creature Comforts films.

However,

“In August, we had a conversation about alcohol,” Landreth remembers. “I hadn’t planned to bring the subject up, but I needed to, and it came out the way you hear it in the film, verbatim. At that point, I kind of knew the film was not going to be just about Ryan.” The interviewer, whose mother had alcohol problems, had become enmeshed in the story; the conversation about alcohol became a climactic moment in the film.

The film developed into a moving portrait of not only Larkin’s life but also of Landreth’s own history and his developing understanding of it. It became an example of what Bill Nichols calls the interactive mode (1991) which, by allowing the audience to see the relationship between the interviewer and interviewee and draw our own conclusions about it, the textual authority is shifted towards the subject. Nichols says that the mode “introduces a sense of partialness” (p.44); we become aware of the power relationship between the two characters because we can see how it affects them. As the film progresses it becomes clear that Landreth has his own agenda and, as Robertson suggests, the section when that agenda becomes explicit is a key point in the film. However, Ryan does not only use the dialogue of the two subjects of the film (interviewer and interviewee) to tell us their individual and converging stories, but it actively shows us what effect their words have on them by using vivid visual metaphor combined with photo-realistic computer generated animation to demonstrate it. In his notes for the film Ryan, (Landreth in Singh 2005), Chris Landreth says;

What I’m most interested in is not achieving photorealism in CGI, but in co-opting elements of photorealism to serve a different purpose, to expose the realism of the incredibly complex, messy, chaotic, sometimes mundane, and always conflicted quality we call human nature. I refer to this as ‘psychorealism’.

Landreth’s definition of psycho-realism is a reaction against the dominant push towards making 3D computer generated animation as photo-realistic as possible with the inherent dangers associated with falling into the ‘uncanny valley’1 and repulsing the viewer rather than drawing them in. In the case of Ryan and his subsequent film The Spine (2009) Landreth uses techniques more readily associated with achieving photo-realism to create visual metaphors, which inform and comment on the characters behaviour. For example his own fear of failure as an artist is articulated in Ryan through the stringy, muscular, multi-coloured ties which wrap themselves round his head and body at different points in the film. His definition of psychorealism, then, is a kind of bastardised photo-realism re-purposed to serve a subjective perspective through visual metaphor. The images flesh out the words; they stem from the indexical sound but they are not the same as it. They do not merely illustrate, they hold their own information as Sybil Delgaudio argues, “animation itself acts as a kind of ‘meta-commentary’ within a documentary” (1997, p 192).

The use of extended visual metaphor might be seen as one more step away from the profilmic event, another layer of mediation which takes the viewer further away from reality. It can be argued though that the use of visual metaphor creates a “visual resonance” (Ramachandran 2011, p 237) in the viewer, by presenting information which is understood by the instinctive right brain hemisphere long before the logical processing of the left hemisphere can articulate it. Ramachandran claims that visual metaphor is one of the aesthetic universals; a set of visual ‘rules’ by which the human brain processes the visual. In this context then, the use of visual metaphor in Ryan (and many other animated films) is not so much a design choice as a way of directly tapping in to the audiences’ understanding of the information conveyed. When Landreth talks about human nature there is nothing logical about it (“complex, messy, chaotic”) so it makes sense for him to choose a method of conveying this that appeals to the instinctive, non-logical part of our brains first.

On first reading the term psychorealism I was excited by what it represented in relation to my experience of animated documentary as a practitioner. It seemed to encapsulate a way in which non-fiction animation can side step the well-rehearsed arguments about indexicality and get to the point of what animation can do so well in a documentary context, which is to represent the internal. What Landreth is doing can certainly be said to be in the interactive mode but can also be said to sit squarely in both the subjective (Wells 1997) and penetrative2 modes; taking a subjective point of view and visualising what is beneath the surface. This made sense to me as I was working on a film about audio-visual synaesthesia that needed to visually articulate a perceptual processing unique to an individual and not experienced by most of the audience. Visual metaphor could have no place in the treatment of this material since the synaesthetic reactions themselves are entirely abstract to the non-synesthetic eye; they only represent themselves. What this film was doing was a contrast to Ryan; instead of using metaphor as a short cut to convey generally understood internal feelings, I was visualising unusual internal feelings as literally as I could and then trying to explain them with context. However both approaches dislocate elements of realism and re-place them into the innately fantastical realm of animation. Landreth uses the tropes of photo-realism, I use a two dimensional painterly style. Both place themselves simultaneously outside and inside the world. Bill Nichols makes a nice distinction between fiction, which is “a world” and documentary, which is “the world” 1991, p 109). The definition of psychorealism that I have extrapolated from Landreth’s term is a way of representing the ‘truth’ of an internal experience without being wholly dependent on photo-realism. The psychorealistic then exists in a liminal space between the external, objective and the internal, subjective.

Two films that also demonstrate this approach are Paul Vester’s 1995 film Abductees, about a group of people who believe that they have been abducted by aliens, and Tim Webb’s 1992 A is for Austism, both non-fiction animation that deal with the internal. Both films use the perspective of the interviewees through their own drawings as well as their spoken testimony, and the authenticity of these documents as ‘real’ translations of their experiences is more compelling than a solely photo-realistic treatment. Although both films use elements of photo-realism too they do so in a non-traditional way; re-placed into a slightly different context. Vester includes video testimony of the interviewees but uses a jerky, degraded treatment of the footage which makes much less of a contrast between the animation and the live action. Webb uses the live action in a stop-motion context where live action people are seen interacting with animated elements, conforming the fluidity of live action into the staccato dynamic of animation.

I tried to draw on my interpretation of ‘psychorealism’ when making An Eyeful of Sound3 (2010), an animated documentary about audio-visual synaesthesia. The film was a collaboration between the author (University of Wolverhampton), a group of people with synaesthesia and neuro-psychologist Dr Jamie Ward at the University of Sussex4.  Synaesthesia is a brain condition, when one sense is stimulated, more than one reacts. We worked with audio-visual synaesthetic people who have a colour/shape/movement experience along with sound (see Fig. 1 for an example of this).  Any two senses can be linked in this way (and for some people more than two senses can be linked, or senses can be linked both ways, for example colour from sound and sound from colour). Synaesthetic people might have coloured days of the week (e.g. Monday is blue, Tuesday is chartreuse etc.), coloured letters, words or numbers5 (grapheme-colour synaesthesia), they may smell or taste sounds6 or even feel touch when they see others being touched (mirror-touch synaesthesia). A suggestion for why this might happen is that in a synaesthete’s brain senses may have more neural connections (Harrison 2001 p.20-22) than in a non-synaesthetic brain, so they will experience a second (third, fourth…) sensory perception simultaneously with the one being externally stimulated.

The film attempts to show the audience what it is like to have synaesthesia from the inside out. The experience that it shows cannot possibly be captured photo-realistically since the perceptual processing that produces synaesthesia is entirely internal, and unique to each individual. Therefore it uses digital animation, both 2D and 3D along with some elements of stop motion7, to convey the synaesthetic reactions as closely to the real experience as possible. Of course, as with all brain phenom, there is no profilmic event to compare the veracity of the interpreted outcome with. The only way to check whether or not what we were animating was ‘realistic’ was to have a system of checking and re-checking the material with the interviewees (see Fig. 2). We catalogued the synaesthetic experience in several ways, so that we could cross-reference the material. We used a Munsell colour chart, drawings and paintings done with various art materials, and voice recordings of the synaesthetic subjects’ reactions as they listened to the individual sounds (the sounds would eventually be used on the sound track but the audio descriptions would, on the whole, not). This was then translated into digital still images and, once it had been checked by the synaesthetic person as being correct, into moving image (see Fig. 3).

There were several problems encountered when collecting and verifying material. All the synaesthetic people we interviewed found it easier to react to sound when it was part of a more complex piece of music, rather than individual sounds (see Fig. 4 for a drawing of a minute of mixed soundtrack). This was surprising8 and made it trickier to review which bit of visual information went with which sound. Sometimes the interpretation was just wrong, as Julie Roxburgh’s clarification of an animated section shows;

The beginning is fine, but really it would be better just to have a thick white cloud moving continuously from left to right with the little silvery things as they are. It is the flower-like images which are wrong. What is the final sound of silver? It looks like a very thin vertical pole. Black balls are not there in the music at all!9

I also layered different synaesthetes’ sound/visuals together at times, making artistic decisions about what worked better or was more visually interesting. This directorial decision making was the point where the shared input had to end – ultimately as director I held power over how the final film would be presented although I tried to make this as transparent a process to the interviewees as possible10 so that they would still be able to contribute if they wanted to.  At the end of the process the synaesthetic interviewees were extremely positive about the final film. They felt it represented not just their individual synaesthetic reactions but gave a more rounded over view of the condition. Emma Suddaby felt that the film was made “very synaesthetically” and Tessa Verracchia reported that her non-synaesthetic husband finally understood the condition more clearly after watching the film.

Paul Ward (2005) argues that animation is good at representing the internal and subjective in the documentary genre, saying that it “can perfectly trace the contours of … a shifting and rapidly condensed thought process in a way that is out of reach for live action” (p.91). Ultimately then not every kind of reality can be represented by photo-realism. There are more resonant alternative ways of communicating a reality to an audience, and animation in particular is able to make the unlikely (exuberant, dramatic, colourful, fantastical) worlds of our internal lives manifest within its own context. By dislocating elements of realism, whether through the use of 3D computer generated imagery or by drawn representations of the world (see Fig. 5 for an overlaying of synaesthetic reaction on a ‘real’ landscape), and re-placing them into the realm of animation, ‘psychorealistic’ animation allows the audience to engage with the material differently. It is simultaneously real and fantastical, much like our inner lives.  As Chris Landreth says, it can express “the realism of the incredibly complex, messy, chaotic, sometimes mundane, and always conflicted quality we call human nature.” (Singh 2005, p 22). Psychorealism allows the audience to get behind the eyes of the subject, explaining and evoking their unique perspective through sound, visual metaphor, and interpretation of internal experiences, rather than a straight illustration of the audio track in an “illustrated radio interview” (Dreissen 2007 p 1). Bill Nichols asks, about the interactive mode of documentary, “how far can participation go? How are the limits beyond which a film maker cannot go negotiated?” (1991 p.45). Animated documentary’s contribution, through films such as Ryan, can only add to that debate.

Samantha Moore is an independent film maker who makes non-fiction animation. In 2010 she finished An Eyeful of Sound for The Wellcome Trust, about synaesthesia, which won the ‘Nature’ award for Scientific Merit at the 2010 Imagine Science Film Festival, New York amongst others.
She has animated and directed The Beloved Ones (2007) for the UK Film Council, doubled up (2004) for Arts Council England & Channel 4 and Success with Sweet Peas (2003). Her work has been screened and won awards internationally. She has written about animated documentary for various journals, co-authored a book chapter and given papers and talks at several conferences about her work on synaesthesia and animation. Sam is currently studying for a PhD at the University of Loughborough Animation Academy, is senior lecturer in animation at the University of Wolverhampton, and is a member of CADRE (The Centre for Art, Design, Research and Experimentation).
This paper was presented at Animation Evolution, the 22nd Annual Society for Animation Studies Conference, Edinburgh July 2010.

Notes
1 The term was coined by Japaense roboticist Masahiro Mori in 1970 to describe the eerie effect that making artificial humans look increasingly too close to reality elicits from the viewer. Chris Landreth has written about this on his blog here: http://films.nfb.ca/the-spine/blog/?p=120″http://films.nfb.ca/the-spine/blog/?p=120

2 The idea of the penetrative mode comes from a Halas and Batchelor quote Penguin Film Review (1949) cited and expanded upon by Paul Wells. Wells:1998 p 122

3 Funded by a Wellcome Trust Arts Award, 2007-10

4 This was a continuation from our R&D project ‘Synaesthesia and Sound’ made with the New London Orchestra in 2006.

5 See Duffy: 2001 pp 7 -15 for an autobiographical description of the authors developing childhood awareness of her synaesthesia

6 See Ward (2008) for a discussion of synaesthete James Wannerton’s taste-word synaesthesia

7 The technique used was dependent on the synaesthete’s reactions. After much experimentation we found that one of Tessa’s reactions could only be adequately represented by animating glitter glue on a green screen and then compositing it in.

8 My original idea was to capture each synaesthetic reaction to a set of 40 different everyday sounds and then make up a sound track with the composer using these sounds (and by definition the corresponding imagery) along with audio interviews with the subjects. It was not quite that easy.

9 Julie Roxburgh, An Eyeful of Sound [email] Personal communication June 2009

10 We used a blog to document the making process to which the other animators, scientists and synaesthetes were invited to contribute their experiences. We used a blog to document the making process to which the other animators, scientists and synaesthetes were invited to contribute their experiences: http://eyefulofsound.blogspot.com

References
Carter, R.1(998, 2010) 2nd edition, Mapping the Mind. London: Phoenix

Duffy, P. (2001) Blue Cats and Chartreuse Kittens: How synesthetes color their worlds New York: Henry Holt

Driessen, K. (2007) ‘More Than Just Talking Mice’ IDFA magazine [online] Available at: <www.idfa.nl/…/More%20than%20just%20talking%20mice%20(Paul%20Fierlinger).pdf> [Accessed 27 April 2011]

Harrison, J. (2001) Synaesthesia: The strangest thing, Oxford: Oxford University Press

Nichols, B. (1991) Representing Reality. Bloomington: Indiana University Press

Ramachandran, V.S. (2011) The Tell Tale Brain: A neuroscientist’s quest for what makes us human New York : Norton
Robertson, B. (2004) Psychorealism Computer Graphics World Issue: Volume: 27 Issue: 7 (July 2004)[Available at <http://www.cgw.com/Publications/CGW/2004/Volume-27-Issue-7-July-2004-/Psychorealism.aspx> [accessed 16th July 2011]

Singh, K (2005) Oscar Projections: An Interview with Dr. Karan Singh. In MITACS Student Advisory newsletter Vol. III, No. 2 May 2005

Ward, J. (2008) The Frog Who Croaked Blue: Synaesthesia and the mixing of the senses Hove: Routledge

Ward, P. (2005) Documentary – the margins of reality. London: Wallflower Press

Wells, P. (1997) ‘The Beautiful Village and the True Village: A Consideration of Animation and the Documentary Aesthetic’. In his (ed.) 1997 Art & Animation, Profile no. 53 of Art & Design magazine, London: Academy Group

Wells, P. (1998) Understanding Animation. London: Routledge

© Samantha Moore

Edited by Nichola Dobson

PDF To download this article as PDF, click here.

This entry was posted in Samantha Moore - Animating unique brain states and tagged , , , , , , . Bookmark the permalink.

One Response to Samantha Moore – Animating unique brain states

  1. Baoliang Peng says:

    very good article with cross-disciplinery approach.

Leave a Reply

Your email address will not be published. Required fields are marked *