Peter Moyes & Louise Harvey – She’s Not There: When New Illusions Meet Ol’ Time RealTtime; Mo-cap, Virtuality and Live Music Performance

(Animation) began as something other than drama. It was not a story in three acts; it was a singular event, like an implausible trapeze act …
(Norman Klein, 1993)

“There  is  no  interest  in  achieving  the  possible,  but  it  is  exceedingly interesting to perform the impossible.”
“Astound me!”
(Serge Diaghilev, 1909-1929; Director of The Ballets Russes)

Project Overview

 She’s Not There: new illusions meet ol’ time real time opened the CILECT (Centre International de Liaison des Ecoles de Cinéma et de Télévision) congress, Queensland Conservatorium Theatre, Brisbane, on the morning of November 20, 2016 (see Figure 1). Conceived and directed by the authors, Louise Harvey and Peter Moyes, and produced by Peter, the project used motion capture of a vocal performance by Christine Johnston for the 3D animation of our fictional cabaret singer, “Ms. Burly Chassis.” Louise modelled Burly and was assisted in the animation by Griffith Film School graduate Jessica O’Neill. Retaining the vocal track laid down by Christine, composer Cameron Patrick arranged the music for The Zombie’s 1965 hit She’s Not There with reference to a first pass of animation to assist with synchronisation and the creation of musical highlights. Re-presenting this animation on a screen aloft an empty theatrical stage, a virtual camera operator traverses this space, his movements in replication on the screen above, tracked by the motion capture cameras surrounding ‘the volume’ of the stage. The live orchestra accompaniment further embeds the sense of immediacy created by the virtual camera in its affectation of a ‘live broadcast’. Real confronts virtual towards the end of the song as Christine enters the physical stage to take up the refrain (and the accolades) as Burly’s attempts to hit a high note descend into uncontrollable coughing.

moyes-and-harvey-figure-1

Figure 1: Queensland Conservatorium Theatre interior

Magic and Mayhem

The She’s Not There project explores the nexus between music, performance and animation that has motivated animation practice since its inception as silent imagery accompanied by live music performance in theatre halls over a century ago. Coming out of a vaudeville tradition of magic and mayhem, Winsor McCay’s seminal animation Gertie the Dinosaur of 1914 was first presented as a “two- hander” between a live performance of McCay and his animated creation in a call and response routine that transgressed the boundaries between stage and screen. Contemporary performers such as Miwa Matreyek and Kathy Rose continue to explore this dynamic between real time (dance) performance and projected animation. Both practitioners incorporate animation, video and design into synchronised choreographed presentations, their bodies as performer and subject. The recent pop cultural phenomenon, virtual performer Hatsune Miku, brings these considerations into the 3D realm as holographic presentations are synced to live music performance. Conceived by illustrator KEI as a mascot for Crypton Future Media, the project brings new iterations to the traditional relationship between subject and audience, the virtual idol evolving through crowd-sourced software, fans contributing both songs and choreography. So too, the virtual band Gorillaz, created by Damon Albarn and Jamie Hewlett, ventured into live performance in 2001, the live band playing behind a screen of projected animation.

The She’s Not There project brings greater novelty and complexity to these relationships between past (pre-recorded) and present (live performance) by incorporating a virtual camera at the interface between the real and the digital, effectively re-capturing pre-rendered footage in an illusion of live broadcast. A live orchestral accompaniment further exacerbates this relationship between pre-rendered and real time, as does a mock exchange towards the end of the song, when real time performer Christine Johnston, responsible for vocal track and motion-captured performance, takes over from the fictional 3D character Ms Burly Chassis, and effectively steals the show.

The Virtual Camera

moyes-and-harvey-figure-2

Figure 2: Virtual camera and (one of the) motion capture camera (top right)

A virtual camera is comprised of a monitor with reflective markers on a shoulder-mounted frame. The markers provide positional data via motion capture sensors surrounding a ‘volume.’ A virtual environment with CGI characters is mapped onto this real-space volume. The tracked markers on the virtual camera allow the camera’s movements to be re-targeted to a virtual CGI camera within the digital environment. The monitor provides real-time feedback of the framing to the camera operator, thereby providing a window into the CGI world. This allows the space to be framed in much the same way as a live-action performance would be.

Virtual cameras are primarily used as a pre-visualisation tool in animation and film pre-production to establish framing and camera movements to be used during the production of VFX and animated shots and related live action plates. Operators are able to improvise within the CGI environment, following pre- rendered digital characters around the scene, re-framing as necessary, all the while generating the animation keys directly onto the CGI camera. This enables the successful creation of ‘hand held’ shots and of improvised framings that are motivated by the movements of the character in the same way that they would be when filming live action. For the She’s Not There project, these processes literally take centre-stage, delivering an entirely new visual and aural experience for a live audience. The concept of using virtual cameras to frame and ‘film’ a pre-rendered animated performance for real time presentation alongside a live orchestra is unique.

moyes-and-harvey-figure-3

Figure 3: Virtual camera operator and corresponding screen image for She’s Not There

Referencing motion capture sensors on a physical theatrical set, the virtual camera re-captures/re-presents a pre-recorded motion-captured 3D animated performance with the mobility and immediacy of live action reportage. The accompaniment of live music supports the illusion of real time performance. The conceit of training the virtual camera on the empty set further mystifies and astounds in homage to the magic of early animation acts, and in acknowledgement of the technical trickery that continues to drive audience engagement with the medium. An appearance of the actor responsible for the motion-captured footage closes the performance resolving past and present, real and digital. The ways in which tensions are played out between old and new, between pre-recorded/rendered and real-time performance, between absences and presences are at the heart of this exploratory project. See Figure 4 for a diagram of the project’s performance space.

moyes-and-harvey-figure-4

 

 

 

 

 

 

 

 

 

 

Figure 4: The She’s Not There performance space

Live Music and Animation

Music and animation have enjoyed a close and curious relationship over the last hundred years. With both disciplines grounded in a foundation of mathematics and an intense appreciation of movement(s) in time, the demands and possibilities animation has made of, and for, the workings of music have produced an arguably more intense and rewarding relationship with the sonic arts than has its live action cousin. In the early days of silent cinema, as shorts before the live action feature, a live piano or organ would denote the action and dramatic arc – speed up for the chase, slow down for love and loss– and to a certain extent authenticate the illusions on screen by providing a second channel of auditory information.

moyes-and-harvey-figure-5


Figure 5: Conductor Cameron Patrick and The Queensland Conservatorium Ensemble Orchestra

She’s Not There exploits these early relationships between music and animation. In particular we are interested in the tensions between real time performance and animated illusion articulated by live orchestral accompaniment of theatrical cinema. How might this tension be increased, teased out, exacerbated with the conceits afforded by motion capture technology? How might the suspension of disbelief that allows for a self-conscious enjoyment and complicity on the part of the audience be pushed that little bit further, yet still short of incredulity and our losing audience attention?

Old and New

Since its inception, animation has traded on its novelty value, its sleight of hand. As Donald Crafton (2013, 1982) and Norman Klein (1993) have argued, there is a performance value to the celluloid illusions. Vaudeville lightning sketch artists used animation to speed up what heretofore had relied on optical illusions and slower reveals. Winsor McCay, as performer, in real time, slipped behind the screen, his effigy to re-appear aloft in the dinosaur’s jaws. Crafton (1979, 1982) discusses how the presence of the hand of the artist that articulated this performance aspect retreated behind cartoon characters as the medium evolved. Klein (1993) examines the reflexive aspects of cartoon animation, notably the convention of the chase, which drew attention to the mechanics of animation while, at the same time, to its artistry and contagious mayhem, maintaining the magic of its illusion. In “An Aesthetic of Astonishment: Early Film and the (In)Credulous Spectator” (1989), Tom Gunning counters the myth of a naïve and credulous reception for the Lumiere’s L’Arrivée d’un Train en Gare de La Ciotat (1895) with assertions of an audience was comprised of “sophisticated urban pleasure seekers,” and well-versed in the machinations of “a pleasurable vacillation between belief and doubt” (p. 37). This curious tension between what audiences know of the mechanics of animation and how the consecutive images still manage to insist on their veracity – of particular interest given the high levels of animation literacy of contemporary audiences – enlivens our project. To present animation in a forum that might draw on the same novelty values that excited audiences of the first moving images, to stage contemporary illusions in an old time communal setting, to challenge audiences of a collective cinema literacy of 120 years to reach that ‘wow’ factor, is what motivates The She’s Not There project.

A history of animation has witnessed, and indeed been driven by, technical development towards greater cartoon seamless illusion. The currency of this development is animation’s ability to conjure and astound. Technology pushes further out, audiences are amazed, and novelty thus becomes the new normal. In “The Blow Book, Performance Magic, and Early Animation: Mediating the Living Dead” (2011), Colin Williamson draws parallels between technological advances in cinema and “the evocations of magic.” Williamson argues that contemporary discussion on cinema’s capacity for digital simulation could benefit from a “rigorous analysis of magic’s rich and longstanding relationship with animation” (p. 113). The She’s Not There project mines this area twixt digital simulations and the conjurer’s glove – the twilight of not-yet-familiar technology and its novel illusions. The virtual camera trains spectres on an empty physical set, apparently shooting nothing, and yet the digital space displayed on screen is enlivened by just those same moves, as a cabaret character struts her stuff in time with live music. The fictional story world grounded in a timeline of live music harkens back to the days of silent cinema – digital conceits are made real in performed space. The theatrical denouement of voice talent drifting onto the set to complete what the digital cipher has failed so miserably at, equates to the magician’s final bow, gloves off and nothing left up her sleeve (see Figures 6 and 7 for digital and real world performers). We’ve come a long way since McCay’s interaction with a drawn dinosaur, and yet the basic hook remains the same: novelty, sleight of hand and an audience eager to be delighted.

moyes-and-harvey-figure-6


Figure 6: 3D character “Ms. Burly Chassis”

moyes-and-harvey-figure-7

Figure 7: Christine Johnston in performance

Pre-recorded/Rendered and Real Time

One of many paradoxes inherent in animation is the exuberance, the apparent spontaneity and immediacy of the animated product in comparison to the painfully slow, overthought and overwrought means of its production; the illusion of life occurs only in the playback. Alexander Sesonske’s observation of the ephemeral nature of cartoons, that “neither these lively creatures nor their actions ever existed until they were projected on screen. Their projected world exists only now, at the moment of projection” (qtd. in Cavell 1979, pp. 167, 168) is at odds with the pre-determined, repetitive and rationalised approach inherent to cartoon production. In aiming to invigorate the process of animation, various practitioners have sought to incorporate a sense of liveliness and improvisation into the production process. Norman McLaren and others, for example, employed under-the-camera techniques which are open to chance effects and happy accidents via scratches and paint; Oskar Fischinger’s wax-slicing contraption generated sequences of images beyond his control, and puppet animator Barry Purves, singing the praises in 1996 of a stop-motion technique that “by its very process, has a slight unpredictability and spontaneous feel to it […] Wherever possible, I strive to make each take the equivalent of a live performance, and hope that some of that energy and spontaneity comes across” (n.p.)

For a medium capable of projecting such intense vitality, it’s a rather cruel irony that animation cannot offer the levels of real time performance and genuine improvisation as can its close cousin in abstraction: live music. The challenge of hitting the right notes, in time, pitch perfect; the adrenalin, the in-the-moment flow, the joy of getting it right (and the tragedy of messing it up), have, in the main, been lost on animation.

“Liveness” in the cartoon for Crafton is “proximal,” or close enough (2013, p. 52). Cartoon characters and their actions are informed by, and resonate with, the performances and creativity of animators, script-writers and the various others that contribute to the animation production process. In utilizing motion-capture data (see Figure 8), She’s Not There evidences the immediacy of a real-time performance. The (albeit) uncanny lifelikeness of animation produced via the motion capture process has been the subject of much debate in recent times. Further, in re-capturing the animated performance with a virtual camera, the apparent spontaneity of a ‘live’ broadcast is generated, as camera operator negotiates the stage in front of the audience, these manoeuvres faithfully executed in the virtual space on screen. The accompaniment of live music –with conductor providing the vital link between orchestral performance and pre- recorded animation via an audio ‘click track’ – supports the illusion of real time, lending the animated performance a sought after unpredictability and edge.

moyes-and-harvey-figure-8

 

 

 

 

 

 

 

 

 

 

Figure 8: Motion capture for The She’s Not There project

Discussing the period of silent cinema before mechanically synchronized sound and image, Gillian B. Anderson (1987) speaks to the impact live music had on the moving imagery:

At their best they [live musicians] introduced an element of suspense and of unpredictability into the musical accompaniment for films. Filmgoers could wait with bated breath to see what a virtuoso improviser would do next, or they could wait to see if the orchestra made the next cue on time. Anyone who has had the thrill of listening to two-and-one-half hours of live orchestral accompaniment to a classic silent film knows how much more exciting such a presentation can be than a totally mechanical presentation. The virtuosity of exact synchronization is made palpable. The non-stop music emphasizes and reinforces the filmic rhythms, the shot changes, and changes of scene. The volume, intensity, and direction of the sound are more affecting than that heard in most theaters. (p. 295)

moyes-and-harvey-figure-9

Figure 9: Conductor arranger Cameron Patrick

The tension between artifice and illusion is exacerbated by these elements of the real-time authenticating the pre-recorded. In explaining an audience’s suspension of disbelief in the “Tooniverse,” Crafton calls up Feyersinger’s concept of “metalepsis;” or, the “combining of two worlds that are perceived as mutually exclusive (yet are) connected at the same time” (2013, p. 50). Crafton continues,

We live in these cartoons as we do in other films without reflecting on the absence of blood actors or the stacks of drawing that made the toons, precisely because the films are constructed to showcase the irony between the corporeal and the intangible in the Tooniverse. (2013, p. 52)

It is the aim of the She’s Not There project to make this paradox apparent, to tease out Gunning’s “trompe l’oeil play of give-and-take” and the “desire to test the limits of an intellectual disavowal – I know, but yet I see” (1989, p. 37). The aforementioned interchanges between corporeal performer and her digital doppelganger – as Christine bumps out a performer past per prime and clearly struggling with the material – reference vaudeville animation acts in their transgressing the boundary between real and cinematic, conflating the distinctions between performance now and performance past, all the while maintaining a nod and a wink to an audience complicit in the illusion.

Absences and presences: She’s Not There

For our animated performance and orchestral score, The Zombies 1965 song She’s Not There was chosen as an apt track to articulate our conceit, to cue the audience into the sleight of hand.

Cholodenko and others have written comprehensively on the implications of Animation as death, and begetting, and on its fundamental ephemeral nature. Animation by its very nature “makes every encounter in cinema […] an encounter with death” (Cholodenko 2007, p. 18). Cholodenko explains that ‘the attraction (of) film (including therefore cinema) and all the more animation are of the order of the uncanny’ (p.17). He argues for the primacy of Animation in Gunning’s examination of the effects of his cinema of attractions. It is our “primitive animistic fears of the return of the dead” that account for ‘the simultaneous attraction and repulsion, fascination and dread – at seeing what was still “come to life” (Cholodenko 2007, p.17). So too, the uncanny effects of what once was, and no more, its traces teasing in their absence; both writers call up Maxim Gorky’s response to the 1896 screening of the Lumière brothers projections at the Nizhny-Novgorod fair in Russia and the first significant review of cinema: “It is not life but its shadow, it is not motion but its soundless spectre” (Cholodenko 2007, p. 16; Gunning 1989, p. 37).

Performance-past is embedded in The She’s Not There project in a twofold fashion: in the motion-captured traces of the vocal performance of Christine Johnston, and in the recapturing in real-time of that performance retargeted to our digital character via the virtual camera. The exchange between real and virtual performers mentioned above, the articulation of live music accompaniment – both raise the stakes at this interface between past and now, as they flag the uncanny machinations of Cholodenko’s animatic: “the very logics, processes, performance and performativity of animation … the very ‘essence’ of animation” (2007, p. 15).

To return to the topic of magic, as well as to connect it with the uncanniness at the heart of Cholodenko’s animatic, resonance can be found between The She’s Not There project and the subject of Williamson’s inquiry “The Blow Book, Performance Magic, and Early Animation: Mediating the Living Dead” (2011). Williamson describes the apparatus’s operation:

A transformational conjuring book, apparatus, or prop that is traditionally constructed like a flip-book […] a magician would manipulate the open edge of the book by flipping a series of concealed tabs each of which harbors its own unique and repeated set of images. What the spectator perceives […] is the mysterious transformation of a blank book into an illustrated text. (p. 112)

Just as pre-cinema audiences were confounded by the transformation of the Blow Book’s empty pages into its colourful array, so too audiences for The She’s Not There project alternate their attentions between an empty stage and its generation of live imagery evident on the screen above via the manipulations of the virtual camera – both connect with a tradition of magic performance which “has its own history of mediating […] the boundaries between life and death, the animate and the inanimate” (Williamson 2011, p. 112). The significance and impression of each illusion relies on a spectator “forced to oscillate precariously between the rationality and the apparent impossibility of the event” (p. 118); between a there and not-there. Williamson cites the trick films of Blackton, Méliès, and Pathé as performing a “materialization” of “the basic properties and possibilities of the cinema (and the experience thereof) conceived as an animistic or animating medium” (pp. 123, 124). The She’s Not There project draws on this same spirit that enlivened the performance of early animation as magic trick, that quality of ‘coming to life’, from nothing: something.

Coda

As virtual camera operator paces an empty stage, those movements only made sense of in virtual space, as the past is re-animated by just those moves in real- time, as the real and the virtual is presented as both connected and as disjunctive, notions of there and yet not there, of past and present, are played out for an audience knowingly and happily deceived. She’s Not There played to an audience of over 100 film school educators that made up the CILECT congress in Brisbane, Australia, on the morning of November 20, 2016. Some were perplexed, others indeed astounded, all were thoroughly engaged in, and entertained by, a performance of animation that might be seen to connect with the spirit of experiment and playfulness attending the first animation presentations – “a singular event […] an implausible trapeze act” (Klein 1993, p. 22).

moyes-and-harvey-figure-10

 

 

 

 

 

 

 

 

 

 

Figure 10: Christine Johnston closes the show

Dr Peter Moyes is Director of the Animation Program, Griffith Film School. He specializes in Animation and Film history and contextual studies, having taught at Griffith University for almost twenty years. Peter’s Doctorate of Visual Art in Animation addresses interactive picture books and pedagogy. Current research interests include digital media for ethics education in schools, and live music animation relations. Peter was Director of the Brisbane International Animation Festival from 1996 to 2000; his animated film Sunday has been included in major retrospectives and has won a number of awards including The Yoram Gross Animation Award at the 40th Sydney Film Festival.

Dr Louise Harvey is a 3D artist and filmmaker who has been combining her interests in animation production, research and teaching since 2001. Her 2007 doctoral thesis – an examination of 3D animation production techniques and principles – formed part of a major ARC-funded study on the topic. Her primary field of research is focused on the development of efficient animation production workflows, addressing the ongoing challenge of how to produce quality animation on time and on budget. Outcomes from Louise’s research have been articulated via conference presentations and papers, numerous digital art works and animated films.

 

References

Anderson, G. B. (1987) “The Presentation of Silent Films, or, Music as Anaesthesia.” The Journal of Musicology 5 (2). pp. 257 -295.

Cavell, S. (1979) The World Viewed: Reflections on the Ontology of Film. Cambridge, MA: Harvard University Press.

Cholodenko, A. (2007) “(The) Death (of) the Animator, or: the Felicity of Felix Part I: The Kingdom of Shadows.” Animation Studies 2.

Crafton, D. (1979) “Animation Iconography: The ‘Hand of the Artist.’” Quarterly Review of Film Studies 4 (4). pp. 409-428.

—. (1982 ) Before Mickey: The Animated Film 1898-1928. Cambridge, MA: MIT Press.

—. (2013) Shadow of a Mouse: Performance, Belief, and World-Making in Animation. Berkeley: University of California Press.

Gunning, T. (1989) “An Aesthetic of Astonishment: Early Film and the (In)credulous Spectator.” Art and Text 34 (Spring). pp. 34-53.

Klein, N. (1993) Seven Minutes: The Life and Death of the American Animated Cartoon. London: Verso.

Purves, B. (1996) “The Emperor’s New Clothes.” Animation World Magazine. Available from http://www.awn.com/mag/issue1.1/articles/purves.html. Accessed 17 July 2015.

Williamson, C. (2011) “The Blow Book, Performance Magic, and Early Animation: Mediating the Living Dead.” Animation: An Interdisciplinary Journal 6 (2). pp. 111–126.

© Peter Moyes and Louise Harvey

Edited by Amy Ratelle