Current Trends of VFX
Week 1: Introduction to the module
During lecture 1, we got introduced to the module and tried to think about the current VFX trends. We discussed the current trends in breakouts and tried to find some examples. For instance, we came up with some trends such as photorealism, motion capture, deep-fakes, green screening, virtual filmmaking (Unreal Engine), real-time rendering, and so on. Also, we looked at the topics of "the age of the image" and the "trend of the lens" by watching a few first minutes of James Fox's documentary "The Age of the image: EPISODE 1". He talked about the idea of "the age of image" and how photograph/image is the main part of our era. However, we were encouraged to watch the whole series in our own time. We then started thinking about the relationship between VFX, photography and lens.
Volumetric video
Bullet time
Bullet Time is a visual effect that is used to slow down time and pan around a subject during an action scene. The effect allows the audience to view high-speed motions that would otherwise be impossible to see, such as flying bullets.
Hundreds of still cameras were used by the Wachowskis at their studio when they shot the Bullet Time scene for The Matrix in 1998. Every still image was meticulously interpolated to produce a flowing effect when the cameras shot in sequence, circling the characters. After the Matrix films popularised the word, it has become a widely used expression in popular culture.
Check out the video below:
We also looked at the work of Harold Edgerton. Harold Edgerton was an MIT professor and artist who invented the strobe flash, stop-action photography and a way of taking super-fast photographs. . Click here to see some of his works. The following is one of his most famous works, a bullet through an apple.
Task
Create a gallery of images of Edgerton, and try to pair up a few of his photographs, each with a VFX shot:
The following examples are from Dredd (2012), a science fiction action film directed by Pete Travis. I matched each of them with Edgerton's photographs.
Dredd
.png)
Edgerton
Dredd
.png)
Edgerton
Dredd
.png)
Edgerton
Dredd
.png)
Edgerton
Week2: The Photographic Truth Claim- Can we believe what we see?
This week, we learned about the photographic idea of 'Indexicality,' also known as the 'Photographic Truth Claim.' First, we tried to get a sense of the truth claim by looking at some important philosophical concepts.
The allegory of the cave
One of the most well-known philosophical concepts in history is Plato's "Allegory of the Cave." Plato made this allegory to reflect on the nature of belief vs knowledge. According to the allegory, there are inmates chained together in a cave for all their life, with no natural light in the cave. Instead, there are shadows by the light of the fire on the cave wall. The inmates keep an eye on the shadows, thinking they are real. One prisoner, according to Plato, could be set free. Finally, he notices the fire and understands that the shadows are not real (phantoms). This prisoner is able to escape from the cave and find a whole new world of which they were previously ignorant. The freed prisoner wants to return to the cave to help other captives and set them free. When the returning prisoner re-enters the cave, his eyes, which have been accustomed to sunlight, get blind. According to Plato, the inmates would deduce from the returning man's blindness that the journey out of the cave had damaged him and that they should avoid making a similar journey. Plato believes that the chained prisoners would murder anyone attempting to taking them out of the cave as they think they will be harmed if they leave the cave.
Photographs and Reality
Let's ask ourselves what it is about photographs that we find so powerful and seductive? To think more deeply about it, we need to refer to semiotic studies and their central concepts.
According to Charles Sanders Peirce, a sign can be categorised as one of these three types: icon, index, or symbol.
-
icon: signifier has a physical resemblance to the signified
-
index: signifier shows evidence of what's being signified - like smoke
-
symbol: no resemblance between the signifier and the signified - like stop road sign
The photograph is both iconic and indexical - the photograph looks like something and shows evidence of something both.
Index examples (evidence that something was there ): shadows, bullet holes, coffee stains, tyre marks
Fake Views
Fake Views is the fourth episode of the BBC's Age of the Image series, a part of a series in which art historian James Fox investigates how images have shaped the contemporary world.
Blog post: the Photographic Truth-Claim
Are photographs (necessarily) reliable, accurate and truthful? To answer that, we must deal with the Photographic Truth Claim. In the process of forming a photograph, light is emitted from the subject and passed via a camera lens to generate the picture by either the transformation of light-sensitive surface (traditional photograph) or by storing numeral data about light (digital photograph). The Truth Claim refers to a reputed belief that considers traditional chemical photography as a direct imprint of reality. Tom Gunning (2004) discusses the Truth Claim, maintaining that the claim relies on Charles Pierce’s term indexicality and visual accuracy both. He elaborates on how visual accuracy should be seen as a combination of indexicality and iconicity, and one must not muddle them up. An index does not need a resemblance to the subject it points to, but it is influenced and determined by the subjects. In the same way, the digital photograph can be as indexical as the traditional photograph since both show evidence of the things in front of the camera. As Gunning suggests, although the digital manipulation of the photograph can weaken the indexicality based on the truth claim, it offers plenty of techniques and processes to explore photography as an art mode. “Pictures generally are more than signs” (Gunning, 2004, p46), and the task of photography is not limited to making truth claims. I believe that is a part of the reason we have too much fascination with the photograph.
Reference
Gunning, T. (2004) ‘What’s the Point of an Index? or, Faking Photographs’, Plenary Session II, Digital Aesthetics, 1(25), pp. 39-48
Week 3: Faking Photographs- Image manipulation and computer collage
This week, we explored image manipulation history and got a feel for how photographs have always been manipulated from the dawn of photography. We discussed how photographic fakery has been going on since the early days of photography and cinema - but most think digital manipulation is the culprit of breaking the connection between photography and reality.
Cottingley Fairies (1917), a series of five photographs taken by Elsie Wright and Frances Griffiths, two young cousins lived in Cottingley. The fairies were paper cut-outs, but the photographs were widely thought to be authentic and robust evidence of fairies and spiritual worlds.
Frank Hurley's iconic photograph, "Battle of Zonnebeke (1918)", is a composite of various photographs from the Battle of Zonnebeke in Belgium during the First World War.
An animation of the creation of "Battle of Zonnebeke (1918)"
A few famous faked analogue photographs - before Photoshop
.jpg)
The Falling Soldier (1936) By Robert Capa
Josef Stalin removed a commissar from the photo in approximately 1930
Hitler Removed Joseph Goebbels, 1937
Mao Tse-tung Removed Po Ku, 1936
A few famous digitally faked photographs - after Photoshop
Kim Jong Un Fake Korean Missile launch, 2015
Fake Asian Tsunami in 2004
The dead body of Osama Bin Laden
Faked shots from visual effects
Terminator: Dark Fate (2019)
Inception (2010)
Jurassic Park (1993)
Blog post: Definition of VFX compositing- what is it, and how does it work?
Compositing is the art of integrating various visual components from several sources into a single image/scene, usually to give the impression that they were all part of the same image/scene. Compositing in cinema may be traced back to the works of Georges Méliès, one of the most significant earlier filmmakers, around the beginning of the twentieth century. Méliès’ Double Exposure techniques led to the development of compositing in film. Then, in the middle of the century, Background Projection (also known as Rear Projection) and Matte Painting (matting) became other popular compositing methods and techniques used by pioneers in cinema. Today, greenscreen is prevalent tool filmmakers employ to add visual effects in the compositing stage. One familiar example of compositing is placing a weather forecaster in front of a green screen with weather details behind them. However, in VFX, it is widely believed that the final stage of the pipeline is compositing. VFX compositing seamlessly blends digital assets with live-action footage to put together the final version of a scene. Compositing is frequently used to construct sets, locations and scenes that would be hard (and sometimes impossible) to produce wholly in front of the camera otherwise. Along with it, another task of compositors is to improve VFX shots to represent the director’s vision better and address any issues that arose during the production, like removing/changing specific elements from a scene (Rotoscoping). Finally, compositors put together all the pieces, transforming a shot into something spectacular previously unthinkable.
Week 4: Defining Trends of Photorealism
We discussed Photorealism, its definition, and trends this week. Thinking of Photorealism in VFX, we tried to find a few examples from films and games. Before that, we were introduced to a brief history of Photorealism and a movement of American painters in the late 1960s and through into the 1970s (e.g. Chuck Close, Richard Estes) who were replicating the clarity and accuracy of photographs in painting. Finally, we categorised the examples as two main trends: Completely constructed by using CGI, and the composite.
by Chuck Close
by Richard Estes
Fully CGI photorealism examples
The composite examples
Blog post: Photorealism
Photorealism in VFX
In art, Photorealism is the quality of representing characters, objects, environment, and things as if they were all captured by a camera, making viewers assume that the artist used a camera to produce the image/scene. That is, a photorealistic artwork portrays the photographic reality, not absolute reality. In photography, the reality is captured by a camera lens, so the image is affected by light, depth of field, shutter speed, point of view and perspective, framing, colours, etc. However, in the context of VFX, Photorealism is the art of making a synthetically created image appear like a photograph of something in the real world. However, VFX is not limited to the real world, and verisimilitude here can be interpreted as the appearance of truth or reality within the context of the film, not necessarily reality in our world. For instance, when designing a scene, a VFX artist may create a CG alien aircraft or a fantastical creature; those images still can be photorealistic or even realistic but in terms of indexical and perceptual factors. To achieve Photorealism in the computer-generated trend, artists utilise 3D software to mimic the function and effect of lights and cameras, thereby producing a highly life-like, natural, and detailed representation of reality. Therefore, the quality and techniques of rendering are critical to Photorealism in CGI, as the rendering process compiles all digital materials into the final scene. Along with that, as Barbara Flueckiger (2015, pp. 78-98) suggests, to accomplish Photorealism, CGI needs to simulate some digital artefacts of analogue film artificially. Grain and dust, depth of field, lens distortion and chromatic aberration, lens flare, vignetting, bokeh and motion blur are some of the effects VFX artists can make use of to emulate a film look and enhance the reality effect.
Reference
Flueckiger, B. (2015) Special effects: new histories/theories/contexts. Edited by D. North et al. London: Bloomsbury, pp. 78-98.
Week 5: Digital Index: Bringing truth into VFX via the Capture of Movement
This week, we discussed Motion Capture (Mocap) as a widely used trend of capture in VFX and film. In the few next weeks, we will be exploring other types of capture, such as Scanning – Laser scanning, Lidar scanning, etc. Categorising types of capture, we consider the trends as Motion-Capture, which gives us movement data and Reality Capture (3D scanning), which provides data about the object characteristics.
Motion capture in cinema (a short Youtube video)
Blog post: Compare Motion Capture vs Key Frame Animation
Motion Capture (abbreviated as ‘mocap’) is a technique by which points or markers on an actor’s body are recorded using cameras and a specialised outfit. Data is captured from all or part of the actor’s performance and then sent to a computer, where it can be mapped onto a CG character or asset. In other words, Motion Capture is the process in which filmmakers and artists use live-action as a reference to create more natural and realistic movements and performances for CG characters. Motion capture has several advantages compared to keyframe animation and other 3D animation approaches. In comparison with the keyframing method, Mocap can be more cost-efficient overall, reducing the time needed for the keyframing process. Mocap converts real-world movement into data that can be applied to a CG character without going through the process of manually animating every move, facial expression, etc. As a result, motion capture can produce intricate, lifelike, and physically accurate motion that is difficult to recreate otherwise. Nonetheless, there are a few challenges when using Mocap. In some cases, Motion Capture is simply impossible to achieve. The movement of anything that does not follow the laws of physics cannot be captured. Also, as movement is captured in our real world, the place where Mocap is going to be done must have certain conditions. For instance, we cannot make a long run animation in a small studio. All in all, both techniques, Mocap and traditional keyframing, have their advantages and disadvantages, and whether to use them depends on the project’s needs and requirements.
Week 6: Reality Capture (3D Scanning) and VFX
This week, we explored another trend of capture in VFX, which is Laser Scanning. As discussed last week, Laser Scanning is a type of Reality Capture, which provides us with data about the objects. In addition to Laser Scanning, Reality Capture (3D Scanning) also includes Depth-based scanning and Photogrammetry. LIDAR, which stands for Light Detection and Ranging, involves the utilisation of laser beams that reflect off a surface or environment to create a high-resolution point cloud of data. When digital models are unavailable, exceedingly complicated, or very expensive to create, LiDAR is perfect for digital replication of large-scale objects and settings.
The state of LiDAR in VFX
Florance in 360
Blog post: a Case Study post on Reality Capture (LiDAR)
LiDAR, or Light Detection and Ranging, is a technology that employs laser beams to gather information about objects, surfaces, surroundings, etc. LiDAR sends out quick laser pulses, up to 150,000 per second, and measures how long it takes for each reflected pulse to return to the receiver. The information captured, which is a LiDAR-generated point cloud, can then be utilised to produce high-resolution and highly accurate 3D representations of objects, buildings, and environments. LiDAR has a wide range of applications, from archaeology, geology and robotics to game design and visual effects. For example, in VFX and films, LiDAR provides artists and filmmakers with on-set data acquisition used in various ways throughout film production, particularly for making CGI and 3D effects. One example was Joker, where LiDAR enabled the VFX crew to enhance productivity and generate more realistic visual effects. As John Ashby, a Lidar supervisor (from Aura FX) who worked on Joker, asserts, LiDAR helps to capture “the most data you can get in the fastest amount of time without holding up the production” (Failes, 2020). This is particularly important to improve realism in VFX, as LiDAR provides artists with detailed measurements by which “match-moving” can get easier and smoother. “What Lidar really helps you do is, when you are working on multiple shots from different camera angles, you can track everything to the Lidar data, and it snaps all the cameras into the same world space” Ashby explained (Failes, 2020). He scanned many sets for Joker, including Arthur’s flat and the Murray Franklin TV show set. Ultimately, VFX artists working for Joker used the captured point cloud data in order to add CG blood, get a flawless camera track of the sets and produce more accurate and realistic results.
References
Dollard, C. (2020) The BLK360 Goes Hollywood: VFX Workflows with Allan McKay. Available at: https://shop.leica-geosystems.com/ca/blog/blk360-goes-hollywood-vfx-workflows-allan-mckay (Accessed: 10/11/2021).
Failes, I. (2020) Tales from on set: Lidar scanning for ‘Joker’ and ‘John Wick 3’. Available at: https://beforesandafters.com/2020/07/06/tales-from-on-set-lidar-scanning-for-joker-and-john-wick-3 (Accessed: 10/11/2021).
Seymour, M. (2012) Pointcloud9 – a LIDAR case study. Available at: https://www.fxguide.com/fxfeatured/pointcloud9-a-lidar-case-study (Accessed: 10/11/2021).
Blog Post 6 for Assignment 1
Blog Post 6 was for me to choose and write about a trend in Visual Effects. I decided to write on the subject of deepfakes.
Deepfake
Deepfake is a technology that uses machine learning techniques and algorithms, as well as artificial intelligence (aka AI), to produce synthetic media (mainly video and image). Deepfake substitutes one person’s resemblance with a different person’s face and match up face and body to create an illusion of reality. For a long time, however, VFX artists have been putting actors’ faces on stunt doubles’ bodies, but deepfake, on the other hand, is relatively new. In deepfake, faces are replaced more easily by AI-powered software instead of meticulously by VFX artists. Deepfake has a wide range of applications among Internet users – mainly for entertainment and fun, but sometimes for blackmailing and pornography uses. However, in VFX and cinema, various quality difficulties and shot-specific needs have proven deep fakes unsuitable to be used for final scenes. Nonetheless, deepfake technology is advancing enormously as machine learning algorithms are evolving. Using deep fakes rather than fully CG actors allows filmmakers to use AI-based face replacement to transform their actors into any character they want more easily and quickly (and, of course, inexpensively). As a result, filmmakers can utilise deepfake as “digital makeup” to age or de-age actors, for instance. Moreover, AI-generated synthetic media in VFX may be able to bridge the uncanny valley effect compared to computer-generated footage, as the technology advances. VFX companies and artists have not yet fully embraced Deepfakes, but they may adopt similar AI technologies to create a high level of realism as the technology behind deepfakes improves.
References
Aldredge, J. (2020) Is Deepfake Technology the Future of the Film Industry? Available at: https://www.premiumbeat.com/blog/deepfake-technology-future-of-film-industry (Accessed: 19/11/2021).
Bridging the uncanny valley: what it really takes to make a deepfake (2019) Available at: https://www.foundry.com/insights/film-tv/digital-humans (Accessed: 19/11/2021).
Failes, I. (2020) Deep Fakes: Part 1 – A Creative Perspective. Available at: https://www.vfxvoice.com/deep-fakes-part-1-a-creative-perspective (Accessed: 19/11/2021).
Jaiman, A. (2020) Positive Use Cases of Synthetic Media (aka Deepfakes) Available at: https://towardsdatascience.com/positive-use-cases-of-deepfakes-49f510056387 (Accessed: 19/11/2021).
Week 9: Reality Capture (Photogrammetry) and VFX
We explored the first two types of Reality Capture (LiDAR and Depth Scanning) during the previous sessions. This week, we got a grasp of the third type of Reality Capture, Photogrammetry. Photogrammetry is a 3D scanning technology and technique that uses images (photographs) to produce 3D models. This process can be an essential tool and component of the 3D modelling workflow. The captured data can create digital assets and characters (copies or facsimiles) that are remarkably lifelike. That is to say, Photogrammetry produces deep realism. Furthermore, it is the most accessible kind of 3D scanning, as it can be done using phone cameras, apps, as well as desktop software like Metashape or Capturing Reality.
Experiencing the Veronica Scanner at the Royal Academy of Arts
Photogrammetry examples (from Sketchfab)
Week 10: Simulacra, Simulation and the Hyperreal
This week, we looked at some ideas and theories introduced and developed by French sociologist/philosopher Jean Baudrillard to help us grasp the concepts of Simulacra and Simulation. According to his theories, Simulacra are images with no origin, whereas simulations attempt to emulate something in the real world. He refers to Simulacra as hyperreal, the generation of models without origin or reality. He firstly states in his renowned and fascinating piece of literature, Simulacra and Simulation, that our view of reality is heavily influenced by signs and symbols. "The simulacrum is never what hides the truth - it is the truth that hides the fact that there is none," he writes, arguing that the image now precedes (therefore does not depend on) the reality. "The simulacrum is true."
According to Baudrillard, simulation can be viewed as a four-step cycle from representation to simulation. In his book, Simulacra and Simulation, he explains the four successive phases of the image as bellow:
An example: Pumpkin Spice Lattes
Images travel through four stages, from reality to its absence—or from pumpkins to a flavouring that, despite its name, has nothing to do with pumpkins.
My choice of essay title
Week 11: Virtual Filmmaking
We learned about Virtual Filmmaking this week. Using CGI, game engines, and virtual reality technology, virtual filmmaking seamlessly blends physical and virtual images/shots in real-time. Studios can shoot on a stage while simultaneously viewing virtual images and shots. For example, filmmakers can easily change backgrounds to move between different locations. Therefore, with virtual production, VFX is no longer limited to post-production.
The Virtual Production of The Mandalorian (Youtube video)
The Transition to Real-Time Filmmaking - The Pulse by Unreal Engine
My Presentation (PowerPoint slides)
My Essay
The Rise of Motion Capture in Animation and VFX Industry
Introduction
Animation works based on an optical illusion of motion. We see a series of still images shown in a fast enough succession as if they are gradually moving and changing form and position. This effect is the main principle behind live-action films and motion picture technology. Therefore, the optical illusion of a persistent image, or as film theorists describe, the “persistence of vision”, lies behind animation and moving images overall. To make smooth motion and lifelike animations, animators have always been looking for innovative methods and technologies.
Figure 1: The illusion of motion created by the Persistence of Vision
As a unique type of expressive art, animation allows the artists to convey every narrative they want by manipulating the look and movement of characters, objects and environments. On the one hand, animators have a great deal of freedom as they have control over almost every aspect of their animation, which when employed effectively, their animations can have a significant impact on a wide range of audiences. On the other hand, although this freedom might be a huge benefit, it can equally become a challenge when animating intricate and realistic movements, for instance, which can be tricky and demanding.
What is Mocap?
The cutting-edge technology of Motion Capture has been used by animators, VFX artists and filmmakers to create natural and realistic movements and performances for CG characters and objects. Motion Capture, often known as Mocap, is the technique of recording the motions of actual actors or animals and mapping those movements onto CG characters so that the digital characters move and behave in the same way as the real actors do. A few technologies are there to accomplish Motion Capture, but a widely-used method is placing markers on performers and collecting data about their motion while they are making the intended actions. Needless to say, actors play a central role in the performance of virtual characters. Here Mocap shows its strength when producing intricate, lifelike, and physically accurate movements and performances that are difficult to recreate by keyframing otherwise. Hence, Mocap is sometimes referred to as Performance Capture, especially when capturing facial expressions or finger movements.
Figure 2: Actors wearing Mocap suits used in optical passive systems
In optical marker-based Mocap systems, cameras are there to track the spatial locations of markers (either LEDs in Active systems or reflective markers in Passive Infrared systems) and then send the captured data to relative software to track the markers as a result. In reflective optical Mocap systems, Infrared (IR) LEDs are placed around the camera lens, whereas approaches based on Pulsed-LEDs measure the Infrared light generated by the LEDs rather than light reflected off markers. There are a few more approaches to Motion Capture, and some can even make animation for a simplified model in real-time, providing the animators with an instantaneous result during the production stage. However, the collected data is still binary codes, which the software needs to decode and translate into digital model movements.
Motion Capture has come a long way since its early stages. It is now used in almost every film featuring CG components and nearly every big-budget video game. What used to be a costly and demanding process has now become more versatile and achievable and can be accomplished at a lower cost. Polar Express was one of the first feature films to excessively rely on Motion Capture to capture motions as well as facial expressions. In the case of Polar Express, although the narrative was based on a generally well-liked novel, the animation was vastly criticised. Several reviews criticised the characters’ animation and described it as lifeless and eerie. Although the motions appeared natural, the lack of exaggeration gives the impression that the movements and actions were fake or somehow bland and weird (Mou, 2018).
Rotoscope, Photostats and now Mocap
Since the advent of animation, animators have been studying the movement of human actors and animals in order to create realistic motion (Johnson and Thomas, 1981). The Rotoscope was the first method; it was a device that projects live-action film frames onto the animator’s workspace, providing a frame-by-frame guide for drawing.
Figure 3: Rotoscope machine designed by Max Fleischer, 1917
The Rotoscope machine, designed by Max Fleischer in 1917, could project images from a film onto an animation table. Then, animators could translate the motions and apply them to characters frame by frame. From below, a projector was adapted to focus one image at a time onto a square of clear glass placed on the drawing table. Finally, drawing paper was laid over the glass with pegs at the bottom of the glass to keep the paper in place, making tracing possible for each frame. The purpose of using this method was akin to Motion Capture technology; artists could generate detailed animations based on animal or human motions while simultaneously speeding the whole process up. Thus, Mocap and Rotoscope can be seen as two methods for creating movement. The most important thing that separates them is their technological capabilities.
The use of live-action film has long been part of the animation industry. Disney animators Ollie Johnston and Frank Thomas (1981) describe live-action as “an assist to animation, a partner to animation.” Animators used to draw from real actors and objects regularly in the early 1930s. However, when the need for more detailed movement and realistic motion in films arose, this form of drawing from observations ultimately became inadequate. To tackle this, Walt Disney commissioned a team to develop a process for reproducing each film frame onto photographic paper the same size as the drawing papers. These photostat sheets were trimmed to fit pegs on an animation workspace, allowing the animators to observe the actions by changing between frames (Johnson and Thomas, 1981).
Digital Indexicality
According to the Disney animators, every time animators stayed too close to the photostats or directly recreated human actions, the results became increasingly odd. Although the movements appeared natural, the character lost its lifelike aspect. “It was impossible to become emotionally involved with this eerie, shadowy creature who was never a real inhabitant of our fantasy world” (Johnson and Thomas, 1981). Then, their idea was to have the cartoon figures do the same actions as the live performers, with the same timing and staging, but because characters require different proportions, the figure and its live reference could not do everything exactly the same. As a result, animators came to the conclusion that the actors’ performance needed to be reinterpreted, not to be merely copied.
As a result, Disney used photostats to determine what should be in the scene and how. Looking at photostats, the animator picks just the actions relevant to the scene’s main point, then amplifies the actions and motions until they become dominant, with everything else weakened. “A work of art is never a copy; for it to have meaning to people of many generations and numerous cultures, it must be the personal statement of an artist” (Johnson and Thomas, 1981).
Similar to photostats or projected images from a rotoscope machine, the captured data in Motion Capture can be used as a reference. Tanine Allison (2011, p325-341) suggests that Mocap can be seen as an example of “digital indexicality.” Motion Capture, according to Allision, is a combination of CGI and data recorded from reality that blends the capabilities of CGI with traces of real-life occurrences. She goes on to say that Mocap shows the heterogeneity of digital visual culture, an amalgamation of older media traditions and computers’ greater automation and manipulation capabilities. As a result, she argues that indexicality still exists in the digital era, as we can see it in Motion Capture technology.
Mocap vs Keyframe Animation
Despite its incredible potential and strength, Mocap is not yet wholeheartedly welcomed, particularly within the animation community. It might be because Motion Capture does similar tasks as animators. This mindset has caused traditional keyframe animators tension. Many ask themselves whether Mocap will eventually supplant animators in the future? This seems a prevalent misunderstanding, but upon closer examination, it does not seem a probable future unless humans can push the boundaries of physics and do superhuman deeds. We all know it is not possible to capture the movement of anything that does not obey the rules of physics.
The schism between animators and Mocap technicians and enthusiasts in the animation field can be due to a few unrealistic expectations about what motion capture can do, such as the idea that Mocap will displace animators. Moreover, another source of this tension is that Motion Capture technological development has not yet developed technologies to use the captured data easily, leaving animators with difficult-to-manage information (Gleicher, 2000).
Figure 4: Ruffalo playing Hulk in The Avengers (2012)
The movements have to be pushed and manipulated to increase appeal even in live-action films. For example, in the film Hulk, the VFX crew used Mark Ruffalo’s mocap data to reproduce his characteristics in the Hulk’s facial expressions and movements. However, when the Hulk fights with aliens, it is evident that Mark Ruffalo would be unable to act it out. Moreover, even if he could, the motion capture data would lack the intensity and aggression necessary to convey the Hulk personality. At the example of Amazing Spider Man 2, CG components were used extensively, and Spider Man was created using traditional keyframe animation to reproduce the character’s famous superhuman motions.
Data Processing and Clean up
Despite its name, “recording the motion” is only one step in the Mocap process of creating animation from real-world captured data. Cleaning up and making adjustments are both necessary after recording the data. That is why current research and developments have aimed to automate the process and minimise human involvement (Asraf, Abdullasim and Romli, 2019).
Figure 5: King Kong (2005)
Mocap, therefore, should not be seen as a perfect replication or a direct reproduction of the actors’ performance and motion. In practice, Motion Capture is a process full of a massive amount of data, inconsistencies, and errors. Data processing, manipulating, gap-filling and smoothing the data are some typical necessary works after capturing the motion. For instance, in the King Kong film, the translation of human movement into the enormous ape movement proved problematic and challenging at times. Although we know that the fundamental joints of the body are the same from man to ape, the proportions vary. Therefore, capturing Andy Serkis performance as Kong, the Motion Capture data had to be tweaked to lessen the distance between Serkis’ hip and knee. Also, gorillas can move in ways that humans cannot, so the software must be tweaked to consider these physiological differences (Allison, 2011, p325-341).
Thus, we can conclude that Mocap could be highly beneficial if the usage of the data acquired, including the mapping and editing process, is thoroughly considered. Every animation has some unique requirements, and a capture technique can only be helpful as a tool for motion production if these requirements are taken into account.
The Future of Mocap
As the range of high-quality Motion Capture technologies available on the market grows and their cost and complexity declines, the Mocap industry becomes more accessible to those without enormous budgets or large technical teams. One of the most critical factors driving this growth for films and video games is the demand for high-quality realistic 3D animations. Mocap enables the development of more lifelike characters in a shorter amount of time. In the film industry, even a slight delay in production may be extremely costly. As a result, not only does Motion Capture help animators and filmmakers create appealing animated characters, but it also helps them keep production costs down and complete projects on schedule.
Cloud technology can be beneficial to facilitate the Mocap process. Uploading Mocap data straight to the cloud and process it without requiring sophisticated on-premises gear is a huge step forward for the motion capture industry. Traditionally, when we have more captured data from the Mocap, we need a more powerful computer to process the data. With the cloud, however, this is not the case anymore. The cloud would not only allow for faster processing time, but it would also allow for worldwide cooperation among users from different studios and even countries. For example, a studio in London can capture and store the motion data directly into the cloud, and then a different team based in Tokyo use the data to make the animation. This would significantly reduce the time it takes for teams to collaborate, giving instantaneous access to data for numerous users and increasing remote working productivity. However, the motion capture industry’s potential use of the cloud is just getting started, and we have to see what new solutions and workflows emerge in the coming years (Löring, 2021).
Another Motion Capture trend is the use of machine learning techniques and AI algorithms, allowing artists to employ Motion Capture in a faster and more efficient way than they could before. Machine learning is the method of teaching computers to make decisions using massive databases. Machine learning promises to make the Mocap process more efficient, as it currently contains various time-consuming and complex procedures (Foundry, 2020). As a result, many VFX companies are turning to it and investigating its possibilities, even though it is still a relatively new concept.
References:
• Allison, T. (2011) ‘More than a Man in a Monkey Suit: Andy Serkis, Motion Capture, and Digital Realism’, Quarterly Review of Film and Video, p325–341.
• Foundry (2020) Machine Learning: changing the game of Motion Capture. Available at: https://www.foundry.com/insights/machine-learning/motion-capture (Accessed: 27/12/2021).
• Future Learn (no date) Persistence of vision: how does animation work? Available at: https://www.futurelearn.com/info/courses/explore-animation/0/steps/12222 (Accessed: 26/12/2021).
• Gleicher, M. (2000) ‘Animation From Observation: Motion Capture and Motion Editing’, Computer Graphics, 33(4), p51-54.
• Hazry Asraf, S., Abdullasim, N. and Romli, R (2019) ‘Hybrid Animation: Implementation of Motion Capture’, IOP Publishing Ltd. doi: doi:10.1088/1757-899X/767/1/012065
• He, K. (2020) XR For Animation: The Future Is Less, Not More. Available at: https://www.vrfocus.com/2020/12/xr-for-animation-the-future-is-less-not-more (Accessed: 27/12/2021).
• Johnson, O. and Thomas, F. (1981) The Illusion of Life. New York: Disney Editions.
• Löring, R. (2021) Mocap in the cloud: The future of motion capture. Available at: https://www.broadcastnow.co.uk/tech/mocap-in-the-cloud-the-future-of-motion-capture/5159864.article (Accessed: 27/12/2021).
• Mou, T. (2018) ‘Keyframe or Motion Capture? Reflections on Education of Character Animation’, EURASIA Journal of Mathematics, Science and Technology Education, 14(12). Available at: https://doi.org/10.29333/ejmste/99174 (Accessed: 26/12/2021).
• Roberts, R. (2018) Converting Motion Capture into Editable Keyframe Animation. PhD thesis. Victoria University of Wellington. Available at: http://researcharchive.vuw.ac.nz/handle/10063/6924 (Accessed: 26/12/2021).
