Siggraph 2017 Rewind - Aude Guivarc’h: Hand Animation with Mocap in C4D

So we use Cinema 4D for a lot of our productions. From small motion design to complex VFX work and even to control our robots and such. We do a lot of visual compositions, with not only creating the visuals, but also the immersion and sometimes even robotics. What I'll be talking about today is the Muse Drones Tour that we created in 2015 and has been touring for two years. First up, I'm going to put you in context and explain the amazing complexity and the canvas that we've been working with. So this is a stage design that's made out of tons of panels of semi-transparent fabric. This is the projection study that shows you that we are projecting from both sides in a 360 stage, showing visuals exactly the same way from one side to the other. We also do maquettes to be able to test out our content, so that's a really interesting way to approach our production is that, until we finally get there on-site, we cannot really know how it will look, so it's really useful to create these little maquettes. Give us a feel of how the transparent material will look and how the ratio to the musicians on stage will look like. We even had the access to these drones that were flying around the stage and were also controlled by a 3D pattern that was pre-programmed during the show. This is a first test of what I'm going to show later is the Kinect- detected silhouettes that will then be controlled by these Mocap hands that I will demonstrate a bit later. We have also a lot of other interactive features in that show like face detection, other types of face detection like this. We also used a real-life model that was also present during the entire show as this androgynous figure that was dominating the entire stage, and kind of taking over and being the leader of the performance. So we 3D-scan her. Had this model in various forms throughout the show. We also did a lot of shoots. We shot this model and suspended her in the air using the 3D as well as the shot to composite this piece where she becomes decomposed in the air. This is the final result. We also use the... Oh, sorry. This is another way we used this 3D scan of the model to create these faces and hands and shapes appearing through fabric. Which was a really great way to use the actual fabric that was on-stage to give this impression of presence that comes in and out that kind of has this creepy feeling. But let's get down to business and I'm going to talk to you about one of the pieces that I created for this show, which is called The Handler. That's the name of the song. And I'm going to show you how I did build these hands that seem to control the musicians on-stage. So what you see here is a 3D render with interactive content, which is the wires and the silhouette of the musicians. That was all generated on-stage. So a cool part of my job for this was to actually develop a storyboard with the director. And we thought of this story, how it would evolve. And I really like to decompose each part of the song. Not necessarily as shots, but as sequential elements that make the song evolve and have these hands that are hands of this dominator androgynous figure that's controlling everything on-stage. So for this one, I'm going to demonstrate today is this really cool Mocap setup that I've been able to use for this show. And I really needed something that was easy and I could change up stuff to be really quick on my feet. And we found out that this really cool setup using the Leap which is this little device right here that just plugged to the computer, using another software called Brekel, allows you to do motion captures right away on your computer super-easily. So this is kind of what I look like during the first phase of me finding choreography that matches the music. And we're used to just keyframing stuff and animating the old way, and I found it absolutely exhilarating to be able to figure out a dance, really. Become like a motion dancer and find a choreography to create this. So this is kind of the first test. You see the wires are super-wobbly. We had to find out how to do the physics a bit better and...just a...it was a first test where we still had two hands controlling each musicians, but we decided to go more for one hand on each side. One had for each musician. So what was interesting with this is that we had to figure out a workflow, not only to render out and animate these pair of hands, but also a workflow with the interactive system. What this is, is an animation with a camera, and at the same time, the same animation and camera in an interactive system producing these generative wires leaving from the third knuckle of each hand, and attaching and giving this type of feeling of restraint and physics to this silhouette that was generated by Kinect at the end of the stage. So it's kind of a simple setup animation- wise, but there was a lot of complexity throughout this process to get these two together and work seamlessly. So I'm going to go out of this presentation for now and I'm going to show you just how I did this. So first up, I will load Brekel Pro. So what you see at the bottom, I'll full-screen this, is me. Hello, this is my face. Once I put my finger on my two hands, you can see that I have this skeleton right in front of you working. It's very fluid, it's very seamless. You can see kind of the edges of the box. If I get out of this the hand will either get all fucked up or simply disappear. So you kind of have a good motion range. Me personally, my limitations were the size of the screen. So I had a 1500 by 1440 pixel wide screen for each hand to exist in. I decided to record the two hands at the same time so that the entire motion would feel organic and human and like true and not like disconnected. So I'll just show you, for example, I will record a 10-second little choreography and we'll record that and see how we can link this to hands on live. So first up I will go in and decide where I want to put this FVX animation. I will drop it in here. Select folder. Perfect. Right now I'm sampling at 120 frames per second, which is awesome if you want to have a lot of freedom and use a lot of slow-mo, and have a lot of little nitty-gritty details. I will put it down to 60, because at the end, my animation is at 30 FPS, but if I want to still have a bit of freedom and fine-tune some stuff I will be able to do it. So I'll put file name...I'll just leave it at this. It's not really important. So I will make sure it still sees my hands. Start Recording. Put my hands in there. And let's do it. Let's do hand choreography up, up, up, and like do crazy movements. And you can see when it changes colors it's kind of a, "Ooh, this is forcing. I'm not so sure I understand what I am." So let's say that this is a good recording and we'll take my hands away. Stop the recording. Whoops. And go into the preview. See? What happened is that I pressed Start Recording too fast, so I'll record again. Let's say that I'm happy with this hand animation. Let's stop the recording. Go into the preview. I'm just going to check real quick if that animation is satisfactory. Okay, my hands are here. And I just want to really look out if there's any glitch. Fingers flipping out and the hands really looking a bit weird. Fingers intersecting and stuff like that. So this looks pretty chill, pretty mellow. This will work. So I'll keep the take. Close this. And now if I go into my FVX, I have my captures, so this is the last one. This is the one I'll keep. And let's go into my rigged hands. So this is a model that was created by one of our artists, [inaudible], and it was textured by him as well. So I got these two hands that were rigged with exactly the same bone structure as the FVX from Brekel. Because there's tons of ways to create a skeleton. There's tons of ways to rig hands, but which is important here, is to take exactly the same skeleton structure that your motion capture got you, and to be able to integrate it perfectly. This will go smoothly. So let's load up the last FVX we had. Say Yes to everything. And here we go. So at first it will probably look like there's nothing here so I'll just select this object. Hit H to find the object. And here is my animation. So I did a first checkup in the Brekel software just to make sure that the animation was fine, but I will still want to verify in here that everything makes sense and that I'm still happy with it. The third way to check it is to actually link the hands and make sure that no fingers are intersecting and like no polygons look super-weird. But first off what I'll do, I will bring up another... I will bring up the timeline here. All right. So let's make sure that we see all of our keyframes. What I did every time I wanted to record verse, pre-chorus, chorus, is that right before the music would start at this specific time, I would have kind of a cue in my animation. So it would kind of clap or make something like really obvious so that I know where in my animation I want to start recording. And it's really important for me to know the exact perfect frame that this song starts so that then when I give the FVX animation to the interactive system, we are perfectly synced up and everything is actually perfectly synced up to the music. So next thing that... I don't want to keep like this thing at the beginning. This is not really interesting. I said I want to keep these frames here up, delete that, and especially delete the part at the end where there's one hand that disappears. So I'm going to settle and keep it here. All right. That's great. So that's good. Let's link the hands now. So what I'll do is go pick up my rigged hands, paste it in my capture, or the other way around. Whatever suits you. All right. So I have my hands sitting idle by right here. And what we're going to use is this really amazing and simple tag that's called the Retarget Tag. So what this allows us to do is to find a skeleton structure and relink it to the exact same one from our reference. So I put it on here. It doesn't really matter where you put the tag but I've always done this this way and it worked perfectly fine. So the source of the motion is from this other left arm that's in reference. You can rename it if it's easier if you've got a lot of assets, but with just two hands it was really easy for me to just pick out the right ones. So the source of the motion is the one from our reference from Brekel, and the target is this actual left forearm. Oh my God! This looks really awful. I don't know why that is. I've asked myself that question for a very long time before figuring out that it was simply that little checkbox right here that makes everything fine right after. That's not using hierarchy instead of names. So let's do the same thing for this other hand. Character tag. Retarget. I'm going to select the source of my motion and the target of it. Uncheck this and we've got these super- cool, very human and very realistically moving hands. And that was the most exciting part for me, is that you could almost, you know, feel the identity of a certain human behind this. It's...you really feel like there's something and someone is controlling these hands. And in that case, it was me. I was controlling Muse and that was a really special feeling. So now the next steps will be to light and set up the cameras. I always remind myself this motion needs to be stuck and never go out of the screens because I wanted to give the impression that these hands are actually here. So first up, the important thing is to set up the camera. I usually go with a 55 millimeter camera. And I'm going to set up each camera for each hand. Then I will put them in layers and takes and make sure that you won't see the right hand in the left front hand render, and vice versa. And for those of you that just arrived, let me show you again what we're trying to accomplish here. I have a movie to show you. So this is kind of the goal of the render here. ♪ [music] ♪ Another one I want to show you is this one. It will give you a good idea of what we're trying to do. It's lit...the lighting is following the music, and each hand is in each of the screen. What you see at the bottom is the mask generated from the front fingers that allow the generative screen to not appear in front of the other content. So that's kind of how we did it. So you see that the lights are very dynamic and very responsive to the music. How that was created is that I exported the light separately and animated them in After Effects. There's two reasons for that. It's way faster to work this type of animation unless you link it directly to the music in C4D, but I wanted to have a quick workflow that was editable on-site and be able to change everything really quick. So I did the animation in After Effects. So let's go back to our render here. I will set up the size of my screen, which was 1501... Nope. 1501...by 1440. So that gives it kind of a square ratio. It feels a bit weird. To be able to see a bit better I hit Alt+V to be able to see more what my framing is. Almost bring it down to 95%. So let's set up the camera for the right hand. This guy. So the challenge here is that I want to be able to see the motion easily. I want the hands to not feel too small in the scale of the stage, because I want this to be really impressive. Like the musicians are actually puppets controlled by this gigantic hand. But I don't want the hand to go out of our board and be cropped by the screen. So it's going to be a bit of back and forth feeling how the scale fits. So let's see that here. I see that a lot of motion is going overboard. What's really fun with this workflow is that I can easily change and tone down some of the motion. So I'd say that I'm pleased with this scale. I did animation that was quite, quite erratic. If you see the result after you'll see that I kind of toned down and chilled a bit with the hands, because not only did it make the interactive wires go crazy, but also it was really hard to keep the hands within the realm of that screen. So for the purpose of this I will scale it back down. So zoom out with the camera a little bit. And this feel nice. This feels like a good scale. I really like this scale, but I also want to correct a few keyframes and maybe have the hand not go out so much. Like, for example, here, what I'll want to do is go inside of the keyframes of the animation and correct some curves and make sure that it doesn't go out-of-bounds that much. So let's go find the keyframes for the right hand right here. Let's say for that example, what I want to do is find what is doing that crazy motion. Is it the arm? Is it the wrist here? So I'm going to hit the... I'm going to actually hide this for now and go and grab the right joint. And what I want to do is show which one of these... I want to show the right keyframes for this. So I will go into View and link View to Object Manager. So that will give me...whenever I hit on this I want to see this keyframe specifically. It's really way easier to go find the source of your problem. And what I want to do is look at these curves, and most of the time, the rotation is responsible for these crazy out-of-bounds. So I'm going to go and try to figure out if this is the problem. I see this is what was the problem. So I will scale up and go in there and see from where it seemed to be a problem. See, I have two things here. It's also the rotation and the position of this arm. So I'll start by killing all these keyframes here, for example. And making those curves a little bit sexier, because whenever you have an unsexy curve you have an unsexy animation. So let's say that I'm happy with this and generally getting rid of all of these keyframes, for example. Let's do that. Making this tangent chill. All right. And I will also lower the position of this right forearm. So I will go into the position and see. It's probably the Y so that's really easy. So for all of this position, it's really annoying when you get... So I'm going to try to grab only... Nope. All right. So I don't want it to go up. I want it to lower down in general. So you can correct that pretty easily, and again, making sure that the curves are a bit sexier. I could have also just motion captured everything at 30 FPS so that it's easier to work with this kind of stuff, but again, I think it's really nice to have still the choice of having the animation at a very good speed. So I'm going to go back here. Get out of this View function. I'm going to keep this here on the side. All right. So there I have all my keyframes here. Let's say that... I could do this for hours except...absolutely exhilarating to correct keyframes and correct animation curves. I personally think it's really relaxing. But let's say that we're happy with this animation. It didn't go out-of-bounds too much. And what I'll want to do is... Okay, this animation starts at the right time. So I will make sure that the first frame of the document is actually not zero, but is the first frame that my keyframes are at. I could also slide the keyframes all to the zero points, but I have found in the past that it's been super-heavy to do that. So instead, I will go to my 159th frame. Go in my project settings, and set the movement to 159. All right. Hit H to focus back on this. And what I'll do now is bake the animation at 30 FPS. So I want my project to be at 30 FPS. So I will bake this animation for two reasons. One is to get the project way lighter, and also to give this 3D information to the interactive team. So to bake the animation, I will grab the two skeletons here that will be baked. Right now I do not have any information on them. I will go in here and grab all their children, which out of context sounds really weird. And I will also go and grab these right here that are the keyframes that have the information. So what I'll do is, in Function, Bake Object. I do not want to create a copy, because I've found that when you create a copy and you take out the tag, the information is still left on the previous bone. So I'd recommend to...do not create a copy. All right. So, see, I did an error here and what I did is rebaked the exact same bone structure that I had before. So I'm going to go ahead and do that again and select in the timeline the right bones. Set the children. Get these here. Function. Bake. Do not copy. Hit OK. And there we go. Now we have the keyframes that created this bunch of empty keyframes at the end because I didn't close down the entire animation. But what I can do now is delete the reference and there we have these hands that are at 30 FPS. This is animation that we have approved. We can take out the tag and we have this all baked in. So now what I'll want to do is light up the scene. Very simple. My lighting setup was quite simple because I wanted it to be very expressive and simply musical. So I had one on the side, one on the left, one at the top, and ended up rendering the passes separately and animating them in After Effects. Now before I go too deep on this, there's a few other things that I wanted to show you that I did for the Muse concert. So let's go back into the presentation. So I'll show you again the final result. ♪ [music] ♪ So after rendering all my passes and being really happy with all of these animations, I brought them all together and had them exist in this environment. On each screen I have one hand and left enough space at the bottom for the live interactive generative content to exist. ♪ [music] ♪ So I'll just scroll a bit forward for you to see what else was happening. I also had these...during the bridge there was this crazy hand tunnel that I did using some plugins in After Effects. But the interesting part is that I kept going back into Brekel. I kept going back into motion capture and just recording a few little movements with my hands and had them comped in various different ways. So, for example, this is an array of like five different motion capture hands that I could just record like one second or two seconds, made them loop, and have these animations just go crazy however I wanted. So it was really cool. ♪ [music] ♪ So I will go up to the next video to show you another part of the show. So this is a part that didn't make the cut, but it was a really, really fun piece, and it was actually the most fun I've had with motion capture. It was this really awesome and creepy intro. ♪ [music] ♪ So what I used for this was simply recorded a few hands going like this. Like crazy, like, scratching on the floor and going like bat-shit crazy, and laid them out in a tunnel, one behind the other, and had a very heavy fog that would hide the ones in the back so that...and I offsetted all the animations so that they would start pretty much when the camera would be seeing them. So that unfortunately didn't make the cut. That's what happens a lot in the industry. You have last-minute requests. Everything goes crazy. You work insane hours on-site to fix everything to make things even better, and you have last-minute requests. So the reason why this intro didn't make the cut is because we had a last-minute request to merge the two songs together. So the song The Handler had to be merged with a previous song that existed separately. So this is the result of that transition. We had to find a way to merge these two styles together and create a transition that would merge at the precise frame exactly to match the beginning of the next song. So that was another case where I was super-happy to use this live motion capture on-site, is that I just plugged in this little device and animated my hand like kind of sitting like this, and not doing anything and kind of like waking up and stretching the keyframes, I was able to create that motion where the hands lift up. And using just...the keyframes would just glide into the next part where the hands are already here and ready for the motion to appear. So let me jump back into C4D where you have this hand that's in there kind of decomposing and sliding in and out of textures and creating the next one. So what you have here is basically three hands. A white hand, the color hand for the end, and a black wireframe hand from underneath. So I used this little MoGraph tag that's super-useful, especially since I had a very already existing complex model and didn't want to break it in a weird way. I just wanted to feel the transition between that other piece that was really wireframe-ish and really black and white, and transition it to this color hand. So using the PolyFX Tool and a few shader textures, I was able to simply decompose. There's a...what I'll have here is a scale, rotation, and a bit of randomness. So we can get close in here and it's simply this scale going down. So yeah, basically the same principle for the hand coming back on. It's this shader that's forming by the hand. And in the back I have this wireframe hand that was actually not made with the classic...what is it called? This would be the Atom Array, it's simply the texture that is checkered black and white check with only one pixel of white outside that's mapping this. So this is how I made that piece, which ended up rendering super-well and communicated way...a very powerful sequence where two songs merged together. And another fun last-minute request was the outro, because we used to end the song with simply motion capture with the Kinect and generative wires and everything. And we felt like, "No, this wasn't enough, so let's create a third chorus with a little surprise at the end." So there's no sound but I'll let you hear it after. So what was happening is that I just created like last minute this choreography for the final chorus and had my hands fall to the ground. And that was actually really funny to do because I had to figure out a way for my hands to feel like they were falling, but if I would cross a certain boundary, this wouldn't detect it anymore. So I took me a bunch of times to figure out how to kind of create like a little bump to absorb the fall of my hands so that it would still be detected by the thing. So I would like climb on a chair and kind of like simulate that falling motion and like let my whole body drop for the motion to work. So that was a really fun part of animating and being on-site and like doing last-minute requests like crazy and getting all of this stuff done. So to finish, I will show you the...well, the video that's kind of a recap of everything that we did for the show. ♪ [music] ♪ All right, guys. My name is Aude Guivarc'h from Moment Factory.