Siggraph 2017 Rewind - Aude Guivarc’h: Hand Animation with Mocap in C4D
Share this video
00:29 | Explaining the stage canvas for projection visuals |
---|---|
03:26 | Introducing The Handler project |
07:06 | Motion Capturing hands with Brekel Pro software |
11:08 | Importing MoCap hands into Cinema 4D |
13:16 | Bind animated hand rigs to mesh using the Character Retarget tag |
15:27 | Set up camera for hands |
17:30 | Set up Render Settings |
19:12 | Correcting f-curves on keyframes with unwanted motions |
24:04 | Baking hand animation to 30 FPS |
26:16 | Lighting the scene |
27:34 | Process for creating hand tunnel effects |
30:04 | Creating hand decomposing effect using the PolyFX |
34:35 | Final results shown at concert |
Transcript
So we use Cinema 4D for a lot of our productions. From small motion design to
complex VFX work and even to control our robots and such. We do a lot of visual
compositions, with not only creating the visuals, but also the immersion and
sometimes even robotics. What I'll be talking about today is the Muse Drones
Tour that we created in 2015 and has been touring for two years.
First up, I'm going to put you in context and explain the amazing complexity and the
canvas that we've been working with. So this is a stage design that's made out of
tons of panels of semi-transparent fabric. This is the projection study that shows
you that we are projecting from both sides in a 360 stage, showing visuals exactly
the same way from one side to the other. We also do maquettes to be able to test
out our content, so that's a really interesting way to approach our
production is that, until we finally get there
on-site, we cannot really know how it will
look, so it's really useful to create these little maquettes. Give us a feel of
how the transparent material will look and how the ratio to the musicians on stage
will look like. We even had the access to these drones that were flying around the
stage and were also controlled by a 3D pattern that was pre-programmed during the
show. This is a first test of what I'm going to show later is the Kinect-
detected silhouettes that will then be controlled by these Mocap hands that I
will demonstrate a bit later. We have also a lot of other interactive features in
that show like face detection, other types of face detection like this.
We also used a real-life model that was also present during the entire show as
this androgynous figure that was dominating the entire stage, and kind of
taking over and being the leader of the performance. So we 3D-scan her.
Had this model in various forms throughout the show. We also did a lot of shoots.
We shot this model and suspended her in the air using the 3D as well as the shot
to composite this piece where she becomes decomposed in the air.
This is the final result. We also use the... Oh, sorry. This is another way we
used this 3D scan of the model to create these faces and hands and shapes appearing
through fabric. Which was a really great way to use the actual fabric that was
on-stage to give this impression of presence that comes in and out that kind
of has this creepy feeling. But let's get down to business and I'm going to talk to
you about one of the pieces that I created for this show, which is called The
Handler. That's the name of the song. And I'm going to show you how I did build
these hands that seem to control the musicians on-stage. So what you see here
is a 3D render with interactive content, which is the wires and the silhouette of
the musicians. That was all generated on-stage. So a cool part of my job for
this was to actually develop a storyboard with the director. And we thought of this
story, how it would evolve. And I really like to decompose each part of the song.
Not necessarily as shots, but as sequential elements that make the song
evolve and have these hands that are hands of this dominator androgynous figure
that's controlling everything on-stage. So for this one, I'm going to demonstrate
today is this really cool Mocap setup that I've been able to use for this show.
And I really needed something that was easy and I could change up stuff to be
really quick on my feet. And we found out that this really cool setup using the Leap
which is this little device right here that just plugged to the computer,
using another software called Brekel, allows you to do motion captures right
away on your computer super-easily. So this is kind of what I look like during
the first phase of me finding choreography that matches the music.
And we're used to just keyframing stuff and animating the old way,
and I found it absolutely exhilarating to be able to figure out a dance, really.
Become like a motion dancer and find a choreography to create this.
So this is kind of the first test. You see the wires are super-wobbly.
We had to find out how to do the physics a bit better and...just a...it was a first
test where we still had two hands controlling each musicians, but we decided
to go more for one hand on each side. One had for each musician.
So what was interesting with this is that we had to figure out a workflow,
not only to render out and animate these pair of hands, but also a workflow with
the interactive system. What this is, is an animation with a camera, and
at the same time, the same animation and camera in an interactive system producing
these generative wires leaving from the third knuckle of each hand,
and attaching and giving this type of feeling of restraint and physics to this
silhouette that was generated by Kinect at the end of the stage.
So it's kind of a simple setup animation- wise, but there was a lot of complexity
throughout this process to get these two together and work seamlessly.
So I'm going to go out of this presentation for now and I'm going to show
you just how I did this. So first up, I will load Brekel Pro. So what you see at
the bottom, I'll full-screen this, is me. Hello, this is my face.
Once I put my finger on my two hands, you can see that I have this skeleton right in
front of you working. It's very fluid, it's very seamless. You can see kind of
the edges of the box. If I get out of this the hand will either get all fucked up
or simply disappear. So you kind of have a good motion range. Me personally,
my limitations were the size of the screen. So I had a 1500 by 1440 pixel wide
screen for each hand to exist in. I decided to record the two hands at the
same time so that the entire motion would feel organic and human and like true and
not like disconnected. So I'll just show you, for example, I will record a
10-second little choreography and we'll record that and see how we can link this
to hands on live. So first up I will go in and decide where I want to put this FVX
animation.
I will drop it in here. Select folder. Perfect. Right now I'm sampling at 120
frames per second, which is awesome if you want to have a lot of freedom and
use a lot of slow-mo, and have a lot of little nitty-gritty details.
I will put it down to 60, because at the end, my animation is at 30 FPS,
but if I want to still have a bit of freedom and fine-tune some stuff I will be
able to do it. So I'll put file name...I'll just leave it at this.
It's not really important. So I will make sure it still sees my hands.
Start Recording. Put my hands in there. And let's do it. Let's do hand
choreography up, up, up, and like do crazy movements. And you can see when it changes
colors it's kind of a, "Ooh, this is forcing. I'm not so sure I understand what
I am." So let's say that this is a good recording and we'll take my hands away.
Stop the recording. Whoops. And go into the preview. See? What happened is that I
pressed Start Recording too fast, so I'll record again.
Let's say that I'm happy with this hand animation. Let's stop the recording.
Go into the preview. I'm just going to check real quick if that animation is
satisfactory. Okay, my hands are here. And I just want to really look out if there's
any glitch. Fingers flipping out and the hands really looking a bit weird.
Fingers intersecting and stuff like that. So this looks pretty chill,
pretty mellow. This will work. So I'll keep the take. Close this.
And now if I go into my FVX, I have my captures, so this is the last one.
This is the one I'll keep. And let's go into my rigged hands. So this is a model
that was created by one of our artists, [inaudible], and it was textured
by him as well. So I got these two hands that were rigged with exactly the same
bone structure as the FVX from Brekel. Because there's tons of ways to create a
skeleton. There's tons of ways to rig hands, but which is important here,
is to take exactly the same skeleton structure that your motion capture got
you, and to be able to integrate it perfectly. This will go smoothly.
So let's load up the last FVX we had. Say Yes to everything. And here we go.
So at first it will probably look like there's nothing here so I'll just select
this object. Hit H to find the object. And here is my animation.
So I did a first checkup in the Brekel software just to make sure that the
animation was fine, but I will still want to verify in here that everything makes
sense and that I'm still happy with it.
The third way to check it is to actually link
the hands and make sure that no fingers are intersecting and like no polygons look
super-weird. But first off what I'll do, I will bring up another...
I will bring up the timeline here. All right. So let's make sure that we see all
of our keyframes. What I did every time I wanted to record verse,
pre-chorus, chorus, is that right before the music would start at this specific
time, I would have kind of a cue in my animation. So it would kind of clap or
make something like really obvious so that I
know where in my animation I want to start
recording. And it's really important for me to know the exact perfect frame that
this song starts so that then when I give the FVX animation to the interactive
system, we are perfectly synced up and everything is actually perfectly synced up
to the music. So next thing that... I don't want to keep like this thing at the
beginning.
This is not really interesting. I said I want to keep these frames here up,
delete that, and especially delete the part at the end where there's one hand
that disappears. So I'm going to settle and keep it here. All right.
That's great. So that's good. Let's link the hands now. So what I'll do is go pick
up my rigged hands, paste it in my capture, or the other way around.
Whatever suits you.
All right. So I have my hands sitting idle by right here. And what we're going to use
is this really amazing and simple tag that's called the Retarget Tag.
So what this allows us to do is to find a skeleton structure and relink it to the
exact same one from our reference. So I put it on here. It doesn't really matter
where you put the tag but I've always done this this way and it worked perfectly
fine. So the source of the motion is from this other left arm that's in reference.
You can rename it if it's easier if you've got a lot of assets, but with just two
hands it was really easy for me to just pick out the right ones.
So the source of the motion is the one from our reference from Brekel,
and the target is this actual left forearm. Oh my God! This looks really
awful. I don't know why that is. I've asked myself that question for a very long
time before figuring out that it was simply that little checkbox right here
that makes everything fine right after.
That's not using hierarchy instead of names.
So let's do the same thing for this other hand. Character tag.
Retarget. I'm going to select the source of my motion and the target of it.
Uncheck this and we've got these super- cool, very human and very realistically
moving hands. And that was the most exciting part for me, is that you could
almost, you know, feel the identity of a certain human behind this.
It's...you really feel like there's something and someone is controlling these
hands. And in that case, it was me. I was controlling Muse and that was a really
special feeling. So now the next steps will be to light and set up the cameras.
I always remind myself this motion needs to be stuck and never go out of the
screens because I wanted to give the impression that these hands are actually
here. So first up, the important thing is to set up the camera.
I usually go with a 55 millimeter camera. And I'm going to set up each camera for
each hand. Then I will put them in layers and takes and make sure that you won't see
the right hand in the left front hand render, and vice versa. And for those of
you that just arrived, let me show you again what we're trying to accomplish
here. I have a movie to show you. So this is kind of the goal of the render here.
♪ [music] ♪
Another one I want to show you is this one. It will give you a good idea of what
we're trying to do. It's lit...the lighting is following the music,
and each hand is in each of the screen. What you see at the bottom is the mask
generated from the front fingers that allow the generative screen to not appear
in front of the other content. So that's kind of how we did it.
So you see that the lights are very dynamic and very responsive to the music.
How that was created is that I exported the light separately and animated them
in After Effects. There's two reasons for that. It's way faster to work this
type of animation unless you link it directly to the music in C4D,
but I wanted to have a quick workflow that was editable on-site and be able to change
everything really quick. So I did the animation in After Effects.
So let's go back to our render here. I will set up the size of my screen,
which was 1501...
Nope.
1501...by 1440.
So that gives it kind of a square ratio. It feels a bit weird.
To be able to see a bit better I hit Alt+V to be able to see more what my framing
is. Almost bring it down to 95%. So let's set up the camera for the right hand.
This guy. So the challenge here is that I want to be able to see the motion easily.
I want the hands to not feel too small in the scale of the stage,
because I want this to be really impressive. Like the musicians are
actually puppets controlled by this gigantic hand. But I don't want the hand
to go out of our board and be cropped by the screen. So it's going to be a bit of
back and forth feeling how the scale fits. So let's see that here.
I see that a lot of motion is going overboard. What's really fun with this
workflow is that I can easily change and tone down some of the motion.
So I'd say that I'm pleased with this scale. I did animation that was quite,
quite erratic. If you see the result after you'll see that I kind of toned down and
chilled a bit with the hands, because not only did it make the interactive wires go
crazy, but also it was really hard to keep the hands within the realm of that screen.
So for the purpose of this I will scale it back down. So zoom out with the camera a
little bit. And this feel nice. This feels like a good scale. I really like this
scale, but I also want to correct a few keyframes and maybe have the hand not go
out so much. Like, for example, here, what I'll want to do is go inside of the
keyframes of the animation and correct some curves and make sure that it doesn't
go out-of-bounds that much. So let's go find the keyframes for the right hand
right here. Let's say for that example, what I want to do is find what is doing
that crazy motion. Is it the arm? Is it the wrist here? So I'm going to hit the...
I'm going to actually hide this for now and go and grab the right joint.
And what I want to do is show which one of these... I want to show the right
keyframes for this. So I will go into View and link View to Object Manager.
So that will give me...whenever I hit on this I want to see this keyframe
specifically. It's really way easier to go find the source of your problem.
And what I want to do is look at these curves, and most of the time,
the rotation is responsible for these crazy out-of-bounds. So I'm going to go
and try to figure out if this is the problem.
I see this is what was the problem.
So I will scale up and go in there and see from where it seemed to be a problem.
See, I have two things here. It's also the rotation and the position of this arm.
So I'll start by killing all these keyframes here, for example.
And making those curves a little bit sexier, because whenever you have an
unsexy curve you have an unsexy animation. So let's say that I'm happy with this and
generally getting rid of all of these keyframes, for example.
Let's do that. Making this tangent chill. All right. And I will also lower the
position of this right forearm. So I will go into the position and see.
It's probably the Y so that's really easy. So for all of this position, it's really
annoying when you get... So I'm going to try to grab only... Nope.
All right. So I don't want it to go up. I want it to lower down in general.
So you can correct that pretty easily, and again, making sure that the curves are a
bit sexier. I could have also just motion captured everything at 30 FPS
so that it's easier to work with this kind of stuff, but again, I think it's
really nice to have still the choice of having the animation at a very good speed.
So I'm going to go back here. Get out of this View function. I'm going to keep this
here on the side. All right. So there I have all my keyframes here.
Let's say that... I could do this for hours except...absolutely exhilarating to
correct keyframes and correct animation curves. I personally think it's really
relaxing. But let's say that we're happy with this animation. It didn't go
out-of-bounds too much. And what I'll want to do is... Okay, this animation starts at
the right time. So I will make sure that the first frame of the document is
actually not zero, but is the first frame that my keyframes are at.
I could also slide the keyframes all to the zero points, but I have found in the
past that it's been super-heavy to do that. So instead, I will go to my
159th frame.
Go in my project settings, and set the movement to 159. All right.
Hit H to focus back on this. And what I'll do now is bake the animation at 30
FPS. So I want my project to be at 30 FPS.
So I will bake this animation for two reasons. One is to get the project way
lighter, and also to give this 3D information to the interactive team.
So to bake the animation, I will grab the two skeletons here that will be baked.
Right now I do not have any information on them. I will go in here and grab all their
children, which out of context sounds really weird. And I will also go and grab
these right here that are the keyframes that have the information.
So what I'll do is, in Function, Bake Object. I do not want to create a copy,
because I've found that when you create a copy and you take out the tag,
the information is still left on the previous bone. So I'd recommend to...do
not create a copy.
All right.
So, see, I did an error here and what I did is rebaked the exact same bone
structure that I had before. So I'm going to go ahead and do that again and select
in the timeline the right bones.
Set the children. Get these here. Function. Bake. Do not copy.
Hit OK. And there we go. Now we have the keyframes that created
this bunch of empty keyframes at the end because I didn't close down the entire
animation. But what I can do now is delete the reference and there we have these
hands that are at 30 FPS. This is animation that we have approved.
We can take out the tag and we have this all
baked in. So now what I'll want to do is
light up the scene. Very simple. My lighting setup was quite simple because I
wanted it to be very expressive and simply musical. So I had one on the side,
one on the left, one at the top, and ended up rendering the passes separately and
animating them in After Effects. Now before I go too deep on this, there's
a few other things that I wanted to show you that I did for the Muse concert.
So let's go back into the presentation. So I'll show you again the final result.
♪ [music] ♪
So after rendering all my passes and being really happy with all of these animations,
I brought them all together and had them exist in this environment.
On each screen I have one hand and left enough space at the bottom for the live
interactive generative content to exist.
♪ [music] ♪
So I'll just scroll a bit forward for you to see what else was happening.
I also had these...during the bridge there
was this crazy hand tunnel that I did using
some plugins in After Effects. But the interesting part is that I kept going back
into Brekel. I kept going back into motion capture and just recording a few
little movements with my hands and had them comped in various different ways.
So, for example, this is an array of like five different motion capture hands that I
could just record like one second or two seconds, made them loop,
and have these animations just go crazy however I wanted. So it was really cool.
♪ [music] ♪
So I will go up to the next video to show you another part of the show.
So this is a part that didn't make the cut, but it was a really,
really fun piece, and it was actually the most fun I've had with motion capture.
It was this really awesome and creepy intro.
♪ [music] ♪
So what I used for this was simply recorded a few hands going like this.
Like crazy, like, scratching on the floor and going like bat-shit crazy,
and laid them out in a tunnel, one behind the other, and had a very heavy fog that
would hide the ones in the back so that...and I offsetted all the
animations so that they would start pretty much when the camera would be seeing them.
So that unfortunately didn't make the cut. That's what happens a lot in the industry.
You have last-minute requests. Everything goes crazy. You work insane hours on-site
to fix everything to make things even better, and you have last-minute requests.
So the reason why this intro didn't make the cut is because we had a last-minute
request to merge the two songs together. So the song The Handler had to be merged
with a previous song that existed separately. So this is the result of that
transition. We had to find a way to merge these two styles together and create a
transition that would merge at the precise frame exactly to match the beginning of
the next song. So that was another case where I was super-happy to use this live
motion capture on-site, is that I just plugged in this little device and animated
my hand like kind of sitting like this, and not doing anything and kind of like
waking up and stretching the keyframes, I was able to create that motion where the
hands lift up. And using just...the keyframes would just glide into the next
part where the hands are already here and ready for the motion to appear.
So let me jump back into C4D where you have this hand that's in there kind of
decomposing and sliding in and out of textures and creating the next one.
So what you have here is basically three hands. A white hand, the color hand for
the end, and a black wireframe hand from underneath. So I used this little MoGraph
tag that's super-useful, especially since
I had a very already existing complex model
and didn't want to break it in a weird way. I just wanted to feel the transition
between that other piece that was really wireframe-ish and really black and white,
and transition it to this color hand. So using the PolyFX Tool and a few
shader textures, I was able to simply decompose. There's a...what I'll have here
is a scale, rotation, and a bit of randomness. So we can get close in here
and it's simply this scale going down. So yeah, basically the same principle for the
hand coming back on. It's this shader that's forming by the hand.
And in the back I have this wireframe hand that was actually not made with the
classic...what is it called? This would be the Atom Array,
it's simply the texture that is checkered black and white check with only one pixel
of white outside that's mapping this. So this is how I made that piece,
which ended up rendering super-well and communicated way...a very powerful
sequence where two songs merged together. And another fun last-minute request was
the outro, because we used to end the song with simply motion capture with the Kinect
and generative wires and everything. And we felt like, "No, this wasn't enough,
so let's create a third chorus with a little surprise at the end."
So there's no sound but I'll let you hear it after.
So what was happening is that I just created
like last minute this choreography for the final chorus and had my hands fall to the
ground. And that was actually really funny to do because I had to figure out a way
for my hands to feel like they were falling, but if I would cross a certain
boundary, this wouldn't detect it anymore. So I took me a bunch of times to figure
out how to kind of create like a little bump to absorb the fall of my hands so
that it would still be detected by the thing.
So I would like climb on a chair and kind
of like simulate that falling motion and like let my whole body drop for the motion
to work. So that was a really fun part of animating and being on-site and like doing
last-minute requests like crazy and getting all of this stuff done.
So to finish, I will show you the...well, the video that's kind of a recap of
everything that we did for the show.
♪ [music] ♪
All right, guys. My name is Aude Guivarc'h from Moment Factory.