Siggraph 2017 Rewind - Eric Nicolas Smit: 3D Motion Tracking and VFX in C4D

Photo of Cineversity

Instructor Cineversity

Share this video
  • Duration: 42:01
  • Views: 10229
  • Made with Release: 19
  • Works with Release: 19 and greater

Eric Smit, master Cinema 4D artist from the Maxon headquarters in Germany, visits us today to show the best ways to use the Motion Tracker tools inside Cinema 4D for VFX shots.

This presentation starts with a traffic shot taken from the Golden Gate bridge. Eric begins by using the Motion Tracker tool to create a point cloud in 3D space, then shows how to use the Motion Tracker tags to more accurately reconstruct the shot in 3D space. From there Eric adds a car model, and shows how to properly add lights, shadows and reflections to the car model so that realistically blends within the shot. He moves onto showing how to use the Camera Calibrator tag to do the same type of effects with static cameras. The presentation wraps up by showing us the new Recontruction features in the Motion Tracker, new to R19. This feature allows you to more easily see the reconstructed 2D scene through the point cloud in 3D space. This presentation is a must for anyone looking to learn motion tracking for VFX inside C4D.

03:30Motion Tracking Golden Gate Bridge shot
10:26Using Motion Tracker tags to help reconstruct scene
15:23Create a ground plane aligned with the motion tracked scene
17:32Texturing ground plane so it's seemless with footage
18:47Import car model into motion tracked scene
20:35Animating car model driving on road
22:00Create lighting, cast shadows, and ambient Occlusion for car model
25:07Adding realistic reflections to the car model using HDRI
28:14Adding Motion Blur to the scene
30:23Rendering the scene with Multipasses for further post production work
32:07Reconstructing 3D scene for a static image using the Camera Calibrator tag
38:00New 3D Reconstruction feature in R19

show less

Transcript

Hi, there. So I'm Eric Nicolas Smit. I'm a visual designer. I come from Luxembourg in Europe and it's great to be here to talk to you about my work and also about my tool, Cinema 4D. So this presentation will focus on motion tracking and visual effects or how you can integrate 3D elements in film-it sequences. Now, this is a very difficult task, because even if you have a very nice 3D model, putting it in front of your video won't make it. You need to match the camera setting, the camera angle, the camera motion. You also need to match the illumination, the co-stakes [SP], the reflection. It's a very long work, but luckily, with Cinema 4D, we can do it a bit faster. So what I would like to do now is to work with you on two visual effect sequences. We will be integrating a car in a film-it sequence. But before starting, I wanted to show you a bit of my work, and especially my work in the visual effects. So this is an example of a 3D model integrated into a film-it sequence. It's from my short movie, ControFigura, and I'll just play it back, and we have other examples in this video. So here, I integrated the statue in this place that was filmed and here, you can see the statue alone. I sculpted it in Cinema 4D. Here is the location with the model, the textured model, and then with atmospheric effects. And here is another shot from the same short movie. Here, it's a static camera. And another one where I actually [inaudible] a statue and even replaced the vegetation. And the last one from this short movie. And these shots comes from a serie of spots advertising that I did for this company. I integrated the truck in many different location. The idea was to express the expansion of the company by making it travel all over the world. So it goes to London. Here, you see the shot without the truck. And here you see Paris, in front of the Arc de Triomphe and the truck is right here. And again, Paris on the Boulevard with the truck. And the final shot in Boga-Boga Island. Okay, so now here's the scene we will be working on. I'll try to do it from beginning to end. So we will do every steps. The first one will be 2D Tracking, then we will be doing 3D Solving, then integration, and then rendering, and post production. This is a view of the Golden Gate Bridge. The camera is moving with the traffic and the camera is on a truck. And we travel from the gates to the exits. So here, I have already loaded my Motion Tracker. This is the tool that I will use to get the camera movement, the camera angle, and also the scale, and the orientation of the scene. So this tracker will help me to get a 3D camera that matches the 2D camera in the footage. So this is my sequence and here, I have my Motion Tracker. I've already loaded the footage here. It's a stock footage. And then here, in the 2D Tracking, I have many option. I can use Auto Track, I can use Manual Tracking, but I will try Auto Track now. I think it's always better to try it with a Auto Process and then fix later on if you see problems. I'll be lazy and I'm just click here, Auto Track. And, oh, when I click Auto Track, Cinema 4D will scan the image and try to find features that stay in view and that can be traced for a very long period. So it can get an ID of the movement in 2D and then extract this data and transform it into a 3D motion. So I'll run the Auto Track here. And it's now processing the frames and the 2D Track is finished. It was quite fast. So now we see that we have points here that appears on my footage and these points are attached to the structure of the footage. We see the points on the bridge here and on the fence, on the concrete blocks, and they are all pretty glued to these structures. There is, however, a few tracks acting weird. If you look here, in this area here, right here, one is just moving a bit erratically, chaotically. This is an error and we need to remove it. So I'll just grab it here and just delete it. And then we have also tracks here in front of the car, right here. We have sliding tracks. These are errors. These are issues that we need to get rid of. Also, you'll notice that we have tracks here on the car and we don't want that. We don't want to trace the movement of the car. We just want to trace the movement of the camera. So I'll just have to remove them as well. I'll just click Delete. This one as well. This cleaning is necessary. You cannot avoid it. Oh, sorry, up, Delete. And that should be it. We have also here, sliding tracks. I'll just get rid of them. And here, again, a track acting weird. And instead of doing this manually, I could also use here, a filter that is called Smart Acceleration. And Smart Acceleration, it will analyze groups of tracks and see if they all behave coherently, if there is not one that is acting really weird compared to the other. So if I reduce here the Smart Acceleration, you'll see that many tracks will disappear and these are the bad ones. These are the one that are just noisy and with an incoherent behavior. So Cinema 4D gets rids of them. I'll just remove here, an extra track here in the sky, because it's useless to have a track on the sky, because there's absolutely nothing to trace. So now it seems we have a decent 2D Tracking. I'll remove just this one and then I'll stop. So we can now do the 3D Solving. And 3D Solving, Cinema 4D will just take these 2D Tracks, and try to find their position in a 3D space and also try to find what was the camera movement and the focal lengths. So here we go in 3D Solving. And the Solve Mode, I'll leave it to Full 3D Reconstruction. We have also Planar Reconstruction and Nodal Pan and these are for a static camera or camera moving just like this. But here, we want to analyze it with a full movement of the camera. And we have here, the focal length and I said it's known and constant. Constant because I doubt that the focal length was changed in this sequence and known because it's always good to indicate a focal length. I will talk about this later in the second part, but to get this 24 millimeters, I actually rent a camera calibrator. But this will be the second part of my presentation, so just keep it in mind and I'll talk about it later. So here, I have all the settings to run the 3D Solver. I'll just click on it and we will see what happens. So it's running the 3D Solver and the solving is finished. And now we have what is called Features. And you see all these big green circles, and if I move my camera, they are glued to every element in the footage: the road, here we have the lamp, the fence. They stick to it. And if I move away from my source camera here in a 3D view, you can see my points here, my features. It's a representation of the 3D space. And then my camera is moving. I go back here to my source camera. And we see here, all the points in the 3D space. Now, if we start adding geometries here, for example, a plane...just move it here. Let's move it on the z-axis. The issue here is that its position and its orientation are not very useful. We don't know where this plane should be in the footage. It's just in the air and we have absolutely no notion of its position compared to the film-it sequence. So what we need to do now is to calibrate the cluster of points here, the group of features, to get a scene that is correctly orientated at the correct position, at the correct scale. So then we will be able to integrate our car or any other geometries. So to transform this cluster of point that is just not organized, I need to use a special tag. These are the Motion Tracking Tags. And we have here, in the motion…sorry, Motion Tracking Tags, we have many different tags. We have Planner, Position, Vector, and these are all little helpers that will help us to transform this unorganized cluster of point into a usable 3D scene. So let me switch here to the Motion Tracker Layout here. I'll go back to my camera. And here, I just had these constrained. Now, let me talk about them a little bit more. We have here...the first we will be using is the Position Constraint here. And the Position Constraint, what it will do is that it will move the whole group of point to a position in the 3D space. And for me, I want this group of point to have its center on the road. I want to grab one point and make it the center of my 3D universe. So for this, I will grab, here, my Anchorman in Cinema 4D. It's the Create Position Constraint. And now, I have to click on a point here, on one of the track, and this will be the center of my world. So I'll try to find one that could act as this. And I will grab this point here, right here, and I'll say that this is the center of my world. I click on it and now I have a new tag. It's the Position Constraint. And it says, "Feature to Position." It means that the whole group is now moved here, linked to this point, and this will be my zero origin, the center of my world. Now that we have done this, we need to set the orientation of the scene and I want my scene to be orientated exactly in the same direction as the road. So I want my z-axis to be on the road. So for this, I need to use a Vector Constraint and for the Vector Constraint, I cannot use a single point. I'll have to use two points and say to Cinema 4D that these two points are an axis. So now, I need to find two points that are correctly aligned here in this direction. So we could use, for example, these two points because they are on the z-axis. We could also use here, the marking on the road as a line. So I need to check, to search for two points that could be perfect for this. So let me check. Here, I found two here just on the markings, this one here. So I'll click here, the Vector Constraint command. I'll just click on the first here and on the second and I will say here that the axis should be the z-axis. So now this line is the z-axis of my scene and I will say that the length is known. So this line is probably… Let me check by just putting a car next to it. I would say it's maybe, 6 meter, 8 meter. I'll just try with 8 meter. Okay. So now my scene is correctly scaled. You can see that the point are now much smaller. And now I can just put geometry cars and they will be at the right position, at the right angle, and maybe at the right size as well. So let's start by placing here, a plane. So I'll just put a plane here. And you see it's directly here at the center that I indicated with my Position Constraint. It's exactly where I had my track. So I'll just make it larger. This could be my road, like so. And just longer. Let's try this. Like so. And I'll switch here to a Line Shading, so we can see the perspective lines. Okay. So now you see that my plane here is exactly aligned to the marking on the floor. So that means that my perspective are correct. It's also aligned here to the border. Let me move it a little bit here and maybe reduce its size. We only have a little issue here on this side. We see it's not exactly aligned. I think it is caused by this road that is not perfectly flat. There is maybe a curve to it on this side or this side, so we need to fix it. You always need to fix a Motion Tracking because it's never perfectly correct. But this was already pretty good, but I'll just fix it here, by changing the coordinates. I'll try 0.5 here. No, that's worse. I'll just use minus. Okay, that's already better and maybe here. I'll just Shift-click here. No, that's too much. Zero, it's at 0.2. Okay. Yeah, that's correct. I think now that we have something that is correctly aligned, more or less… And this now can be my road. So I have a road and this is already a big part of the job because the next step will only be to put a car on top of it. So this road will not be visible. I only want it to be an object that catch shadows and occlusion. So I will make it invisible. I'll now switch to my standard layout, because we don't need the tracking anymore, by dropping the footage in it. So this is my road. I drop the footage, and I'll switch here, the Mapping to Camera Mapping, and I'll drop here, my camera. It means that it will get… The footage will be projected on it exactly as it is on the view we see behind it. And since I don't want it to be visible, I'll just put on it a Compositing Tag. So we have here, a Compositing Tag and I'll say, "Compositing Background." It means when I render it, it won't have any illumination, any shading. It will be just flat, just like the background. And I'll just place another tag here, a Display Tag, and switch here to Constant Shading, so that it doesn't get in the way and we can work without noticing it. So now my setup is ready. I have my road. I'll make it a bit larger. I can bring in my car and I would like to have a car here in the center line, right here, going from here to here. So I'll just open up my car. It's right here. This is my car, my sport car, the one that we will be integrating in the shot. It's already animated. I just animated the wheels and also included a little bit of vibration to simulate irregularities on the road. It's just a little Vibration Tag here. It's just a random motion, a bit like to simulate the bumps and to make the movement a bit more interesting. So now I can grab my car. I'll just copy it and I'll just load it in my scene, paste. Switch the display here to Quick Shading. And let me [inaudible] like this. Here, it's right here. So we have the car now in the scene and in a static position. You can see that it's already correctly integrated. It doesn't move. It's fixed to the road. I'll just have to adjust its size a little bit here. It seems a bit small, so I'll make it a bit larger. Okay. It seems okay. Now I'll make it the trial of my road, to just change its orientation so that it's perfectly aligned to the road. Okay. And now, I'll just animate it, the car, and I will animate it just by using two keyframes. One for the start, one for the end. So this is my beginning. I'll just check if I'm really on the road, yeah. And I'll add another one at the end. Switch here to this view, like so. So, here again. And I'll check again. I have to make my road a bit longer. Oh, that's smaller. Okay. I'm just checking if it's correctly positioned on the road. Then I have here, the second keyframe. Okay, so my car is now animated in the scene. I will switch the Curve here to a Linear Curve, because I want a constant speed. So I'll go here, open the Curve Editor, and just switch here to a Linear curve. So now the car has a constant speed, in my view. And in terms of movements, position, perspective, alignment, it's more or less correct. I could work on it a little bit further. But now if I hit Render, the issue is that there is no interaction between the car and the road. We have here, just a black car. There's no shadow, no occlusion, and it doesn't look like it belongs to the same environment. So what we need to do now is to work on the interaction between the car and its environment and this is done by using shadows, reflection, occlusion, and all kinds of effects that can create a link between the world in the footage and my 3D object. So the first thing I will do is to add a sunlight, a direct light. And to find its orientation, it's easy in this shot because we have very strong shadows. And we see that we have shadows right here, below the car. That indicates that the sun is coming from the top, that it's the middle of the day. And if I look at the highlights here, I can see that the sun is coming a little bit from the side. We see it also here, in the concrete block. It's not totally vertical, so I will just create here, a sun. I will use an infinite lights. Like so. And what I will do to set its orientation, I will use here, a command that is called Set Active Object as Camera. This allows me to see the scene from the light as if I was holding a projector. So I'll click on it. And I can now just move around. And I have my car. So we have a God-sun view. So we see my car. And I've just put my light right on the top here, a bit from the side. And I'll just activate here, Shadows. I'll use Area Shadows. And now, if I move back to my view, hit Render, we have shadows here, that match more or less the orientation of the other one. Now what is missing is a diffuse lights. We have only one direct lights. So I'll re-create a diffuse lights. I will do a quick diffuse light using an Evironment Object. An Environment Object creates a totally flat illumination. And I will include, as well, Ambient Occlusion here in the effects. Ambient Occlusion. Ambient Occlusion plus Environment Light creates a very fast and cheap… I mean, you don't need any kind of complicated setup. It creates a nice global illumination, an omnidirectional global illumination. Now, if I hit Render, we have now…the car has now more shadowing and the contact between the car and the road, we see has a bit of occlusion, here. We don't notice that much, but there is a bit of occlusion. Now what is missing is the reflection. We have the car and it's now totally black because it doesn't reflect anything. This car has a metallic shader, so it should reflect something. There should be something around it that creates, generates reflection. But since the footage is behind the car, it's totally black. Our universe is completely black. So what I need to do is to create something that will create the reflection. So for this, I will be using an Environment Map on a sky. So I create a sky here and I create a new material with an Environment Map. I will a use High-Dynamic-Range Environment Map here. It's an image that is more or less similar to my bridge. It's a structure with, yeah, the same kind of color tone. So I drop it here, on the sky. And now we have a sky, but I don't want it to be visible. I want to see the footage, but not this image. So I will make it invisible to the camera, but visible only to the reflection. So I use...here, for this, I use the Compositing Tag. Compositing Tag are really useful for visual effects. And I'll just turn off everything excluding the scene by reflection. So now this sky will only be seen by the reflection. So now if I hit Render, we have a car with a reflection on it. So the reflection are a bit too strong, so I will reduce here, the intensity of my High-Dynamic-Range Image, like so, and maybe change its color a little bit to match the sky in the footage. And now, if I hit Render, we have something much better. That's maybe too pinky. Okay. And now to set the orientation of the sky, I will use here, just a sphere here in my scene, make it larger, and drop here, the same material and just rotate it, so that the light is coming from the same direction, from the front and a bit from the top. And then I'll use the rotation value of this sphere. I'll just copy it to my sky. So I'll just go into Coordinates, copy this value, copy and paste it in my sky. And it's done. Just delete my sphere. And now, we should have a more accurate reflection, something that is more [inaudible] to the scene. And so here we have my car driving on the Golden Gate Bridge. So what is missing now is that we don't have any kind of motion blur. And this is essential when you're doing compositing shots. You need to match the motion blur, otherwise, it will look totally fake. Now, since the camera is moving with the traffic, we don't have that much blur. The relative position between the car and the camera is minimal. On the other side, we have a very strong motion blur, but here, not that much. If we check this car here, the car here on the right, it has a strong motion blur, but not as intense as the other one. And we need to match this. So for this, we will activate Cinema 4D's motion blur, it's in the Physical Settings here. I have the Motion Blur. And now, the only thing that I have to do is to adjust the shutter speed of my physical camera. It's now set to 130 and we'll just try with 130 to see how it looks. Since motion blur is calculated at the end, we have to see it in the Picture Viewer. So I'll open the Picture Viewer and make a test render. So now we have the car with the motion blur. If we compare it to the other car, it looks already more integrated. I'll just move it a bit. Let's bring it further here. And I'll hit... I have Render again. Okay, now we see there is a stronger motion blur when it reach the lower part of the image and it's looking already really good. Now, the next step for us would be to...of course, we could work further on it. We could work on the shaders, improve the illumination, improve the pass [SP] of the car. But we don't have much time, so I will move to the next step which is rendering and post production. So now what I do is that I render this scene in multi-passes. And in multi-passes, what I need is a mask for the car, so I can filter it independently, and then I'll render also shadows pass, occlusion pass. I'll show you how they look here. So here, I have my rendered. I rendered it a few days ago. [inaudible] here. No, this is not it. So here's the render. Okay. And here, I have my car mask, an object buffer, and here, I have my ambient occlusion pass. And here, I have the shadow pass. And now what I do is that I blend them in my compositing software. It can be After Effect on any other. And then I filter each passes and add a grain, I add a color mapping, I add all sort of highlights, glows, to mimic better the style of the footage. Because we can see here, in my footage, that we have very strong highlights here. Some sort of glows, shine. So that I need to reproduce not in Cinema 4D, but in a post production package. So I will show you now the finish-it sequence and this is the final sequence with post production. I'll just play it back. Okay. So that was tracking and compositing for an animated camera. But sometimes, you have to work with a static shot. And if you have a static camera shot, then the Motion Tracker is useless because you cannot trace something that is static. There is nothing to trace, nothing to solve. So for this, we're using a different set of tools and I'll switch now to a different scene here. It must be… It's loading now. It takes a little while. Okay. Now, this is a shot here in Paris. You saw it earlier in the video, on the Boulevard, and the camera is static. So how can we get now the camera angle, the camera position, and all the information we need to make a correct integration if the camera is not moving? So if it's not moving, we only need the focal length and the camera orientation. And this, we can find it using a tag that is called the Camera Calibrator. And the Camera Calibrator is a tool that extracts the focal lengths, the orientation, and the position of the camera from the footage. And to make it work, we need to trace lines in the footage. And these lines will be my axis. We need to use parallel line for each axis, so Cinema 4D can calculate the vanishing point. So what we do here, is we click on the tag here and we go in Calibrate. And we add a line. So I add a line here and this line, I will place it right here to here and I will say that this will be my z-axis. Again, I want the z-axis to be in the traffic here, along the road. So I Shift-click on it and make it blue. In Cinema 4D, the z-axis is always blue. Now, I need to add a second one so that we have parallel lines for the vanishing points. So I click another line here and I just place it here and here. And again, Shift-click on it to make it blue. So now we see that the z-vanishing point is already solved and we need to add another access to get the final result, so I click Add Line. And now, I will work on the y-axis, the vertical axis, and I will place it here, on the edge of this building. And Shift-click on it to make it green. Green is always y. And then I add another line here and put it here and here. It could be in other place, it doesn't matter. And again, I Shift-click to make it green. Okay, now we see that the x-vanishing point is indirectly solved. The y-vanishing point is solved, z is solved as well, the camera focal length is solved. Everything is solved. So Cinema 4D knows exactly what is the camera focal length, but what we miss is the camera position. And this is done using a pin. So we add a pin here and the pin appears here. It's a bit small. And we'll just put it right here, at this position. And this will be the center of my world. Now, if I click on this access, I can also say that this line has a known length. Here, Z, I can say "Yes," I activate it, and I will say that this line is maybe, let's say, 10 meter. Okay. So now everything is solved. We have a camera. If I move away, here from the shot, I'll just hide it. [inaudible]. And click. And I'll just move here, here. Just hide the background. And, okay, I have my camera here, and Cinema 4D evaluated it to be a 46 millimeter. And we have its position and orientation. And now, again, if I want to place my road, I just have to add a plane. Move it here. I'll switch to Display to Line. And I can just adjust its scale. And we have here, a road that is correctly aligned. You know, the perspective are totally accurate. So now I have my road. And again, I can add the footage as a texture, the composting tag, bring in my car, and do exactly the same that we did before but I won't do it. I'll just move to the finish-it scene because it's exactly the same procedure. So let me find my scene here. Oh, I need to open from here. So here, we have the complete scene with the car and the traffic. And again, I have a sky, and environment illumination, and a sun, and my plane as the road. So you can see it here, my car on the road. Okay, so that was my integration, little examples, my two example. One was an animated camera, one was a static camera. But now I would like to talk to you about a brand-new feature in Release19. It's 3D reconstruction. And what we have now is that if I switch now back to my Golden Gate Bridge sequence… Let me load it. Sorry. Here, my Golden Gate Bridge sequence. What we had from the 3D Solving was only a grid of points, a cluster of features, these little green dots. But now with Release19, Cinema 4D offers you more. And I will show you in a different scene where the effect is really striking. Here, I'll switch to the...here, my building. So we have here… Let me charge it. We have here, fly over buildings. And this shot was already tracked and solved. And we can see here, the points that we have, the cluster of point. If I move out of this view, we have here, this cluster of points. But now we have an extra tab here in the Motion Tracker. It's Dereconstruction. And here, Cinema 4D will try to re-create the scene in 3D. So it will map the footage on a 3D cloud point. So I'll show you how it looks. I could run it now, the run 3D here, the reconstruction, the Run Scene Reconstruction, but since it'll take a few minutes, it was 10 minutes, I'll just skip to the result. So here, we have the result here. And I'll just here, hide everything. This is what we have now from the 3D reconstruction. So what Cinema 4D did, it's tried to re-create the image but in a 3D space. If I move away from my camera here, we can see the building, the structure, the color, and this is extremely useful if you're doing compositing. Because here, you can really understand the 3D shape of the scene, you get a sense of distance, you see the color. You can even see here, plants in the windows. So this is a very nice indication of where is what in your 3D scene. It's much better than if I switch off the point cloud to have just here, a cluster of point. This is way more useful. And if I open here, my previous scene, the one on the Golden Gate Bridge, here, I did the reconstruction as well. It took here, maybe 20 minutes. And here, I can really see the bridge, and see where my car is traveling in the 3D space. I see also the large concrete blocks and the bridge structure. And this is really useful if you need to reconstruct the scene, if you need to replace elements. You have really a fantastic visualization of the environment. Okay, so this was my overview of motion tracking, camera calibration, and integration of 3D elements in 3D scene. I hope it will stimulate you to try it by yourself. I will be here on the computer behind, so if you want to talk to me or you want to try it with me, we can play with it together here. So that was my overview. I'm Eric Nicolas Smit. Thank you and I hope you enjoyed it.
Resume Auto-Scroll?