Motion Tracker - Camera Settings don’t line up. - How to deal with Sony ZV-1, iphones, none full frame cameras
Posted: 10 October 2021 07:02 AM   [ Ignore ]  
Total Posts:  5
Joined  2021-07-26

Hello Everyone,

for the past three days I have desperately searched and tried to find the correct camera settings for none full frame cameras within cinema 4D Motion Tracker in the 3D solver.

All the tutorials are basically just winging that part, have full frame cameras or just don’t care.

Below is a Sony-zv-1 setup from which I want to set up the correct camera (sensor position) for VFX. Neither in the Maxon Documentation nor online could I find a proper explanation how this is calculated or what the strategy is to set it up — what the old school terms like “film gate” mean for cameras like the camera below and smartphones — making it rather complex in this messed up world of none full frame “but 35mm format equivalent” specifications 😭

I would love to know how to precisely set this up to eventually use cinema 4D for the generation of training data for machine learning models. I.e.: I want to use cinema 4D for rendering training images for machine learning systems. There it is important that perspective is actually “correct” or rather it’s clear how to set this up for other cameras.

I’m very thankful for any ideas, tips and links. 😮‍💨

#cinema4d #motiontracking #maxon #SonyZV1

Image Attachments
asking_for_help_cinema_4d_motion_tracker_camera_settings_sony_zv-1small.pngScreenshot 2021-10-10 at 17.05.19.png
Profile
 
 
Posted: 10 October 2021 08:14 AM   [ Ignore ]   [ # 1 ]  
Moderator
Avatar
Total Posts:  10722
Joined  2011-03-03

Hi ds,

To know where a static camera is, use the Camera Calibrator, always with a lens distortion grid.

There is no machine learning in this application.

Lenses have a certain focal length, which will not change based on the sensor size. (When you read Crop factor of a lens, move on, someone has no idea what s/he talking about)
The lens is a piece of glass that can’t change, except you focus or zoom. Sensors have a specific size; how much of that size tales advantage of the “projection” that the lens provides is critical, as it defines the field of view of that combination. That is all there is to it. What this means for 35mm equivalent, that it is a nice piece of information, not relevant nor essential; if you don’t have the data, the indicator is irrelevant. What matters is that the larger the Camera is, with its sensor size or lens-mount, the smaller the problems are—for example, the center of the lens axis to the sensor mid-point. To a certain degree, that is covered with a lens grid. Small cameras with zoom lenses are often very loose to allow for fast focusing, and there the precision goes out of the window.

Find your sensor position in the Camera; typically, the Φ indicates this. Then measure from there a certain distance and how far you can see vertical and horizontal in the image. This gives you all you need, while it doesn’t compensate for any lens distortion.

With that data, you can set up the Camera. Two parameters are enough, type the data in and done.

This is then true for that distance. Still cameras, vs. feature film lenses/cameras, typically breath. This is based on the construction; the focal length changes as well if the focus point changes. Combine that with the none accurate zoom that pocket cameras have, and you get already an idea, why those tiny cameras are not precision tools. Even the optic in an iPhone is a little wonder all by itself.

Motion Tracking works by triangulation of camera and feature points, and pretty much how fast they move. If the lens has a distortion, that movement is not even, and hence the problems start, as different speeds in movement distros the triangulation.
Tim Dobbert has written a great book about Motion Tracking. Old by now, but relevant.

All the best

 Signature 

Dr. Sassi V. Sassmannshausen Ph.D.
Cinema 4D Mentor since 2004

Photography For C4D Artists: 200 Free Tutorials. Texture, Panorama, HDRI, Camera Projection, etc.
https://www.youtube.com/user/DrSassiLA/playlists

Profile
 
 
Posted: 10 October 2021 09:04 AM   [ Ignore ]   [ # 2 ]  
Total Posts:  5
Joined  2021-07-26

Thank you Dr. Sassi,

apologies, I think you miss understood me. From beginning: I am a machine learning engineer / researcher and want to use Cinema 4D to render a bunch of training images for our (my) machine learning models. The machine learning system is connected to the Sony ZV-1 via HDMI and learns through us placing objects on the table as seen in the picture attachments above.

Back Story
Now, I want to place virtual objects on the table via Cinema 4D. I recorded footage usually the machine learning system would see, and use it in the Motion Tracking utility to find the camera position and place objects via C4D on the table. It’s basically VFX — the camera movement above the table you see with the grid is recorded, and populated with 3D objects in 1:1 size to real life. This way I can generate additional training data — yes the camera movements will be the same between takes — but on those many different objects can be observed by our machine learning model by just feeding videos to it.

I am aware that C4D doesn’t use ML for motion tracking (at least not immediately) but triangulates points to the sensor. Now to my problem:

The Problem: Ambiguous Camera Settings
I am not a camera / sensor expert and the motion tracking tool in C4D asks for three values for the camera:
- Focal Length
- Sensor Size (Film Gate)
- Field of View (Horizontal)
- Field of View (Vertical)

Input Footage:
- No crop footage
- In Portrait (!) so H and V is flipped
- Taken with the Sony ZV-1 “at 24mm” wide no crop due to stabilization
Here the sensor specification of the Sony ZV-1
(The Sony ZV-1 has minimal lens distortion even at the wide angle, though it looks like it due to the tilt in the attachment above. Anyway, creating a lens profile is easy.)

Input: Sony ZV-1 Camera Specification?
Here are connected
- Focal Length <- Sony ZV-1: Which one? The 24mm or the one of the sensor (see screenshot from first post)
and
- Sensor Size (Film Gate) <- (Sony ZV-1:  What is the film gate for the Sony ZV-1 ?)

Automatically Calculated in Motion Tracker:
… than the 35mm Equivalent Focal Length (again?) = that should be the 24mm

The specification says, the Sony ZV-1 has a field of view of 84 degrees.
- Field of View (Vertical; because Portrait footage) = 84 Degrees

Help
I am not able to figure out what the correct combination of inputs are to all aligned with the Sony ZV-1 specification.

The lens profile would be the next step.

I hope this makes it more clear. Thank you
Dino Scheidt

Profile
 
 
Posted: 10 October 2021 09:49 AM   [ Ignore ]   [ # 3 ]  
Moderator
Avatar
Total Posts:  10722
Joined  2011-03-03

Hi Dino Scheidt,

Let me start with this: The data given from the manufacturer should be used with caution anyway, as nothing is more off than lens data and sensor sizes in datasheets. To use a pocket camera and expect precision is wishful thinking. This is known to any developer or tracking software, AFAIK, so the data used is a source of error limitation. Nothing more. It will not constrain the outcome.

Minimal lens distortion is lens distortion. What is minimal, that is relative.

Does the focal length data change while focusing near to far? Did you notice breathing?

I have never had a lens in my life that matched the given focal length provided—going by 50 lenses I use for my art, with various cameras from GoPro to over Canon 5ds, Red Epic Dragon: Any idea why the Sony ZV-1 should be any different. Even I certainly enjoy Zeiss glass since many decades. But it is certainly not a Master Prime 24mm.
https://www.bhphotovideo.com/c/product/1287767-REG/arri_k2_47603_0_master_prime_25mm_t1_3.html

Think of a fixed positioned magnification glass; will it change its magnification when you look at a large stamp and then a smaller stamp? Of course not.
Hence why the focal lengths will not be sufficient to solve the data; how much from the projection of the lens was used is expressed in the sensor size, which is more often than not given in a wrong or misleading way. Is the chip size measured or the actual photosite area? There are often more photosites than actually used for the final image, based on the need for image stabilization. Often video cameras are defined in 2.3"rd chips, for example, which measured a surrounding tube, but not the sensor size. Some companies even today use this misleading option to provide larger-looking data than it is.

The 35mm provided data can’t be changed and is for information purposes only. It helps to identify problems.

Use the two numbers you have and let the tracking software do the rest; it will come to a different conclusion of the given data, I’m sure about it.

I’m not sure if they do that for those set ups, but Duclos could measure your glass and that would give you one fixed point in the equation:
https://www.ducloslenses.com/collections/test-equipment

My best wishes for your exploration.

 Signature 

Dr. Sassi V. Sassmannshausen Ph.D.
Cinema 4D Mentor since 2004

Photography For C4D Artists: 200 Free Tutorials. Texture, Panorama, HDRI, Camera Projection, etc.
https://www.youtube.com/user/DrSassiLA/playlists

Profile
 
 
Posted: 10 October 2021 10:17 AM   [ Ignore ]   [ # 4 ]  
Total Posts:  5
Joined  2021-07-26

Thanks Dr. Sassi,

though this still does not explain the weird calculation behavior in Cinema 4D Motion Tracking, I’ve moved to Pixel Farm PFTrack now which has more control and can import into houdini / also C4D.

As said, we have not got to the lens distortion part yet. But the lack of guidance in the Camera Settings within Cinema 4D Motion Tracking. And “for information purpose” is not enough to do proper match moving.

Tangent

The manufacturer specification are also correct, as almost all modern cameras are otherwise modern systems like digital vanishing point autofocus wouldn’t work which are individually factory calibrated anyway. Heck, smartphone cameras are so well calibrated (mostly using the same techniques and sensor technologies) that augmented reality is possible. No need for “this is not exactly XYZ glass” - we’re not in the analog days anymore.

Story short:
Sony’s page spec on the ZEISS Vario-Sonnar T* 9.4-25.7mm f/1.8-2.8 lens was correct. In PFTtrack those numbers lead to a perfect 1:1 representation match of a 10x10cm cube between virtual and real life.

Conclusion:
After spending three days with this, the conclusion is that the Motion Tracking in Cinema 4D is not suited for anything more than eyeballing effects / massive lack of guidance and I would not recommend it for precise match moving requirements.

And I thought I was crazy.

Thank you for your quick reply. Anybody looking how to do Matchmoving in Cinema 4D with cameras beyond standard full frame (have not tested that precision), should find that useful.

All the Best
Dino Scheidt

Profile
 
 
Posted: 10 October 2021 10:39 AM   [ Ignore ]   [ # 5 ]  
Moderator
Avatar
Total Posts:  10722
Joined  2011-03-03

Hi Dino Scheidt,

I used it in professional productions, and it is super stable. I haven’t used any of the specialized packages in years. Even I have a license to one. Any camera data can be typed in, again, the 35 mm info is just a reference.

If you don’t get your results, please contact the support and perhaps create a suggestion.
https://www.maxon.net/en/support-center

You seriously compare a $199/month tracking only solution with a package that provides this as one tool besides over a thousand other tools and functions, while the monthly asking price for Cinema 4D is just a little bit over a quarter of PFTrack. (?) While telling me, you got precise results with others while not implementing any action against lens distortion.

Good luck with your project.

 Signature 

Dr. Sassi V. Sassmannshausen Ph.D.
Cinema 4D Mentor since 2004

Photography For C4D Artists: 200 Free Tutorials. Texture, Panorama, HDRI, Camera Projection, etc.
https://www.youtube.com/user/DrSassiLA/playlists

Profile
 
 
Posted: 10 October 2021 12:05 PM   [ Ignore ]   [ # 6 ]  
Total Posts:  5
Joined  2021-07-26

With all due respect Dr. Sassi,

A] I stated before, that we do have a lens profile and it is not the focus of this thread. This thread concerns the lag of documentation of entering the correct variables into the sparse and poorly documented “camera settings” tab under 3D Solve within Cinema 4D Motion Tracker. And what that thing calculates. So forget lens profiles. Answer this:  What numbers do you put in for a Smartphone like the iPhone 13 Pro main sensor? Which information do you put into the Camera Settings?

You have four fields to populate:
- Focal Length
- Sensor Size (Film Gate)
- Field of View (Horizontal)
- Field of View (Vertical)

The 35mm Equivalent Focal length should be: 26mm
The Main iPhone Sensor is a Sony IMX703, 1.9 μm, 26 mm
Sensor Area is: 44mm² (1/1.65”) - 6.63324958071mm x 6.63324958071mm
CMOS calibrated, Raw, no crop, no distortion in 1px to 250MP

The virtual camera should be at the same position as the real one, and the virtual camera reflect the same focal length to be used in match moving.

B] Yes! Apparently a 5 person company from the UK is able to provide this PFTrack tool that takes tracking serious. And yes, that isn’t fair. Not only do they just have 5 employees (compared to the 209 listed on linkedin at maxon), trade at the stock exchange, but also don’t boast broad marketing campaigns around their strong tracking capabilities and special purpose for VFX.

C] I have the feeling you didn’t understand the problem here. It has nothing to do with glass or lens profiles if there is no documentation how to properly set up the “state of the art” motion tracking system by maxon. Apparently you work at Maxon / Cineversity so I hope you have enough information now to forward this to the right people so maxon can life back up to the standards of it old days before its acquisition.

At least, if a 5 person team can do simply triangulation right, maybe it’s time to spend less on marketing and more on your software engineers. Though trying TFTrack first, now I wouldn’t be surprised if tracking in blender would outperform this here as well since this is a track in front of a grid.

I’ll update this post once I found a definitive solution to what’s wrong here (if this forum / ancient Buletin isn’t crashing like in the last hour.

Thanks, but this should have been clear enough now.
Best
Dino Scheidt

Image Attachments
Screenshot 2021-10-10 at 22.07.57.png
Profile
 
 
Posted: 10 October 2021 12:20 PM   [ Ignore ]   [ # 7 ]  
Moderator
Avatar
Total Posts:  10722
Joined  2011-03-03

Hi Dino Scheidt,

You can place the sensor size and the lens in these fields; it is a parametrical field. As mentioned many times, the 35mm is just an orientation, ignore it.

Please get in touch with the support for any further questions.

PFTrack was over 20 years ago a University-based project, and back then, it was a free version.
With fairness, I meant to have $199 per month to pay for a single-purpose application and compare it to a completely different application. I have no idea why that is even a theme.

I will stop here, seriously, as I have nothing more to add.

Good luck with your project.

 Signature 

Dr. Sassi V. Sassmannshausen Ph.D.
Cinema 4D Mentor since 2004

Photography For C4D Artists: 200 Free Tutorials. Texture, Panorama, HDRI, Camera Projection, etc.
https://www.youtube.com/user/DrSassiLA/playlists

Profile
 
 
Posted: 10 October 2021 12:40 PM   [ Ignore ]   [ # 8 ]  
Total Posts:  5
Joined  2021-07-26

It’s easy: My time of three days is worth way more than that price. And it seems also blender does it correctly. So no. I’m usually a houdini user — using a research license, evaluating maxons tracking capabilities for our next research project for our students; so documenting it here should be more than enough in contribution here in the official forums to the maxon staff; if they have any more questions, just contact me. Good night

Profile
 
 
Posted: 10 October 2021 12:43 PM   [ Ignore ]   [ # 9 ]  
Moderator
Avatar
Total Posts:  10722
Joined  2011-03-03

Good Night to you as well.

 Signature 

Dr. Sassi V. Sassmannshausen Ph.D.
Cinema 4D Mentor since 2004

Photography For C4D Artists: 200 Free Tutorials. Texture, Panorama, HDRI, Camera Projection, etc.
https://www.youtube.com/user/DrSassiLA/playlists

Profile