3 of 10
3
CV-AR: Support
Posted: 24 August 2018 06:40 PM   [ Ignore ]   [ # 31 ]  
Avatar
Total Posts:  52
Joined  2009-02-16

The frequency is only an issue if your friends router has both enabled. Then on your phone you can choose to connect to the 2Ghz or the 5Ghz, just just shows up as a different name, based on the name your friend gave the wifi network. Hard to describe.

Unfortunately with these kind of issues it’s hard to diagnose the exact problem. Just to throw a wild idea out there into the open, you could try connecting via a different wifi system. Such as perhaps enabling hotspot on your phone and your computer and your friends phone both connect to that.

It sounds like you are doing every thing correctly.

At the moment there is no other way to transfer the data.

I have already contacted the folks at Cineversity to see if they can provide a capture that people can use to test out the plugin.

Again really sorry you are experiencing these issues. It is difficult to know what the exact problem would be since it may be something specific to your friends wifi setup.

Profile
 
 
Posted: 25 August 2018 08:42 AM   [ Ignore ]   [ # 32 ]  
Total Posts:  22
Joined  2018-08-20

Hi Kent,

Finally got it connected using another iPhone X. Exciting to look at the data!

A few questions:
- Is there a way to get a higher texture resolution from the phone? It’s currently 1280x720.
- There is a bit jittering in the mesh movement. Do you have any suggestions how to smooth/interpolate that? I’ve tried the delay effector on a baked mesh and it helps a bit but maybe there is a better way.
- The “add Camera” function helps a lot for what I want to achieve (which needs editable camera mapping). After some rotations, the texture maps quite well. Adjusting the FOV manually kind of fit’s too, but maybe you know the exact FOV. Can’t find it online.
- Do you know a good procedure to extend the face to a full head, maybe with a bit of a neck and shoulders while preserving the characteristic facial features?

Many thanks for any suggestions!

Profile
 
 
Posted: 25 August 2018 11:14 AM   [ Ignore ]   [ # 33 ]  
Avatar
Total Posts:  52
Joined  2009-02-16

Hi polypoint,

Glad to hear you got it going!

polypoint - 25 August 2018 08:42 AM

- Is there a way to get a higher texture resolution from the phone? It’s currently 1280x720.

Not at the moment.  But we can look into it for a future update. It is a compromise between quality, speed and storage space at the moment, since it captures an individual image per frame at 30fps.

polypoint - 25 August 2018 08:42 AM

- There is a bit jittering in the mesh movement. Do you have any suggestions how to smooth/interpolate that? I’ve tried the delay effector on a baked mesh and it helps a bit but maybe there is a better way.

There is no way to smooth it out in the current version. It is on the todo list however.

polypoint - 25 August 2018 08:42 AM

- The “add Camera” function helps a lot for what I want to achieve (which needs editable camera mapping). After some rotations, the texture maps quite well. Adjusting the FOV manually kind of fit’s too, but maybe you know the exact FOV. Can’t find it online.

The uvs are adjusted every frame so that they perfectly align to the captured textures taking into account the exact specs of the camera itself.  You should not need to rotate the textures or change any FOV settings at all. Could you let me know what it is you are trying to do that is different from what it gives you by default? My guess is you are not projecting the image each frame (ie you have disabled this checkbox) and they you are wanting to do it yourself manually?

polypoint - 25 August 2018 08:42 AM

- Do you know a good procedure to extend the face to a full head, maybe with a bit of a neck and shoulders while preserving the characteristic facial features?

As long as you do not delete any faces of the model itself, you should be able to extend it through the polygonal modelling tools to add additional detail. The plugin will still adjust the vertices and uvs of the model for the head and it will ignore any additional geometry. Making it easier to extend the model is something we would like to look into in the future. But the best method for now would really be to create your own model with poses, and drive that using the blendshape data with xpresso.

Cheers,
Kent

Profile
 
 
Posted: 25 August 2018 07:09 PM   [ Ignore ]   [ # 34 ]  
Total Posts:  22
Joined  2018-08-20

Thanks for your prompt reply Kent!

I am trying to create an animation where parts of the head preserve the realistic look of the video while other parts blending into CGI. In this regard, the texture camera mapping is quite helpful. For example, if the polygon holes at the eyes and mouth are filled, the camera UV mapping adds the correct texture on it.

From my first impression, the plugin is super cool and it has the capacity of being used for the purposes described above. But the most crucial thing is the texture size. 720p doesn’t provide enough detail. Glad to hear that this is not restricted by the phones camera. An option to allow the user to choose high resolution video instead of 720p would be an enormous feature! Btw, does the phone has the option to switch between 30 and 25 fps? This would save some space and would be also beneficial for PAL productions.

Maybe there is a smoothing function in the AR kit itself that can be used? However, I’m looking for such a feature in C4D for a long time. The delay effector only interpolates present and past values and thus always has latency and inaccurate viewport update when scrub the timeline. Concerning objects with keyframes such as blend shapes or PLAs, the interpolation could be also bidirectional (in theory), although it might only perform well with a rather small amount of values.

I’ll have a look at model creation for pose morphs and blend shapes, but not sure if it is possible to achieve a mesh that overlays exactly with the video. Maybe it’s good enough to just create a rough form of the head and neck to project the texture on it.

Cheers!

Profile
 
 
Posted: 25 August 2018 09:30 PM   [ Ignore ]   [ # 35 ]  
Avatar
Total Posts:  52
Joined  2009-02-16
polypoint - 25 August 2018 07:09 PM

From my first impression, the plugin is super cool and it has the capacity of being used for the purposes described above. But the most crucial thing is the texture size. 720p doesn’t provide enough detail. Glad to hear that this is not restricted by the phones camera. An option to allow the user to choose high resolution video instead of 720p would be an enormous feature! Btw, does the phone has the option to switch between 30 and 25 fps? This would save some space and would be also beneficial for PAL productions.

For the texture size the best thing I can recommend for now is to simply have the camera as close to the actors face as possible. There will be options later to adjust the frame rate, but the most requested feature is to actually increase the frame rate rather than decrease, so provide 60fps. This will help reduce jitter and provide a smoother capture.

polypoint - 25 August 2018 07:09 PM

Maybe there is a smoothing function in the AR kit itself that can be used? However, I’m looking for such a feature in C4D for a long time. The delay effector only interpolates present and past values and thus always has latency and inaccurate viewport update when scrub the timeline. Concerning objects with keyframes such as blend shapes or PLAs, the interpolation could be also bidirectional (in theory), although it might only perform well with a rather small amount of values.

No there is no smoothing in AR Kit. It will be done in the plugin as a post operation on the sampled data. There are plans for this, it will just take some time to get to it. Also better support for subframe interpolation is required, for instance if you want to slow down the animation.

Profile
 
 
Posted: 29 August 2018 05:41 AM   [ Ignore ]   [ # 36 ]  
Total Posts:  22
Joined  2018-08-20
Kent Barber - 25 August 2018 09:30 PM

For the texture size the best thing I can recommend for now is to simply have the camera as close to the actors face as possible.

Yes, getting closer to the camera will increase the texture detail, but also reduces the movement space and the visible texture on the sides of the face. I’m wondering if it would be possible to capture the video outside of the app in higher resolution. For example, if the wire overly is disabled and the displayed video is higher than 720p, the iPhones screen recording could maybe capture the video parallel.

Kent Barber - 25 August 2018 09:30 PM

There will be options later to adjust the frame rate, but the most requested feature is to actually increase the frame rate rather than decrease, so provide 60fps. This will help reduce jitter and provide a smoother capture.

Extending 25 to 50 or 30 to 60 would be great. But converting 30 to 25, etc. costs always a bit of smoothness.

Could you explain how to use the plugin with Team Render and different computers?  When I open the C4D scene in which the data from the iPhone has been stored, the texture links are broken and I could not find a way to relink them. That’s currently not a big problem because it seems that adding a new material with the same texture overrides the texture tag.
However, when using Team Render, I get an “Asset Error” window with “(CV-AR Face Shader)” listed. Team Render fails and the console says: “(Error) Render-Job failed: Several plugins used in this project are missing. Saving may cause loss of plugin data! Plugin ‘CV-AR Capture Object’ (ID 1040464) Plugin ‘CV-AR Face Shader’ (ID 1040942)”.

Profile
 
 
Posted: 29 August 2018 01:37 PM   [ Ignore ]   [ # 37 ]  
Avatar
Total Posts:  52
Joined  2009-02-16

The plugin needs to be installed on the Team Render clients as well. All the captured textures are uniquely named. They are loaded in and applied to the material using the CV-AR Face Shader, which is automatically added to the capture when you import it into your scene.

If you are setting up your materials manually then you will need to also manually add a CV-AR Face Shader to the texture slot so that it gets the appropriate textures. Or you can choose a specific texture and lock it to that texture by turning off the continuous projection option.

If you are manually trying to copy the scene and send it to another computer you should choose “File->Save Project With Assets”. But if you are doing this through Team Render then it should be working already.

If it does not work with Team Render and you have installed the plugin on all your clients then perhaps something else has happened.

Unfortunately I am unable to test Team Render here since I do not have it set up here. But maybe someone else reading this thread can do so.

Profile
 
 
Posted: 03 September 2018 01:55 AM   [ Ignore ]   [ # 38 ]  
Total Posts:  22
Joined  2018-08-20

The plugin is installed on the Team Render client and works correctly when C4D Studio runs in stand allone on this machine. All other plugins I am using in this team render configuration working without problems. Maybe you contact Maxon to solve the problem. Plugins should be normally compatible with Team Render.

Profile
 
 
Posted: 06 September 2018 04:45 PM   [ Ignore ]   [ # 39 ]  
Total Posts:  22
Joined  2018-08-20

Hi Kent,

Have you fixed the Team Render problem? I am planning to use the plugin in a few days for a production that involves Team rendering. Can send a support request to Maxon parallel. Usually, Maxon does not give support on third party plugins, but as this platform belongs to the company, they might be responsible.

Is there a manual that explains the different functions of the plugin in depth? Such as the manual assignment of textures, etc.

Profile
 
 
Posted: 07 September 2018 12:39 AM   [ Ignore ]   [ # 40 ]  
Total Posts:  1
Joined  2012-05-01

Hi, thank you for making this great plugin. I’m currently working on apply facial capture to anime characters (Yes I’m Japanese)
https://drive.google.com/open?id=1we8TEAXvfft6D40jcXxrqRXlv9TBSV1V

And I have noticed iOS 12 have some additional parameters for ARface’s blendshapes
https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation

Possibly because of this, CV-AR plugin doesn’t work with captured data by iOS 12 devices.
CV-AR object is created normally but there’s no animations and mesh.

Are you going to support iOS12? (It might be released soon)

Profile
 
 
Posted: 11 September 2018 05:25 AM   [ Ignore ]   [ # 41 ]  
Total Posts:  1
Joined  2018-09-03

Some wish-list items:
1. Its been mentioned here already, but some form of wired data transfer would be desirable as wifi is not always accessible and/or fast enough
2. Recorded data filtering / smoothing option would be great
3. Option to not record jpeg texture stream to help keep data size down
4. I’m sure there is a legitimate reason to stream / load the data via a custom object in C4D, but it would be great to have the option to import the data into the scene using a conventional pose-morph tag and key-frames

Great plugin tho! Works like a charm.

Profile
 
 
Posted: 16 September 2018 09:20 PM   [ Ignore ]   [ # 42 ]  
Total Posts:  1
Joined  2018-09-15

I wanted to use the CV - AR plugin and went into platinum membership, but I could not find the CV - AR plug - in in the CV - Tool box, but I will make it messed up quickly.

Profile
 
 
Posted: 21 September 2018 11:02 AM   [ Ignore ]   [ # 43 ]  
Total Posts:  4
Joined  2017-08-24

FYI iPhone XS is now shipping and seems to not work with CV-AR (obviously because you haven’t had the chance to even test on one, so that’s to be expected).

Appears the video is black and the plugin thinks there’s one frame (perhaps because no video track in the mp4) so probably an app issue.

Here’s my downloaded capture
https://www.dropbox.com/s/mvoexigiwovogr9/C31D571D-3C45-4EE4-8EBA-A831D6C93F5A.zip?dl=0

Profile
 
 
Posted: 21 September 2018 11:15 AM   [ Ignore ]   [ # 44 ]  
Total Posts:  4
Joined  2017-08-24

Tested on R19 and R20 with no difference.
Also the video playback on the iPhone shows all black

Profile
 
 
Posted: 21 September 2018 11:04 PM   [ Ignore ]   [ # 45 ]  
Total Posts:  3
Joined  2018-09-03

I love this plugin! I signed up for Cinversity specifically for this. Super excited to see where it goes!

But I am having an issue where it only *once* successfully created a face mesh with the data + geometry from iphone (the first time using the plugin). Since then it is only able to insert blank face meshes no matter how many times I try.

I have seen a few other posts here with similar issues (only inserting a blank face mesh) Any ideas or pointers? Attaching an image that should show what I’m talking about - you can see only the first one worked. Thank you!

Max

Profile
 
 
   
3 of 10
3