Cineversity

Hi everyone

Just getting to grips with CV-AR. The tutorial on Cineversity was fantastic, however I do have a query…
This primarily concerns using the ‘facial recording data’ to drive the animation of a separate 3D mesh.

I’ve watched this tutorial.
https://www.cineversity.com/vidplaylist/cv-ar/cv-ar_setup_capture_import_and_rig_an_animji-style_facial_performance
The tutorial makes perfect sense. It is essentially drawing a DIRECT link specifically between the ‘jaw open’ and ‘eyebrows up’ parameters in the CV-AR object, and matching them to an equivalent pose morph with the robot head.

However…

I am attempting to use the facial recording to drive a more complex animation. What I have is 12 ‘blend shapes’ of my 3D characters face, which are generally based on phonetic pronunciation poses (so poses for ‘A, I’... ‘C,D,G’.....‘M,B,P…. etc). Essentially I have multiple blend shapes of my 3D character that look like this…
https://i.pinimg.com/originals/8b/b2/f5/8bb2f58a7840589abf528ae329d3040a.jpg

So my question is… How can I use a facial recording from CV-AR to drive an accurate transition between my various blend shapes for my 3D character?

I completely understand that in the tutorial, there is a DIRECT link between the ‘jaw open’ parameter, and the blend shape he created for his robot character with its mouth open. The problem with what I want to achieve is… It will clearly take a mixture of MANY CV-AR parameters, at many varying percentages to achieve just ONE of my ‘phonetic pronunciation blend shapes’ for my 3D character (let alone transition between one state to the other). I’m kind of at a loss as to how I would achieve it.

Ultimately, what I really want to achieve…. Is the ability to import a CV-AR facial recording, and have it accurately drive between the 12 blend shapes I have for my 3D Characters face.

Thanks!

Hi, Hello Charlie,

I have moved this to the tutorial suggestion forum, as I believe that this needs more attention than a Q&A answer can do.

All the best

Hi Hello Charlie,

Yes I would also suggest looking into using Xpresso and mixing multiple blend shapes values together to drive your Pose Morph strength. Which you yourself also came to that conclusion. However in some cases you may find a perfect match, such as Mouth Pucker would be the Kiss expression.

It may be useful for you to look through the blend shapes in the official ARKit documentation. The following link has a list of all the blend shapes, clicking on each will take you to a page for that blend shape with an image that shows exactly what part of the face that affects.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation

For instance here is the Mouth Pucker I mentioned.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928257-mouthpucker

Mouth Funnel looks like a good fit for “O, U, W”

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928253-mouthfunnel

For smile, since its symmetrical, you could try using just the blend shapes for one side of the model and ignore the other.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928249-mouthsmileleft

“b,m,p” would be mouthRollLower

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928224-mouthrolllower

“c,d,n,r,s,t,x,z” could be a combination of MouthSmileLeft and Jaw Open

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928236-jawopen

‘Sad’ you might have to use 2 blend shape values to drive the strength of the pose morph. Likewise for “f,v”, I think you may need to find a couple and mix them to a single value also to drive the pose morph.

Ideally what I would like to do for a future release is to create a viseme and expression system for this case as well. But that will have to wait for another update sometime in the future. For now I would look at using Xpresso and the Math nodes to mix the blend shapes together manually.

I hope some of this is helpful to you. Would love to see any results that you come up with.

Best,
Kent

Kent Barber - 21 January 2019 07:27 AM

Hi Hello Charlie,

Yes I would also suggest looking into using Xpresso and mixing multiple blend shapes values together to drive your Pose Morph strength. Which you yourself also came to that conclusion. However in some cases you may find a perfect match, such as Mouth Pucker would be the Kiss expression.

It may be useful for you to look through the blend shapes in the official ARKit documentation. The following link has a list of all the blend shapes, clicking on each will take you to a page for that blend shape with an image that shows exactly what part of the face that affects.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation

For instance here is the Mouth Pucker I mentioned.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928257-mouthpucker

Mouth Funnel looks like a good fit for “O, U, W”

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928253-mouthfunnel

For smile, since its symmetrical, you could try using just the blend shapes for one side of the model and ignore the other.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928249-mouthsmileleft

“b,m,p” would be mouthRollLower

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928224-mouthrolllower

“c,d,n,r,s,t,x,z” could be a combination of MouthSmileLeft and Jaw Open

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928236-jawopen

‘Sad’ you might have to use 2 blend shape values to drive the strength of the pose morph. Likewise for “f,v”, I think you may need to find a couple and mix them to a single value also to drive the pose morph.

Ideally what I would like to do for a future release is to create a viseme and expression system for this case as well. But that will have to wait for another update sometime in the future. For now I would look at using Xpresso and the Math nodes to mix the blend shapes together manually.

I hope some of this is helpful to you. Would love to see any results that you come up with.

Best,
Kent

Hello Kent

I’ve been testing CV-AR for my new project for a while. And I’m running into a dead end now since there aren’t many tutorials about CV-AR online. I hope that you don’t mind me to ask some question about blend shapes.

I followed your advice to look up into the ARKit documentation from the link below.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation

One pose I really don’t know how to achieve which it mouth close.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928266-mouthclose#discussion

I tried so many times just couldn’t make it above.
I could make mouthclose = 0; jawopen =0 and mouthclose = 0; jawopen =1 but just couldn’t figure out how to get to the
mouthclose = 1; jawopen =1 pose. therefore now I can’t close my character’s mouth lol

You are the expert. Can you please give me a hint?

Hi Kent,

I’m glad I found this thread since I’m also considering joining CV-AR with a Mixamo import into C4D. I think it would be a home run if we could combine the body mocap of Mixamo with facial mocap from the iPhone!

Can you tell me when the next CV-AR update is coming and will it work with the new iPhone 11?

Thanks,
Matthew

Hi Matthew,

CV-AR should work fine on the new iPhone 11 with R20.

The R21 plugin is currently being worked on and will be out later this year. So for now you will still have to use R20 with CV-AR.

Also an update to the App is underway.

Look forward to seeing what you come up with!

Since you are getting the new iPhone 11, would you be interested in being on the beta testing team for the new version of the App?

Cheers,
Kent

Can’t wait till this is out for R21