CV-AR: Transfer ’’Facial Recording’’ data to a string of Blend shapes
Posted: 21 January 2019 05:12 AM   [ Ignore ]  
Total Posts:  2
Joined  2019-01-08

Hi everyone

Just getting to grips with CV-AR. The tutorial on Cineversity was fantastic, however I do have a query…
This primarily concerns using the ‘facial recording data’ to drive the animation of a separate 3D mesh.

I’ve watched this tutorial.
https://www.cineversity.com/vidplaylist/cv-ar/cv-ar_setup_capture_import_and_rig_an_animji-style_facial_performance
The tutorial makes perfect sense. It is essentially drawing a DIRECT link specifically between the ‘jaw open’ and ‘eyebrows up’ parameters in the CV-AR object, and matching them to an equivalent pose morph with the robot head.

However…

I am attempting to use the facial recording to drive a more complex animation. What I have is 12 ‘blend shapes’ of my 3D characters face, which are generally based on phonetic pronunciation poses (so poses for ‘A, I’... ‘C,D,G’.....‘M,B,P…. etc). Essentially I have multiple blend shapes of my 3D character that look like this…
https://i.pinimg.com/originals/8b/b2/f5/8bb2f58a7840589abf528ae329d3040a.jpg

So my question is… How can I use a facial recording from CV-AR to drive an accurate transition between my various blend shapes for my 3D character?

I completely understand that in the tutorial, there is a DIRECT link between the ‘jaw open’ parameter, and the blend shape he created for his robot character with its mouth open. The problem with what I want to achieve is… It will clearly take a mixture of MANY CV-AR parameters, at many varying percentages to achieve just ONE of my ‘phonetic pronunciation blend shapes’ for my 3D character (let alone transition between one state to the other). I’m kind of at a loss as to how I would achieve it.

Ultimately, what I really want to achieve…. Is the ability to import a CV-AR facial recording, and have it accurately drive between the 12 blend shapes I have for my 3D Characters face.

Thanks!

Profile
 
 
Posted: 21 January 2019 09:58 AM   [ Ignore ]   [ # 1 ]  
Moderator
Avatar
Total Posts:  6617
Joined  2011-03-04

Hi, Hello Charlie,

I have moved this to the tutorial suggestion forum, as I believe that this needs more attention than a Q&A answer can do.

All the best

 Signature 

Dr. Sassi V. Sassmannshausen Ph.D.

Photography For C4D Artists: 200 Free Tutorials: Texture, Panorama, HDRI, Camera Projection, etc.
https://www.youtube.com/user/DrSassiLA/playlists

Profile
 
 
Posted: 21 January 2019 03:27 PM   [ Ignore ]   [ # 2 ]  
Avatar
Total Posts:  48
Joined  2009-02-17

Hi Hello Charlie,

Yes I would also suggest looking into using Xpresso and mixing multiple blend shapes values together to drive your Pose Morph strength. Which you yourself also came to that conclusion. However in some cases you may find a perfect match, such as Mouth Pucker would be the Kiss expression.

It may be useful for you to look through the blend shapes in the official ARKit documentation. The following link has a list of all the blend shapes, clicking on each will take you to a page for that blend shape with an image that shows exactly what part of the face that affects.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation

For instance here is the Mouth Pucker I mentioned.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928257-mouthpucker

Mouth Funnel looks like a good fit for “O, U, W”

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928253-mouthfunnel

For smile, since its symmetrical, you could try using just the blend shapes for one side of the model and ignore the other.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928249-mouthsmileleft

“b,m,p” would be mouthRollLower

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928224-mouthrolllower

“c,d,n,r,s,t,x,z” could be a combination of MouthSmileLeft and Jaw Open

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928236-jawopen

‘Sad’ you might have to use 2 blend shape values to drive the strength of the pose morph. Likewise for “f,v”, I think you may need to find a couple and mix them to a single value also to drive the pose morph.

Ideally what I would like to do for a future release is to create a viseme and expression system for this case as well. But that will have to wait for another update sometime in the future. For now I would look at using Xpresso and the Math nodes to mix the blend shapes together manually.

I hope some of this is helpful to you. Would love to see any results that you come up with.

Best,
Kent

Profile
 
 
Posted: 29 May 2019 01:04 AM   [ Ignore ]   [ # 3 ]  
Total Posts:  1
Joined  2019-04-17
Kent Barber - 21 January 2019 03:27 PM

Hi Hello Charlie,

Yes I would also suggest looking into using Xpresso and mixing multiple blend shapes values together to drive your Pose Morph strength. Which you yourself also came to that conclusion. However in some cases you may find a perfect match, such as Mouth Pucker would be the Kiss expression.

It may be useful for you to look through the blend shapes in the official ARKit documentation. The following link has a list of all the blend shapes, clicking on each will take you to a page for that blend shape with an image that shows exactly what part of the face that affects.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation

For instance here is the Mouth Pucker I mentioned.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928257-mouthpucker

Mouth Funnel looks like a good fit for “O, U, W”

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928253-mouthfunnel

For smile, since its symmetrical, you could try using just the blend shapes for one side of the model and ignore the other.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928249-mouthsmileleft

“b,m,p” would be mouthRollLower

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928224-mouthrolllower

“c,d,n,r,s,t,x,z” could be a combination of MouthSmileLeft and Jaw Open

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928236-jawopen

‘Sad’ you might have to use 2 blend shape values to drive the strength of the pose morph. Likewise for “f,v”, I think you may need to find a couple and mix them to a single value also to drive the pose morph.

Ideally what I would like to do for a future release is to create a viseme and expression system for this case as well. But that will have to wait for another update sometime in the future. For now I would look at using Xpresso and the Math nodes to mix the blend shapes together manually.

I hope some of this is helpful to you. Would love to see any results that you come up with.

Best,
Kent

Hello Kent

I’ve been testing CV-AR for my new project for a while. And I’m running into a dead end now since there aren’t many tutorials about CV-AR online. I hope that you don’t mind me to ask some question about blend shapes.

I followed your advice to look up into the ARKit documentation from the link below.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation

One pose I really don’t know how to achieve which it mouth close.

https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation/2928266-mouthclose#discussion

I tried so many times just couldn’t make it above.
I could make mouthclose = 0; jawopen =0 and mouthclose = 0; jawopen =1 but just couldn’t figure out how to get to the
mouthclose = 1; jawopen =1 pose. therefore now I can’t close my character’s mouth lol

You are the expert. Can you please give me a hint?

Profile