Expressions to Blendshapes

If you have problems understanding something or getting started, ask here

Moderator: joepal

Expressions to Blendshapes

Postby human_maker_77 » Wed May 05, 2021 3:50 am

Hello,
Does anyone know how I can get expressions in MakeHuman as blendshapes in Blender? It seems like this should be done using the MHX2 importer plugin, but I am struggling to find any up to date documentation on how to use it for this (the original author seems to have abandoned it). For example, I found this tutorial (https://thomasmakehuman.wordpress.com/m ... tup-panel/), but the version it describes does not seem to match what I am currently using (.30). Any help would be greatly appreciated.
human_maker_77
 
Posts: 4
Joined: Wed May 05, 2021 3:42 am

Re: Expressions to Blendshapes

Postby nomorecookies » Thu May 06, 2021 6:34 pm

It is not clear what problem your are having, but, make sure you have the MHX2 plugin version from the download page here, not the original version.
There may be other methods, but this is what i tested after reading your question. It is not something i normally do.

Take the base model that you want to transfer the expressions to into blender.
Now, with the same style model in MH, go to the pose tab in MH and pick an expression. Export as MHX2 making sure to checkmark the expressions option. Import into blender and see that your expression is there, on the second model.
Click on the second model, then shift click on the first model. Go to the shape keys section. Click the down carrot below the plus and minus, above the arrows. Select the 'Join as Shapes' option. Now, your original model has had the expression transfered to it as a shape key. Repeat for all expressions that you want.

This is just one way to do that. there are probably others, perhaps better methods.

Good luck
nomorecookies
 
Posts: 82
Joined: Thu Jul 30, 2020 1:34 pm

Re: Expressions to Blendshapes

Postby human_maker_77 » Thu May 06, 2021 11:19 pm

Hey thanks for the reply. Maybe I should have clarified that I am trying get the 52 Apple AR facial blendshapes (https://developer.apple.com/documentati ... pelocation) for mocap into blender. So loading the expressions one by one really is not viable (especially if that would need to be done for each character). I was hoping there was a way to export and MHX2 file from MakeHuman with a large list of expressions included, and then have that list show up as blendshapes in Blender?
human_maker_77
 
Posts: 4
Joined: Wed May 05, 2021 3:42 am

Re: Expressions to Blendshapes

Postby nomorecookies » Fri May 07, 2021 7:53 am

Well, the 52 expressions would have to exist as shape keys first, or as a map within mhx2 importer, such as the limited things that are there now.
Since they do not exist, you would have to create them yourself, then make them into importable shapekeys, then add to or replace the faceshapes.mxa file included with the mhx2 importer.
as i said, the MHX2 importer does include some limited expressions that can be imported by clicking 'Overide Exported Data' then the 'face shapes' option when importing your model to blender, but these will not match the apple ar structure. At this point you could then use the shape keys you now have available and combine any muber of them to create a shape key that matches the apple ar structure. Then you could save these in a particular fashion that would allow you to reuse them later. I will not go in to how to do that right now, as that is rather involved and I would not wish to explain that process unless you have decided that you want to invest the considerable time to do that.

Basically, unless someone else has already added a compatible set of shape keys/expressions to match the apple format, they do not come included. You would have to make them yourself. But, if you do take the time to do that, i could explain to you how to reuse them so you could apply them to any of your models as long as you use the MHX2 importer/exporter.
nomorecookies
 
Posts: 82
Joined: Thu Jul 30, 2020 1:34 pm

Re: Expressions to Blendshapes

Postby human_maker_77 » Fri May 07, 2021 6:48 pm

Got it. I actually dug around a bit in the MakeHuman directory before this and found the expression files. Opening them up in text editor, they seem pretty straightforward, and I do not think it would take too long to make custom files for each of the 52 AR shapes (maybe I am missing something?). It seems like the basic features AR uses are already there. I already made the jawOpen expression as a test and it looks fine when viewed in MakeHuman. The effort would be totally worth it if I could easily load this library of expressions as blendshapes in Blender for any model, but that is what I am currently missing.

I tried loading faceshapes.mxa file you mentioned with MHX2 importer, but got an error KeyError: Proxy, so I cannot say if this get me in the right direction.
human_maker_77
 
Posts: 4
Joined: Wed May 05, 2021 3:42 am

Re: Expressions to Blendshapes

Postby nomorecookies » Sat May 08, 2021 5:48 am

human_maker_77 wrote:I tried loading faceshapes.mxa file you mentioned with MHX2 importer, but got an error KeyError: Proxy, so I cannot say if this get me in the right direction.


Well you are definately getting closer, and it sounds like you have the general idea when it comes to actually making shape keys.
The faceshapes.mxa is the actual file that the information comes from when you check the "Face Shapes" option when importing an MHX2 file into blender. The format of the data is nearly identical to the actual data that is the shape key. So once you have made a shape key, the trick is reusing it. There are two readily available ways that come to mind to do this. 1. Actually convert the shape key data to the format that is used within the faceshape.mxa file and add it to the faceshapes.mxa file so it loads in onto every character that you use mhx2 workflow with, or 2. use the MakeTarget/MakeShapes plugin to load them when you want onto your model after the model is already in blender. I will get back to how to do either option in a minute. My preference is to 'convert' the shapekey data into the faceshapes compatible data. I prefer this method because when you import using this method, the shape keys propagate. For example, if you make a MouthOpen shape keys, and you do it properly, which means making sure the helper geometry is affected, if will not only move the body model when you open the mouth, but will also open the actual teeth, and move the tongue into proper position. If, however, you decide to use the MakeTarget/MakeShapes route, once you load in a saved shape key, it will only affect the body model. There is a fancy trick to get around this, but it multiplies the steps that need to be done, everytime you bring in a shape key. I use the MakeTarget/MakeShapes all the time and they are great tools that i would not have been able to do what i have done without. But, they do not seem the correct fit for what you are doing. So, again, my prefered method is to convert the shape key data into the .mxa format. The process for converting shapekeys to the .mxa format is lengthy, but easy once understood.
I will continue this message in a few minutes with information on how to make the shapekey data work in the .mxa file
nomorecookies
 
Posts: 82
Joined: Thu Jul 30, 2020 1:34 pm

Re: Expressions to Blendshapes

Postby nomorecookies » Sat May 08, 2021 7:09 am

Continuation of my last message:

So a .target file is the index of changes that are made to the vertices when you use the slider for the shape key that is the .target data.
This can be imported directly onto your model in blender by use of MakeTarget/MakeShapes plugins. Or it can be used directly in MakeHuman. The information in these .target files is the information that is read when you move a slider within MakeHuman. As i said earlier, although you can directly apply these targets/shapkeys to you model in blender, they will only move the 'Body' mesh. not the teeth, eyebrows, tongue, etc.
So what we will focus on is getting a shape key that you have made into the .mxa file for reusability, and for the added function the MHX2 workflow supplies, which is applying the shape key data to the teeth, eyebrows, tongue, etc. as well as the Body mesh.
I will assume your are using windows. If not, just adapt the following procedure to your operatioing system.
You will need to have the MakeTarget plugin installed.
Open blender with a pipe to a text file output. In windows, this simply means open a command prompt in the folder that the blender.exe file is.
Then, run blender with a pipe.
Code: Select all
blender > AFileName.txt
This will allow us to use the MakeTarget plugin to Print our shape key values to the console, which is now being captured as a text file of the name we specified. At this point, we will go one by one and click on our shape keys in blender, in the list of the shape keys, and use the MakeTarget plugin option to 'Print shape keys'. You will notice an option to 'save your faceshape.mxa file' This is not really functioning right now, but would have been awesome if it was, as it would have saved us many steps. Since that is not working, i am telling you how to get around all of that. At this point, you should have printed all of your shape keys to the console. close blender. Open the AFileName.txt
Code: Select all
******* Here we go... << Eye catcher
     "NamedShapeKeyWeCreated" : [ [0, [-17.1579833984375,-17.1579833984375,17.1579833984375 ]] , [1, [-17.1579833984375,17.1579833984375,17.1579833984375 ]] , [2, [-17.1579833984375,-17.1579833984375,-17.1579833984375 ]] , [3, [-17.1579833984375,17.1579833984375,-17.1579833984375 ]] , [4, [17.1579833984375,-17.1579833984375,17.1579833984375 ]] , [5, [17.1579833984375,17.1579833984375,17.1579833984375 ]] , [6, [17.1579833984375,-17.1579833984375,-17.1579833984375 ]] , [7, [17.1579833984375,17.1579833984375,-17.1579833984375 ]]  ]
Selected Shape Key 'Key 1' printed to console, #8 verts


You can see that the data there is the same format as the .mxa file data. Well, after we remove the 'garbage' that is. So, that is the next step. remove all of the information that is not needed. You can see the "eye catcher" that the programmer kindly put in to help us. Once all the garbage data is removed, you are left with a compatible data format for the faceshapes.mxa

Code: Select all
    "NamedShapeKeyWeCreated" : [ [0, [-17.1579833984375,-17.1579833984375,17.1579833984375 ]] , [1, [-17.1579833984375,17.1579833984375,17.1579833984375 ]] , [2, [-17.1579833984375,-17.1579833984375,-17.1579833984375 ]] , [3, [-17.1579833984375,17.1579833984375,-17.1579833984375 ]] , [4, [17.1579833984375,-17.1579833984375,17.1579833984375 ]] , [5, [17.1579833984375,17.1579833984375,17.1579833984375 ]] , [6, [17.1579833984375,-17.1579833984375,-17.1579833984375 ]] , [7, [17.1579833984375,17.1579833984375,-17.1579833984375 ]]  ]


At this point, you can copy/paste this as an entry in the faceshapes.mxa file, and it will be imported the next time you use the mhx2 importer on you model. Keep in mind, unless this is the Last entry in the list os keys in the .mxa, you will need to add a comma after the last bracket.

Code: Select all
    "NamedShapeKeyWeCreated" : [ [0, [-17.1579833984375,-17.1579833984375,17.1579833984375 ]] , [1, [-17.1579833984375,17.1579833984375,17.1579833984375 ]] , [2, [-17.1579833984375,-17.1579833984375,-17.1579833984375 ]] , [3, [-17.1579833984375,17.1579833984375,-17.1579833984375 ]] , [4, [17.1579833984375,-17.1579833984375,17.1579833984375 ]] , [5, [17.1579833984375,17.1579833984375,17.1579833984375 ]] , [6, [17.1579833984375,-17.1579833984375,-17.1579833984375 ]] , [7, [17.1579833984375,17.1579833984375,-17.1579833984375 ]]  ],


then you can add further keys underneath.
As i said, this is a long involved process as it stands at the moment, but it can be done.
What i have written may be a bit confusing. Feel free to let me know if something is not understood.

So, basically, use the shapekeys that get imported when you select faceshapes when importing using MHX. Then, using these shapekeys, make your own shapekeys that match the apple structure you want. Then convert these shapekeys into data that can be used by the MHX importer, and implant that data into the faceshapes.mxa file. Then, at the end of all of this, you will have the apple ar shapes available for Any of your models that use the MHX workflow.

Best of luck, and a request.. If you do get this daunting task done, please share those shape keys with the community. I know that many people are interested in them.

Be well!
nomorecookies
 
Posts: 82
Joined: Thu Jul 30, 2020 1:34 pm

Re: Expressions to Blendshapes

Postby human_maker_77 » Sun May 09, 2021 9:19 pm

Hey thanks for all the details. Some thoughts on my end:

This seems feasible (albeit tedious) except that I currently cannot even get the .mxa file loaded into Blender, so I have no way of using/testing it. Can you please confirm that .30 is the current up to date version of the import MHX2 plugin?

This also seems a bit excessive to what I had hoped. My current situation is that I looked in the directory AppData\Local\makehuman-community\makehuman\data\expressions and found a bunch of files labeled as .mhpose. If I open these up, the breakdown of their components is actually pretty clear, for example looking at surprise01.mhpose, I see:

Code: Select all
{
    "license": "CC0",
    "name": "surprise01",
    "copyright": "(c) 2020 Data Collection AB, Joel Palmius, Jonas Hauquier",
    "tags": [
        "MakeHuman™",
        "surprise"
    ],
    "author": "MakeHuman Team",
    "unit_poses": {
        "MouthLeftPullUp": 0.347,
        "JawDrop": 0.518,
        "MouthRightPullUp": 0.322,
        "LeftUpperLidOpen": 0.352,
        "LeftInnerBrowUp": 0.412,
        "RightInnerBrowUp": 0.317,
        "LipsKiss": 0.347,
        "RightUpperLidOpen": 0.417
    },
    "homepage": "http://www.makehumancommunity.org",
    "description": "No description set"
}


Based on this, I feel like the files for the 52 blendshapes should not be too hard, for example, I modified this file to this:

Code: Select all
{
    "license": "CC0",
    "name": "surprise01",
    "copyright": "(c) 2020 Data Collection AB, Joel Palmius, Jonas Hauquier",
    "tags": [
        "MakeHuman™",
        "jawOpen"
    ],
    "author": "MakeHuman Team",
    "unit_poses": {
        "JawDrop": 0.518
    },
    "homepage": "http://www.makehumancommunity.org",
    "description": "No description set"
}


and saved it as jawOpen.mhpose. Now when I run MakeHuman, I can find my jawOpen expression inside of MakeHuman, and it looks exactly like I would expect. I think I could make files for each of the blendshapes like this, but I do not see an easy way to load all of these expression (as blendshapes) into Blender at once with any MakeHuman model. I guess, based on your suggestions, I could load these each expression into Blender, get it's shape keys, add them to .mxa file, and then try to work with that, but again I would need to get the .mxa file import working for that to be viable.

Another minor note is that I actually need to use the game engine rig, rather than the default one, if the matters.

Last, if I do manage to get this working, I would be happy to share results with the community. Even with the .mxa file route, this sounds doable, just going to take a while. Please let me know if I missed anything or you know of any faster way to get the expressions in.
human_maker_77
 
Posts: 4
Joined: Wed May 05, 2021 3:42 am

Re: Expressions to Blendshapes

Postby nomorecookies » Mon May 10, 2021 4:13 am

Ok. To "open" the faceshapes.mxa file in blener just press the checkbox when you are importing as MHX. You, ofcourse, needed to export your model as MHX from MakeHuman before this, and be importing it into Blender now.


Image

Once you have done this, look under shape keys and you will see a handfull of shape keys.
The process i described will add the 'expressions' as shape keys. BUT, if you look at the shape keys that are installed by default when pressing 'faceshapes' on import, you can see something here. You can see, that just as the expressions are made by a combination of several shapekeys, you can make things to match the apple ar by using a combination of shape keys that are in the list right now. you would change the value of one or more, then do a 'new shape key from mix' and name it using the apple ar structure. I have a feeling a lightbulb :idea: will appear above you head at this point :)
also, using the game skeleton is fine.
What you are doing is the same thing, essentially. But you are doing it through code, instead of through sliders. Also, by doing it manually as you are, you would still need to follow the 'import, import copy, capture, change, export' workflow i described previously.
But, once you click the 'faceshapes' when you are importing, you will have A set of shapekeys to start building from, but they will not be THE set of shapekeys that the values you see withing the text file that makes the expressions are. This is all sounds a lot more confusing then it is. The first thing that you need to do is verify that you can now 'open' the faceshapes.mxa file, then we can go from there.
nomorecookies
 
Posts: 82
Joined: Thu Jul 30, 2020 1:34 pm

Re: Expressions to Blendshapes

Postby LynnSpyre » Wed Feb 16, 2022 3:38 am

It's gonna take me a minute, but I think I can do the arkit as part of the base importer, using target data.
It's just a matter of reformatting it into a python array after I've saved all of them, and consolidating the arrays into a single file. Then adding to the list of things that're there now. I've done a lot of this kind of processing, don't feel bothered by it. While I'm at it, I'll do the female visemes and the vroid standard ones. So you could use it anywhere

Might finish it this weekend, might take a minute. Depends how good I get at the expression mixer.
LynnSpyre
 
Posts: 8
Joined: Mon Feb 14, 2022 3:27 pm

Next

Return to Newbies

Who is online

Users browsing this forum: No registered users and 1 guest