phonemes Visemes and facial expressions

Discussions about MHX2 and other plugins that are developed outside the scope of MakeHuman. Note that bug reports should go to the respective code projects and might go unseen here.

Moderator: joepal

phonemes Visemes and facial expressions

Postby Shaba1 » Tue Jun 23, 2015 10:22 pm

A long time ago there use to be a panel under mh pose/animation that would let you choose visemes for the characcter. These got exported as shape keys I assume, I never did use that feature I just noticed that it was there. Now I know that you can chooes to include a facial rig with mhx2 import and that you will get a set of expression shape keys. these are fine but is there any plan to include standard phenome or visemes back again?
Shaba1
 
Posts: 164
Joined: Wed Mar 05, 2008 3:21 pm

Re: phonemes Visemes and facial expressions

Postby Shaba1 » Fri Jun 26, 2015 1:10 am

Never mind. ignore this message
I found the panel myself. I will briefly give an outline for those that were lost like myself.

#1 When you import the mxh2 file into blender be sure you check the box that says "Override Exported Data" on the right side of the screen where you choose the .mhx2 file that you want to import. This will bring more options and checkboxes and menus below that initial checkbox.

#2 Scroll downward and check the box that says Face shapes. Another checkbox will appear below that that says Face drivers. Check that also.

# 3 Scroll the panel again until you see the checkbox that says "Add Rig".Check that box an the rig panel will open up below it. You will not get visemes or expressions or facial bones if you do not have a rig/armature with the mesh.

#4 There will be a drop down menu that says "Exported" That choice will keep whatever mesh that you chose in Makehuman when you saved the .mhx2 file. If you click on the down point arrow to the right of the word Exported you will see a pop up menu with 9 other choices of rigs/armatures. If you choose one of these it will replace the righ that you designated when you saved the .mhx2 file in Makehuman.

#5 Blender will work for a while depending on how detailed your mesh/model and the armature you chose is. The 3d viewport of blender will then appear with your model with the armature selected and in the pose mode.

#6 Press numpad 1 to go to the front view of whatever viewport you have showing.

# 7 Press the T key to pop out the blender toolbox from the left side of that view port.

# 8 Left click on the "MHX Runtime" tab in the toolbox. The mxh panel will appear to the right of the tabs

# 9 Scroll down the panel until you see visemes and click the right pointing arrow next to the word. The panel will expand and have 15 buttons representing each viseme. press each button to see the character's lips and tounge take on that viseme shape.


I know this was a very slow and detailed tutorial but I wanted to go step by step so that other people that might have been confused like me would be helped

Thanks.
Shaba1
 
Posts: 164
Joined: Wed Mar 05, 2008 3:21 pm

Re: phonemes Visemes and facial expressions

Postby Rhynedahll » Thu Nov 12, 2015 3:01 am

Just to verify, these are still in 0.27?
Orphan, an epic fantasy by H. Jonas Rhynedahll
The Key to Magic: Book One
Available on Kindle at Amazon: The Key to Magic
Rhynedahll
 
Posts: 195
Joined: Sat Nov 28, 2009 1:23 pm

Re: phonemes Visemes and facial expressions

Postby ThomasL » Thu Nov 12, 2015 8:20 am

Rhynedahll wrote:Just to verify, these are still in 0.27?

Yes, nothing has been removed.
ThomasL
 
Posts: 1139
Joined: Tue Sep 15, 2009 2:46 am

Re: phonemes Visemes and facial expressions

Postby ThomasL » Thu Mar 17, 2016 9:00 am

SonAmyFan362 wrote:Can someone tell me how to use each Viseme as a Shape Key, please?


That may not be so simple. Each viseme is a combination of the fundamental face units in the Facial Shapes panel. The combinations are defined in the file data/hm8/faceshapes/visemes.mxa. This is a json file that lists the combinations for each viseme. One could write a script that combines the unit shapes into viseme shapekeys, but no such script exists today.
ThomasL
 
Posts: 1139
Joined: Tue Sep 15, 2009 2:46 am

Re: phonemes Visemes and facial expressions

Postby jcpalmer » Thu Mar 17, 2016 6:04 pm

I have done some work with the face shapes, but for export to a Webgl framework, BabylonJS. As Thomas says, the visemes are just a result of a macro which sets values on some of the 50 shapekeys. They are not actual Blender entities.

I have decided to deal with the visemes external to Blender, in the exported environment. There I can execute my own macro to make the various settings on the shapes that constitute a viseme. Should note that there is no single set of visemes, or consistent terminology that everyone uses. Different people / systems designate different phonemes to also be visemes.

Wanting to combine expressions with speech, I was exporting all 50 keys at first. Each time the character was loaded in BabylonJS, I made 6 combined keys for actual speech, 2 shapes for mood, Happy & Angry, and then deleted the 50 original keys. My 2 expressions had 9 settings (actually any value between 0 - 1), but they are not as varied as the ones coming from MH.

I have now made a script, part of the export process, which combines the 6 keys on the Blender side. It also reduced the export file size from 10 mb to 4 only exporting 6 keys. The script could also be used to make mhx2's version of the visemes actual Blender shape keys. I list it at the bottom.

Expressions can be converted to shape keys, which I also have code for. Problem is you can only get them when you have that huge default rig. I needed a way to save the resulting keys, so they can be applied to MH with any or no rig. I made a .blend with scripts in it to convert the expressions where the mouth is not too open to shapekeys (13 keys). Another script writes the keys out to a .mxa file (though only on the body mesh right now) . It loads when I modified the mhx2 runtime to include it with the face shapes, so I know the data is correct, but am still trying to get it working when I try to call the runtime from my module.

Hope this is helpful.
Code: Select all
import bpy

#===============================================================================
def zeroKeys(mesh):
    for block in mesh.data.shape_keys.key_blocks:
        block.value = 0

#assume mesh has already merged teeth, tongue, eye lashes, eye browses to the body mesh
def combineFaceKeys(scene, mesh, delete = False):
    combineKey(scene, mesh, ['mouth_open'], [1], 'FACE-MOUTH_OPEN', delete) # really just a name change
    combineKey(scene, mesh, ['mouth_wide_left', 'mouth_wide_right'], [1, 1], 'FACE-MOUTH_WIDE', delete)
    combineKey(scene, mesh, ['mouth_up_left'  , 'mouth_up_right'  ], [1, 1], 'FACE-MOUTH_UP'  , delete)
    combineKey(scene, mesh, ['lips_mid_upper_up_left', 'lips_mid_upper_up_right'], [1, 1], 'FACE-LIPS_MID_UPPER_UP', delete)
    combineKey(scene, mesh, ['lips_mid_lower_down_left', 'lips_mid_lower_down_right'], [1, 1], 'FACE-LIPS_MID_LOWER_DOWN', delete)
    combineKey(scene, mesh, ['tongue_back_up', 'tongue_up'], [1, 1], 'FACE-TONGUE', delete)
#    combineKey(scene, mesh, ['', ''], [1, 1], 'FALSE-', delete)

    zeroKeys(mesh)
   
def combineKey(scene, mesh, keysToCombine, keyValues, finalName, delete = False):
    # bpy.ops.object.shape_key_add will only work when mesh is the active object
    scene.objects.active = mesh
   
    # delete any key of the final name, so can be done more than once
    deleteKey(scene, mesh, finalName)
   
    # clear all the values of each shape key
    zeroKeys(mesh)
       
    # apply each of the keys to composite the new one
    idx = 0
    for shapeName in keysToCombine:
        mesh.data.shape_keys.key_blocks[shapeName].value = keyValues[idx]
        idx += 1
       
    # create the new key from a mix
    bpy.ops.object.shape_key_add(from_mix=True)

    # name the new key
    mesh.data.shape_keys.key_blocks[mesh.active_shape_key_index].name = finalName

    # delete keys on request
    if delete:
        for shapeName in keysToCombine:
            deleteKey(scene, mesh, shapeName)

def removeCombinedKeys(scene, mesh):
    deleteKey(scene, mesh, 'FACE-MOUTH_OPEN')
    deleteKey(scene, mesh, 'FACE-MOUTH_WIDE')
    deleteKey(scene, mesh, 'FACE-MOUTH_UP')
    deleteKey(scene, mesh, 'FACE-LIPS_MID_UPPER_UP')
    deleteKey(scene, mesh, 'FACE-LIPS_MID_LOWER_DOWN')
    deleteKey(scene, mesh, 'FACE-TONGUE')

def deleteKey(scene, mesh, shapeName):
    scene.objects.active = mesh
    bpy.ops.object.mode_set(mode='OBJECT')

    # cannot just assign mesh.active_shape_key, since it is readonly, must set index
    for idx in range(len(mesh.data.shape_keys.key_blocks)):
        if mesh.data.shape_keys.key_blocks[idx].name == shapeName:
            mesh.active_shape_key_index = idx
            bpy.ops.object.shape_key_remove(all=False)
            print('deleted ' + shapeName)
            return
       
    print('failed to delete key: ' + shapeName)
#===============================================================================
# add import_runtime_mhx2 to sys.path, so can be imported
mhxOnPath = False
def putMhxOnPath():
    global mhxOnPath
    if not mhxOnPath:
        import os
        import sys
        addon_paths = bpy.utils.script_paths('addons')
        for path in addon_paths:
            lib_path = os.path.join(path, 'import_runtime_mhx2')
            if os.path.exists(lib_path):
                print('mhx2 found @ ' + lib_path)
                sys.path.append(lib_path)
                mhxOnPath = True
                break
   
    return mhxOnPath
   
def importMHShapes(human, filename):
    if not putMhxOnPath():
        raise 'mhx2 runtime could not be found'
   
    from os import path
    from import_runtime_mhx2.shapekeys import getScales, addTargets
    from import_runtime_mhx2.load_json import loadJson
    from import_runtime_mhx2.utils     import getMhHuman
       
    filePath = path.join(path.dirname(__file__), filename)
    print('importing shapes from:  ' + filePath)
    mhHuman = getMhHuman()
    struct = loadJson(filePath)
    scales = getScales(human, struct["bounding_box"], mhHuman)
    addTargets(human, struct["targets"], scales)
jcpalmer
 
Posts: 115
Joined: Tue Dec 16, 2014 4:14 pm

Re: phonemes Visemes and facial expressions

Postby caliche » Tue Jun 28, 2016 6:12 pm

jcpalmer, I am very interested in how you did this for BablylonJS? Would it be possible to create a simple video showing your workflow from the MH to BabylonJS?
caliche
 
Posts: 1
Joined: Tue Jun 28, 2016 6:09 pm


Return to MHX2 and other plugins developed outside MakeHuman

Who is online

Users browsing this forum: No registered users and 1 guest