AI based Make Human

This forum is aimed at user contributions, in the form of assets, side projects, code patches and similar.

Moderator: joepal

AI based Make Human

Postby HumanMakeNoob » Mon Dec 14, 2020 9:05 pm

Hi everyone,

I have been playing with MakeHuman and I think it is a great start. But I think there could be an even better system that could quite literally generate any type of human you have in your head. To do this, I am thinking of a MakeHuman style system that would use AI techniques that have come out over the past few years. It would use generative neural networks to create the mesh from rough user sketches of the body type they are looking for.

If anyone would want to work on this with me I would love the help. It's not so easy to make, as you can imagine. I'll give you the rough plan as of now. I've already begun testing but it takes a lot of work to do this.

Idea:
Step 1 - Imagine this, you have a 2D window of your figure with a front pose, 3/4 pose, side pose and back pose. Overtop of the mannequin you can roughly draw
the parameters of the person you want. The program would be able to generate any type of person, from a 2 foot tall toddler to an 8 foot tall giant.
Broad shoulders, to narrow shoulders, large waist to small ect. The model would be rigged with weights already painted and ready to go.

Step 2 - When you have picked out your primary form, it would lock it into place and then you could draw over top of it again to define your musculature. You would decide
the size of biceps, chest, or stomach size, any variation to get any human body you could ever want. The body would have corrective shape keys which would deform muscle
groups based on the motion of the arms or body.

Method:
I have an idea of how to go about all of this, though I am not sure if it will work. It is very much in a testing phase and follows the work done in the SMPL and STAR (Human Generation Systems) system as a compass. I don't like SMPL
because number one it is prohibitively expensive for anyone to use that is not in research and it doesn't generate stylized humans because it was trained on real scans of humans. Also, it lumps females and males together which consistently generates human mesh models that look in between.

For starters, I would try to do Step 1 in full which I think could be a great addition to what is already here. I am confident that step 1 is doable from papers I have already read.

Essentially I am doing Step 1 with this technique. I have a base mesh already made for animation purposes. I am using shape keys to get all of the different body types.
There are currently 24 shape keys and I expect this number to go up to 50-100. Now, as you can imagine that would be terrible for people to use, as no one would want to
move 50-100 shape keys to make a human figure and these shape keys can deform the model to look nothing like a person!

But that is where the GAN comes in. I plan to train a GAN to take the vector of 50-100 shape keys and "fill in" the blanks by generating valid shape key parameters. So essentially
You would only have to pick 5-10 shape keys that you want to change and the GAN would fill in the rest to make it look realistic.

Finally, when the GAN is finished, I will use it to generate a large number of valid body meshes and use Blender Freestyle Render to generate thousands of line drawings
of people. Then train a Convolution Neural Network to learn what type of parameters it should set depending on the picture drawn by the user. To make this more functional
an online method could be used (which shows great success in these sorts of algorithms in research papers) where the drawings would take constant stylus input from the user to help build the figure.

The armature is set by looking at the location of key vertices, (Rib cage bottom, Arm top) and then setting weight vertice groups which constitute edge loops. These edge loops
remain the same even as they are changed using the shape keys so in the end you SHOULD have a primary shape that has an armature and is weight painted and ready to go.

So, in step 1, the person is essentially a vector of human shape key parameters which are generated by a GAN from very little input. This is very much inspired by the SMPL framework currently employed in the deep learning community for generating human shapes.

To take it further, I'd like to add muscles to the model of any shape and size. This is still WIP as I'm not exactly sure how I can do this just yet though I do have an idea I at the beginning stages of testing for which I think will work.

If anyone wants to work on this, please hit me up. I need a lot of help in designing something of this complexity and it would awesome to get people that understand blender and human generating systems to give me input on what they are looking for and what may or may not be possible...

My discord is Pudge#8503
HumanMakeNoob
 
Posts: 6
Joined: Mon Dec 14, 2020 8:22 pm

Re: AI based Make Human

Postby loki1950 » Mon Dec 14, 2020 10:01 pm

That sounds like it could work out to be a life project ;) but is way beyond our project goals which is to be a starting point in the character creation process not provide a finished character we are a very small group of devs so our efforts need to be focused but we should be able to give some help in integrating your AI with the current code.This will be a long project for you :o

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo Ideapad 320-15ABR Win 10/Mint 19
User avatar
loki1950
 
Posts: 1219
Joined: Thu Dec 18, 2014 6:27 pm
Location: Ottawa,Ontario

Re: AI based Make Human

Postby HumanMakeNoob » Tue Dec 15, 2020 1:59 pm

loki1950 wrote:That sounds like it could work out to be a life project ;) but is way beyond our project goals which is to be a starting point in the character creation process not provide a finished character we are a very small group of devs so our efforts need to be focused but we should be able to give some help in integrating your AI with the current code.This will be a long project for you :o

Enjoy the Choice :)


Hey, any help would be amazing! Believe it or not, a lot of this stuff is being done as we speak. The boom in 3D AI over the past few years has been huge and there are systems that already generate human mesh but solely for research purposes, not for art. The most difficult thing about this is that I don't have access to a dataset like they did. The dataset they had was prohibitively expensive. And it doesn't do humans of any shape or size as it was made with real scans of the human body.

I'm currently trying to flesh out a prototype of just the male torso. I'll post progress on this thread.
HumanMakeNoob
 
Posts: 6
Joined: Mon Dec 14, 2020 8:22 pm

Re: AI based Make Human

Postby MTKnife » Wed Dec 16, 2020 5:17 am

FWIW, I'm a data scientist. I don't work much with neural nets, but I understand something about how GAN works. I might be able to provide some advice if nothing else.
MTKnife
 
Posts: 311
Joined: Tue Sep 01, 2015 5:22 am

Re: AI based Make Human

Postby HumanMakeNoob » Wed Dec 16, 2020 12:57 pm

MTKnife wrote:FWIW, I'm a data scientist. I don't work much with neural nets, but I understand something about how GAN works. I might be able to provide some advice if nothing else.


Hey, thanks man!

Ive rethought how I want to do the primary forms. Upon looking at the problem again, I really think that there may be a simpler solution than a GAN. The proportions on humans have a lot of constraints even for stylized humans. Forming a good vector might be a better approach and finding the constraining relationships between these parameters. Then hand coding those constraints may be much faster and easier to do.

For instance, my original vector defining human shape had pelvis depth as a parameter. But that dimension is almost always coupled to the pelvis width. It is a fully dependent variable, and these proportions are well defined in art books. A large amount of that which makes a human is defined by these inter parameter relationships, so why train a GAN to find these constraints when they can be hand coded.

I assume at this stage, at least for making primary forms, it is more about expanding the capabilities of MakeHuman by adding more shape parameters and creating constraints between them. I assume MakeHuman uses something akin to shape keys to generate the body shapes, Ill have to check.
HumanMakeNoob
 
Posts: 6
Joined: Mon Dec 14, 2020 8:22 pm

Re: AI based Make Human

Postby loki1950 » Wed Dec 16, 2020 4:20 pm

We call them targets but yes in blender they become shape keys check our maketarget blender addon for all the math ;)

Enjoy the Choice :)
my box::HP Envy i5-6400 @2Q70GHzx4 8 Gb ram/1 Tb(Win10 64)/3 Tb Mint 19.2/GTX745 4Gb acer S243HL K222HQL
Q8200/Asus P5QDLX/8 Gb ram/WD 2Tb 2-500 G HD/GF GT640 2Gb Mint 17.3 64 bit Win 10 32 bit acer and Lenovo Ideapad 320-15ABR Win 10/Mint 19
User avatar
loki1950
 
Posts: 1219
Joined: Thu Dec 18, 2014 6:27 pm
Location: Ottawa,Ontario

Re: AI based Make Human

Postby MTKnife » Thu Dec 17, 2020 12:26 am

Wasn't someone coding a script to create mass numbers of models for crowd scenes?
MTKnife
 
Posts: 311
Joined: Tue Sep 01, 2015 5:22 am

Re: AI based Make Human

Postby HumanMakeNoob » Thu Dec 17, 2020 6:27 am

MTKnife wrote:Wasn't someone coding a script to create mass numbers of models for crowd scenes?


How long ago was that? \ :P
HumanMakeNoob
 
Posts: 6
Joined: Mon Dec 14, 2020 8:22 pm

Re: AI based Make Human

Postby joepal » Thu Dec 17, 2020 2:48 pm

If we're talking about the stuff Duststorm did, then that was many years ago: https://www.youtube.com/watch?v=9EpqNxUKGdA

But there are other stuff like this too: https://www.youtube.com/watch?v=pvFEtQBvlPU&t=11s

If we're talking about the mass produce plugin (https://www.youtube.com/watch?v=jRHnJX-TdT4), then that should be bundled with your makehuman installation.
Joel Palmius (LinkedIn)
MakeHuman Infrastructure Manager
http://www.palmius.com/joel
joepal
 
Posts: 4465
Joined: Wed Jun 04, 2008 11:20 am

Re: AI based Make Human

Postby HumanMakeNoob » Wed Dec 30, 2020 3:35 pm

Okay, I've figured it out my algorithm for this and I've tested it (to some extent). I think it should work but it would be great if someone with more experience could give me their input.

First, the base armature is put together with constraints set from art books on human proportions. It takes age and a range for each component. Arm length, Shoulder width, etc etc
These proportions are created by scaling the bones in the armature. The armature is parented to the mesh in such a way that scaling bones causes a smooth transition between joints. This is done with basic linear skinning.

Next, secondary forms, muscle groups. A set of base shape keys is created to change musculature and weight of the person. Muscles can be thickened or shrunken to get a large number of different body types.

The mesh above is cloned multiple times for each rotation of each joint. So, for example, the elbow can bend 0 degrees and 90 degrees. There should be two sculptures for this. This is done for each joint of the body. Now, the laborious part of this... Each copy of each cloned mesh needs to be "tweaked" for each shape key on that bone. So, for the upper arm, I have about 5-8 shape keys. Each one needs to be re-sculpted for each joint rotation. However, this sculpting is not very difficult since the basic forms are already there, they just have to be tweaked. A scaling copy constraint is placed on all of the bones in each copy, this preserves scaling. I will design code that will take the difference between the vertices in both statues correlating to a joint, the difference between the joint rotation and combine this into a corrective shape key. All of the old shape keys are removed and are replaced with these corrective shape keys.

All of the above are merged into one final mesh with all of the corrective shape keys on it.

Finally, the old armature will be used as a basis for a new armature that will be for animation. Weights will be applied based on edge loop groups. These edge group loops will change weights around the loop depending on the dimensions of the body.

At the end, the user should have an animate-able male body of any type they need, with muscles that are able to deform.

So far the testing seems to be going quite well. Shape keys are scaling quite nicely. Is there anything I am not seeing that I will find later? Anyone with experience think this may have a problem somewhere?
HumanMakeNoob
 
Posts: 6
Joined: Mon Dec 14, 2020 8:22 pm

Next

Return to User contributions

Who is online

Users browsing this forum: No registered users and 1 guest