Hi everyone,
I have been playing with MakeHuman and I think it is a great start. But I think there could be an even better system that could quite literally generate any type of human you have in your head. To do this, I am thinking of a MakeHuman style system that would use AI techniques that have come out over the past few years. It would use generative neural networks to create the mesh from rough user sketches of the body type they are looking for.
If anyone would want to work on this with me I would love the help. It's not so easy to make, as you can imagine. I'll give you the rough plan as of now. I've already begun testing but it takes a lot of work to do this.
Idea:
Step 1 - Imagine this, you have a 2D window of your figure with a front pose, 3/4 pose, side pose and back pose. Overtop of the mannequin you can roughly draw
the parameters of the person you want. The program would be able to generate any type of person, from a 2 foot tall toddler to an 8 foot tall giant.
Broad shoulders, to narrow shoulders, large waist to small ect. The model would be rigged with weights already painted and ready to go.
Step 2 - When you have picked out your primary form, it would lock it into place and then you could draw over top of it again to define your musculature. You would decide
the size of biceps, chest, or stomach size, any variation to get any human body you could ever want. The body would have corrective shape keys which would deform muscle
groups based on the motion of the arms or body.
Method:
I have an idea of how to go about all of this, though I am not sure if it will work. It is very much in a testing phase and follows the work done in the SMPL and STAR (Human Generation Systems) system as a compass. I don't like SMPL
because number one it is prohibitively expensive for anyone to use that is not in research and it doesn't generate stylized humans because it was trained on real scans of humans. Also, it lumps females and males together which consistently generates human mesh models that look in between.
For starters, I would try to do Step 1 in full which I think could be a great addition to what is already here. I am confident that step 1 is doable from papers I have already read.
Essentially I am doing Step 1 with this technique. I have a base mesh already made for animation purposes. I am using shape keys to get all of the different body types.
There are currently 24 shape keys and I expect this number to go up to 50-100. Now, as you can imagine that would be terrible for people to use, as no one would want to
move 50-100 shape keys to make a human figure and these shape keys can deform the model to look nothing like a person!
But that is where the GAN comes in. I plan to train a GAN to take the vector of 50-100 shape keys and "fill in" the blanks by generating valid shape key parameters. So essentially
You would only have to pick 5-10 shape keys that you want to change and the GAN would fill in the rest to make it look realistic.
Finally, when the GAN is finished, I will use it to generate a large number of valid body meshes and use Blender Freestyle Render to generate thousands of line drawings
of people. Then train a Convolution Neural Network to learn what type of parameters it should set depending on the picture drawn by the user. To make this more functional
an online method could be used (which shows great success in these sorts of algorithms in research papers) where the drawings would take constant stylus input from the user to help build the figure.
The armature is set by looking at the location of key vertices, (Rib cage bottom, Arm top) and then setting weight vertice groups which constitute edge loops. These edge loops
remain the same even as they are changed using the shape keys so in the end you SHOULD have a primary shape that has an armature and is weight painted and ready to go.
So, in step 1, the person is essentially a vector of human shape key parameters which are generated by a GAN from very little input. This is very much inspired by the SMPL framework currently employed in the deep learning community for generating human shapes.
To take it further, I'd like to add muscles to the model of any shape and size. This is still WIP as I'm not exactly sure how I can do this just yet though I do have an idea I at the beginning stages of testing for which I think will work.
If anyone wants to work on this, please hit me up. I need a lot of help in designing something of this complexity and it would awesome to get people that understand blender and human generating systems to give me input on what they are looking for and what may or may not be possible...
My discord is Pudge#8503