AI-powered mesh alignment from background image

[DerekX]

OK, so I was checking out an add-on for Blender called FaceBuilder. It has the function I've been looking for in a 3D modeling program; AI-powered mesh alignment. The problem is, it costs $150 a year at the cheapest. I'm still too casual to pay anything.

So I was wondering if a similar function could be developed for MakeHuman to enable AI-powered mesh alignment/configuration from a background image? All of the corresponding sliders would be automatically set. (but still adjustable)

[Ricardo2020]

Whcxh corporate cesspool AI are you suggesting we integrate? AI is currently being done mostly in cloud data ceneters which appear to the gadgetbound layman to be on their device. Thus, in order to integrate AI for free (our software is free) rather than going the paywall path means we have to use a model from one or the other corporate cesspit like alphagoo or metox, etc.. See the llamma run.

I for one vote NO to putting that spyware anywhere near what we are doing. It will ruin everything. Right now, facial recognition software is under scrutiny since it iw mostly spyware driven by the very AI you want us to let in there. Sue balls are flying all over the world in hopes of reigning this hype train in before it pisses everybody off no end after they learn what it is really doing while it pretends to be something one would want. It is not.

Now, turn AI into a cross platform ad free utility, perhaps billed monthly as such, and then it would be a good thing. Currently, it is just a marketing gimmick, a digital parrot with a really big memory. However, bird is the word and the AI hype train's whistle goes QUACK! In the not too distant future when the smoke clears, we will laugh at all the hubbub. We will realize what a dumb idea that was and learn how to take the dumb part out. Then it will actually be something we might want in there.

By then our wares should be able to do the required math through sample calculus, no AI needed since everything can be reduced to a matrix of probability based on states both known and extrapolated. We do not yet have the compute in an affordable package to do this, but when CPU makers learn how not to contaminate the fab and glorify their offerings, we will see more reliable and capable parts appear at pricetags we can deal with.

Also, we need to take the crap out of the system so it can DO OUR WORK, NOT SLING ADS THROUGH SPYING, STEELING and SELLING. GRRRRRRRRR!!!!!!!!!

Just kidding. End of rant.

[DerekX]

It almost sounds like you're suggesting that this may be a bad idea due to facial recognition software being under scrutiny. Also, here's where I'm getting the idea: https://medium.com/keentools/facebuilde ... b10c717f7c

[gjh]

OK, so I was checking out an add-on for Blender called FaceBuilder. It has the function I've been looking for in a 3D modeling program; AI-powered mesh alignment. The problem is, it costs $150 a year at the cheapest. I'm still too casual to pay anything.

I used a trial version of Facebuilder sometime back (to anyone interested in checking it out: search youtube for Facebuilder videos). Inarguably, it works in terms of mesh creation (better results if you have left and right images as well as front) and it's easy to use ... but it produces triangle meshes which are an ill fit with MH meshes {edit: wrong, after actually checking, I discovered that Facebuilder uses quad meshes - but they're still not really compatible with MH meshes}. Keentools clearly have their own trained model and I'm guessing that they've riffed off've existing work, maybe the BFM model work (e.g. https://github.com/Shade5/FaceFitting), maybe the FLAME work. If anyone's interested in trying this stuff at home - for a decent treatment, see YFeng's implementation (https://github.com/yfeng95/DECA.git) of the 2021 Siggraph paper “Detailed Expression Capture and Animation” (I'm on Linux, so fooling around with DECA is fairly straightforward if one has reasonable Python-fu).

The thing is, all {edit: except Facebuilder} the image-to-mesh implementations I've seen thus far have used triangle meshes (bummer) and pretty much universally require a licencing arrangement for commercial use.

So I was wondering if a similar function could be developed for MakeHuman to enable AI-powered mesh alignment/configuration from a background image? All of the corresponding sliders would be automatically set. (but still adjustable)

According to my current (very limited) understanding, it is feasible in principle (c.f. Facebuilder) but the community would need to train its own foundational model on a standard MH mesh (not infeasible).

I've been idly wondering whether there might be a half-assed approach using landmarks (https://github.com/Shade5/FaceFitting/blob/master/get_landmarks.py) to bridge the gap between the vertices of the BFM triangular mesh and the vertices of an MH mesh but I'm not hopeful.

FWIW, I've created a standalone {edit: not quite standalone, the jquery and fomantic-ui js/css is fetched from cdnjs.cloudflare.com} web page https://mega.nz/file/bQ8ExBAL#LYwXevOjWK2dNp_Oj8_R2yVztlN31axzL5vRYdFM4LQ that describes the path I took from image to blender mesh via DECA - with, unfortunately, a triangular mesh so that's where the investigation stopped, not much use without being able to use shape keys / MH meshes.

Cheers,

Graham