MPFB for Machine Learning

[CodingInProgress]

Hi everyone

In the past one or two years, I have been experimenting to use MPFB as a tool to generate data for machine learning (all in my spare time). I have reached a point where I feel this might be useful for others as well and I would like to make it publicly available (GPL for code, CC0 for generated data).
I know, I am legally allowed to just do that. However, I would like to know whether the people behind MPFB are fine with it! MPFB is an incredible project and I would hate to see people being demotivated to further contribute because it is being used for AI.

To give an example:
Inspired by this project (https://microsoft.github.io/DenseLandmarks/), I randomly generated MPFB characters, rendered their faces and stored the information where the facial landmarks in the images are (though I didn't use actual landmark, but a custom UV map). With this, I was able to train a neural network which gets an image as input and returns its guesses about where the facial landmarks are. Despite only using the default facial expression/pose so far, the results on real images look surprisingly plausible to me. It is clearly not good enough yet for anything practical, but it is a starting point.

I am also experimenting with going from image to pose and a few other projects, like physics based animations using MPFB shapes for collisions.

My intention is to make those datasets and the code to generate them available to everyone, such that everyone who feels like it, can experiment with them and hopefully create useful tools for artists with them. Of course, I can't make any promises about how those datasets would be used and that's why I am writing this way too long post.

What do you think about this kind of usage for MPFB? Would you be okay with it? Do you think it should not be used like this? All feedback is very welcome!

[tomcat]

being demotivated to further contribute because it is being used for AI.

Neo-Luddism, sadly the inescapable companion of progress. People will always be suspicious of the new. (Not to say without reason.)

I can't speak for MPFB, but your approach is very interesting for my project. I'm making the Sims-like game with a peronage generator based on MakeHuman. And it would be useful for me to be able to fill the city with nice (not exactly horrible) looking characters. Your approach could probably help with that.

[CodingInProgress]

being demotivated to further contribute because it is being used for AI.

Neo-Luddism, sadly the inescapable companion of progress. People will always be suspicious of the new. (Not to say without reason.)

As with everything new, it takes time to adapt. Everything new will also be abused in every way possible. I have never seen so people being discouraged by a something new as I have seen with AI. I don't agree with many things being made and I don't agree with the kind of marketing being made with unrealistic or blatantly false claims. That's why I understand everyone with a skeptical view and even the ones with a negative view on this development.
If someone like @joepal (according to what I am seeing the main driving force behind this project) would not feel comfortable with me doing this, I would respect that.

I can't speak for MPFB, but your approach is very interesting for my project. I'm making the Sims-like game with a peronage generator based on MakeHuman. And it would be useful for me to be able to fill the city with nice (not exactly horrible) looking characters. Your approach could probably help with that.

That's one kind of application I had in mind. However, at this point, I have focused more on facial motion capture in 2D, because it is way simpler overall and it serves as a good starting point to explore other directions, like the one you mentioned.

[joepal]

What do you think about this kind of usage for MPFB? Would you be okay with it? Do you think it should not be used like this? All feedback is very welcome!

I think this is very interesting. I've been experimenting a lot with AI lately, and have been fiddling with pytorch and custom nodes in comfyui. There I've been considering building a bridge between comfyui and blender, in order to move data in both directions. It'd be very interesting to be able to have, for example, a pytorch code and a model that would be able to take an image and use that to create an approximate human model in blender.