MocapNET BVH compatible 3D Pose Estimation from RGB

[AmmarkoV]

Hello world,

This is my first post here! First of all thank you for the excellent tools you have provided. I am a PhD student from Greece and since I have been using makehuman as my primary human model generator I would like to contribute something back to the community in the form of my 3D Pose estimation software that can derive BVH poses from RGB images and videos.

The code is written in C++ and uses the Tensorflow C-API as well as OpenCV for the visualization.

The idea is to be able to convert a single person video to a BVH file that can then be used to animate a makehuman model. Ideally you could sit in front of the camera and bring a makehuman model to life by performing poses yourself.

You can both render the animation using the MakeHuman/MakeWalk plugins of blender ( as seen in the Qualitative results on the following video )


However I have also baked a model as a visualization mode inside the demo application ( the project needs to be built with the CMake ENABLE_OPENGL flag and then it is seen if you start the demo application with --opengl )




The repository is the following

https://github.com/FORTH-ModelBasedTracker/MocapNET

Looking forward to your comments, and I hope someone will find it useful.!
Feel free to ask any questions you may have. I am using the github version of makehuman 1.2.0 and the CMU Default Hybrid rig, but scaled to millimeters instead of centimeters.
In order to animate the model with OpenGL I export the model in .dae format which I then load using Assimp convert to my own vertex container file format called .tri


The output quality is not perfect, and hands and face expressions are currently not tracked, however I am working full-time to improve it..!

Ammar

[RobBaer]

Haven't had a chance to try it, but it looks really interesting. Tensorflow is everywhere these days ...

[AmmarkoV]

Haven't had a chance to try it, but it looks really interesting. Tensorflow is everywhere these days ...

Turns out the youtube video in the previous post also does not show the OpenGL visualization, so this is a screen recording of it and me doing stuff in front of the camera

[Shaba1]

If this works "out of the box" this would be great. I have the whole carnage-mellon bvh library and many time there is just not an animation that does exactly what I want. So I end up cutting and pasting thing which leads to fitting problems and frame rate problems ands other things that eventually lead to me just abandoning the whole project. If this program works I could do that posing myself and just import the bvh to my makehuman character. I will leave comment after I test it out.

[AmmarkoV]

A new and improved version of MocapNET is out..! This time it is even more accurate..!

For example with it you can convert it from this youtube to this BVH and then directly load it to your Blender.
The produced BVH is compaitble with the CMU+Face armature I have commited here

=> =>