Skip to content
This repository has been archived by the owner on Sep 5, 2023. It is now read-only.
/ MoPA Public archive

An Blender addon uses ROMP to extract human's 3D poses from image, video or webcam and drive your own 3D character.

License

Notifications You must be signed in to change notification settings

yanchxx/MoPA

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

Mono Pose Animator

License: MIT GitHub repo size

This project is no longer maintained.

This addon helps you drive a 3D character in Blender using ROMP based on image, video or webcam and get a 3D character that can be drived by ROMP.

Demo

  • Use webcam to drive a 3D character on a linux computer with GPU 1070.

webcam_linux_local

  • Use webcam to drive a 3D character on a Macbook Pro connected to a linux server with GPU 1070. (Due to the network latency and program limitation in Blender python, the driving is not fluent.)

webcam_mac_server

  • Use a video to drive 3D characters.
MJ.mp4
MJ2.mp4

Installation

For Blender Addon

  • Download the addon.zip.
  • Install the addon in Blender.
  • Check your 3d view and there should be a new menu item called CDBA.
  • Install opencv in Blender python.
# if your Blender python doesn't have pip, install pip first
/path/to/blender/python -m pip install opencv-python

For Simple ROMP

# create a conda environment separate from blender python
conda create -n simple_romp python=3.8 -y
conda install pytorch torchvision cudatoolkit -c pytorch # if you don't have GPU, don't execute this command
pip install --upgrade setuptools numpy cython
pip install --upgrade simple-romp

# test if romp run successfully
romp --mode=video --show_largest -i=/path/to/video.mp4 -o=/path/to/results # for GPU
romp --mode=video --show_largest --onnx -i=/path/to/video.mp4 -o=/path/to/results # for CPU

How To Use

Panel

panel

Common

  • Character: The character's object name.
  • Armature: The character's armature name.
  • IP: The IP address of the romp_server.
  • Port: The port of the romp_server.
  • Use GPU: Whether to use GPU.
  • Use Translation: Whether to use estimated global translation.
  • Insert Keyframes: Whether to insert keyframes.
  • Insert Interval: The interval of inserted keyframes.
  • Compression: The transferred image's compression quality.
  • Fix Bones: Make mixamo rigged character compatible with ROMP.

Image or Video

  • Offline Animation: Select your image or video file to drive your character.

Webcam

  • FPS: The frame rate of the webcam.
  • Width: The width of the webcam.
  • Height: The height of the webcam.
  • Webcam Animation: Open your webcam to drive your character.

Drive Your 3D Character

python romp_server.py [port]
  • Open Blender using command line.
# if you start the software by clicking on its icon
# it will crash when using webcam animation
blender 
  • Make sure the settings are correct in Blender CDBA Panel
    • Make sure your character object's and armature's name is same with the one in Common Panel.
    • Make sure you can connect to the romp_server using the IP and Port in Common Panel.
    • Other settings are optional, the default is fine.
  • Set the number in the red box to specify the starting point for keyframe insertion. frame_start
  • Click Offline Animation to use image or video to drive your character.
  • Click Webcam Animation to use webcam to drive your character.

The Blender and romp_server.py is not necessary to be in the same computer. You can run romp_server.py on a linux server and use port forwarding to make the romp_server can be accessed from your local computer. A simple method is to use VS Code to set up port forwarding.

Make Your 3D Character Compatible with ROMP

  • Use mixamo auto-rigger to rig your 3D character. Remember to select no fingers (25).
  • Import the rigged 3D character to Blender.
  • If your character is T-pose, skip this step. If your character is A-pose,
    • install the cats-blender-plugin in Blender;
    • select your character and go to Pose Mode;
    • click the CATS tab, then click the Apply as Rest Pose button.

All mixamo rigged characters are T-pose in Object Mode. You need to check whether your character's rest pose is A-pose or T-pose in Edit Mode.

  • Click the CDBA tab, then click Fix Bones to make your character compatible with ROMP.

Acknowledgement

About

An Blender addon uses ROMP to extract human's 3D poses from image, video or webcam and drive your own 3D character.

Topics

Resources

License

Stars

Watchers

Forks

Languages