An executor that wraps 3D mesh models and encodes 3D content documents to d-dimension vector.

Related tags

Image Processingexecutor-3d-encoder
Overview

3D Mesh Encoder

An Executor that receives Documents containing point sets data in its blob attribute, with shape (N, 3) and encodes it to embeddings of shape (D,). Now, the following pretrained models are ready to be used to create embeddings:

  • PointConv-Shapenet-d512: A PointConv model resulted in 512 dimension of embeddings, which is finetuned based on ShapeNet dataset.
  • PointConv-Shapenet-d1024: A PointConv model resulted in 1024 dimension of embeddings, which is finetuned based on ShapeNet dataset.

Usage

via Docker image (recommended)

from jina import Flow

f = Flow().add(uses='jinahub+docker://3DMeshEncoder', \
               uses_with={'pretrained_model': 'PointConv-Shapenet-d512'})

via source code

from jina import Flow

f = Flow().add(uses='jinahub://3DMeshEncoder', \
               uses_with={'pretrained_model': 'PointConv-Shapenet-d512'})

This Executor offers a GPU tag to speed up encoding. For more information on how to run the executor on GPU, check out the documentation.

How to finetune pretrained-model?

install finetuner

$ pip install finetuner

prepare dataset

TBD...

finetuning model with labeled dataset

$ python finetune.py --help

$ python finetune.py --model_name pointconv \
    --train_dataset /path/to/train.bin \
    --eval_dataset /path/to/eval.bin \
    --batch_size 128 \
    --epochs 50

finetuning model with unlabeled dataset

$ python finetune.py --model_name pointconv \
    --train_dataset /path/to/unlabeled_data.bin \
    --interactive

References

  • PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
  • PointConv: Deep Convolutional Networks on 3D Point Clouds
Comments
  • which version should the `finetuner` module be?

    which version should the `finetuner` module be?

    I was preparing the environment of this repo. However, I found it confused that the python import code in finetune.py gives error message:

    from finetuner.tuner.callback import BestModelCheckpoint
from finetuner.tuner.pytorch.losses import TripletLoss
from finetuner.tuner.pytorch.miner import TripletEasyHardMiner

    It seems that there's no finetuner.tuner in finetuner module but finetuner.finetuner. Besides, callback/pytorch is not in finetuner.finetuner now.

    Now my finetuner module installed by running pip install -U finetuner is 0.5.0. Is there any solution? Maybe you can freeze the version in requirements.txt.

    opened by elenacliu 4
  • fix: fix previous structure

    fix: fix previous structure

    I got pretrain and finetune process mixed up, and previously I pretrained the models using finetune model structure, which might lead to worse results. The fix code is in process and may be finished in a few days.

    opened by elenacliu 2
  • Add dataset, preprocess and pointnet2

    Add dataset, preprocess and pointnet2

    Here is the modification of the structure. There are still some bugs in my pointnet2 codes. I will solve the possible problems after you comment.

    @numb3r3 @jemmyshin

    opened by elenacliu 1
  • In `farthest_point_sample` function, the first centroid sampled is not random.

    In `farthest_point_sample` function, the first centroid sampled is not random.

    I'm reading your codes.

    https://github.com/jina-ai/executor-3d-encoder/blob/main/executor/models/modules.py#L75

    The randint line is commented, but in the original implementation and the algorithm itself, it should be random? Maybe we can provide a fixed seed if we want to get the same result?

    opened by elenacliu 1
  • Provide diffusion encoder

    Provide diffusion encoder

    https://github.com/nmwsharp/diffusion-net

    I'm unfamiliar with the jina ecosystem. So this is my first issue.

    Final goal is to have dalle generate 3d digital human meshes like https://github.com/jina-ai/dalle-flow

    opened by fire 2
Releases(v0.3.0)
Owner
Jina AI
A Neural Search Company. We provide the cloud-native neural search solution powered by state-of-the-art AI technology.
Jina AI
GitHub Repository
Fixed Version Of Blender Low Poly Rock Generator For Blender 3.0.0

Blender (3.0.0) - Low Poly Rock Generator This is an addon for Blender 3.0.0 to generate low poly rocks. It was based on an addon that unfortunately h

3 Mar 24, 2022
GTK and Python based, simple multiple image editor tool

System Monitoring Center GTK3 and Python3 based, simple multiple image editor tool. Note: Development of this application is not completed yet. The ap

Hakan Dündar 1 Feb 02, 2022
QR-code Generator with a basic GUI.

Qr_generator_python Qr code generator with a basic GUI. ❔ About the QR-Code-Generator This project Generates QR codes to sites, e-mails and plain text

Tecixck 2 Oct 11, 2021
Extract the ISO 11146 beam size from an image file

laserbeamsize Simple and fast calculation of beam sizes from a single monochrome image based on the ISO 11146 method of variances. Some effort has bee

Scott Prahl 21 Jan 06, 2023
A utility for quickly cropping large collections of images.

Crop Tool A utility for quickly cropping large collections of images. Inspired by Derrick Schultz's dataset-tools. Setup It's suggested that you use A

dusk (they/them) 6 Nov 14, 2021
Make GIFs from time-stacked xarray.DataArrays (time, [optional band], y, x), dead-simple.

GeoGIF Make GIFs from time-stacked xarray.DataArrays (time, [optional band], y, x), dead-simple. from geogif import gif, dgif gif(data_array) dgif(das

Gabe Joseph 47 Dec 22, 2022
Unique image & metadata generation using weighted layer collections.

nft-generator-py nft-generator-py is a python based NFT generator which programatically generates unique images using weighted layer files. The progra

Jonathan Becker 243 Dec 31, 2022
A simple Streamlit Component to compare images in Streamlit apps. It integrates Knightlab's JuxtaposeJS

streamlit-image-juxtapose A simple Streamlit Component to compare images in Streamlit apps using Knightlab's JuxtaposeJS. The images are saved to the

Robin 30 Dec 31, 2022
A large-scale dataset of both raw MRI measurements and clinical MRI images

fastMRI is a collaborative research project from Facebook AI Research (FAIR) and NYU Langone Health to investigate the use of AI to make MRI scans faster. NYU Langone Health has released fully anonym

Facebook Research 907 Jan 04, 2023
A proof-of-concept implementation of a parallel-decodable PNG format

mtpng A parallelized PNG encoder in Rust by Brion Vibber [email protected] Backgrou

Brion Vibber 193 Dec 16, 2022
Alternate Python bindings for the Open Asset Import Library (ASSIMP)

Impasse A simple Python wrapper for assimp using cffi to access the library. Requires Python = 3.7. It's a fork of PyAssimp, Assimp's official Python

Salad Dais 3 Sep 26, 2022
Script For Importing Image sequences into scrap mechanic via blueprints

To use dowload and extract "video makes.zip" Python has to be installed https://www.python.org/ (may not work on version lower than 3.9) Has to be run

2 Oct 30, 2021
Convert any binary data to a PNG image file and vice versa.

What is PngBin? The name PngBin comes from an image format file extension PNG (Portable Network Graphics) and the word Binary. An image produced by Pn

Nathan Young 87 Dec 22, 2022
A minimal python script for generating bip39 seed phrases, and corresponding Seed Signer Seed seed phrase qr code ready for offline printing.

A minimal python script for generating bip39 seed phrases, and corresponding Seed Signer Seed seed phrase qr code ready for offline printing.

CypherToad 8 Sep 12, 2022
Plots is a graph plotting app for GNOME.

Plots is a graph plotting app for GNOME. Plots makes it easy to visualise mathematical formulae. In addition to basic arithmetic operations, it supports trigonometric, hyperbolic, exponential and log

Alex Huntley 138 Dec 14, 2022
A QR Code encode and decode python module

A QR Code encode and decode python module

Fayas Noushad 4 Feb 10, 2022
Fast Image Retrieval (FIRe) is an open source image retrieval project

Fast Image Retrieval (FIRe) is an open source image retrieval project release by Center of Image and Signal Processing Lab (CISiP Lab), Universiti Malaya. This project implements most of the major bi

CISiP Lab 39 Nov 25, 2022
This is a new web-based photo management application. Run it on your home server and it will let you find the right photo from your collection on any device. Smart filtering is made possible by object recognition, location awareness, color analysis and other ML algorithms.

Photonix Photo Manager This is a photo management application based on web technologies. Run it on your home server and it will let you find what you

Damian Moore 1.5k Jan 01, 2023
LGVL helper script to batch and convert with lvgl offline image converter

script to batch and convert with lvgl offline image converter

Yohann 1 Oct 05, 2022
An executor that wraps 3D mesh models and encodes 3D content documents to d-dimension vector.

3D Mesh Encoder An Executor that receives Documents containing point sets data in its blob attribute, with shape (N, 3) and encodes it to embeddings o

Jina AI 11 Dec 14, 2022