Offical code for the paper: "Growing 3D Artefacts and Functional Machines with Neural Cellular Automata" https://arxiv.org/abs/2103.08737

Last update: Jan 01, 2023

Related tags

Deep Learning 3d-artefacts-nca

Overview

Growing 3D Artefacts and Functional Machines with Neural Cellular Automata

Video of more results: https://www.youtube.com/watch?v=-EzztzKoPeo

Requirements

python3.8
This package automatically install test-evocraft-py (https://github.com/shyamsn97/test-evocraft-py), but for further functionality follow installation steps here: https://github.com/real-itu/Evocraft-py

Installation

For general installation

python setup.py install

For ray tune + mlflow

python -m pip install -r ray-requirements.txt
python setup.py install

Usage

Make sure an evocraft-py server is running, either with test-evocraft-py --interactive or by following the steps in https://github.com/real-itu/Evocraft-py.

Configs

Each nca is trained on a specific structure w/ hyperparams and configurations defined in yaml config, which we use with hydra to create the NCA trainer class.

Example Config for generating a "PlainBlacksmith" Minecraft Structure:

trainer:
    name: PlainBlacksmith
    min_steps: 48
    max_steps: 64
    visualize_output: true
    device_id: 0
    use_cuda: true
    num_hidden_channels: 10
    epochs: 20000
    batch_size: 5
    model_config:
        normal_std: 0.1
        update_net_channel_dims: [32, 32]
    optimizer_config:
        lr: 0.002
    dataset_config:
        nbt_path: artefact_nca/data/structs_dataset/nbts/village/plain_village_blacksmith.nbt

defaults:
  - voxel

Generation and Training

See generation notebook for ways to load in a pretrained nca and generate a structure in minecraft

See training notebook for ways to train an nca

CLI training

python artefact_nca/train.py config={path to yaml config} trainer.dataset_config.nbt_path={absolute path to nbt file to use}

Example:

python artefact_nca/train.py config=pretrained_models/PlainBlacksmith/plain_blacksmith.yaml trainer.dataset_config.nbt_path=/home/shyam/Code/3d-artefacts-nca/artefact_nca/data/structs_dataset/nbts/village/plain_village_blacksmith.nbt

Spawning in minecraft

See generation notebook for more details

Example spawning the oak tree

Load in a trainer

from artefact_nca.trainer.voxel_ca_trainer import VoxelCATrainer

nbt_path = {path to repo}/artefact_nca/data/structs_dataset/nbts/village/Extra_dark_oak.nbt
ct = VoxelCATrainer.from_config(
                    "{path to repo}/pretrained_models/Extra_dark_oak/extra_dark_oak.yaml",
                    config={
                        "pretrained_path":"{path to repo}/pretrained_models/Extra_dark_oak/Extra_dark_oak.pt",
                        "dataset_config":{"nbt_path":nbt_path},
                        "use_cuda":False
                    }
                )

Create MinecraftClient to view the growth of the structure in Minecraft at position (-10, 10, 10) (x, y, z)

from artefact_nca.utils.minecraft import MinecraftClient
m = MinecraftClient(ct, (-10, 10, 10))

Spawn 100 iterations and display progress every 5 time steps

m.spawn(100)

Output should look like this:

Structures

see data directory. To view structures and spawn in minecraft see generation notebook. An example of spawning and viewing the Tree:

import matplotlib.pyplot as plt
from artefact_nca.utils.minecraft import MinecraftClient

base_nbt_path = {path to nbts}
nbt_path = "{}/village/Extra_dark_oak.nbt".format(base_nbt_path)

 # spawn at coords (50, 10, 10)
blocks, unique_vals, target, color_dict, unique_val_dict = MinecraftClient.load_entity("Extra_dark_oak", nbt_path=nbt_path, load_coord=(50,10,10))

color_arr = convert_to_color(target, color_dict)

fig = plt.figure()
ax = fig.gca(projection='3d')
ax.voxels(color_arr, facecolors=color_arr, edgecolor='k')

plt.show()

This should spawn and display:

Authors

Shyam Sudhakaran [email protected], https://github.com/shyamsn97

Djordje Grbic [email protected], https://github.com/djole

Siyan Li [email protected], https://github.com/sli613

Adam Katona [email protected], https://github.com/adam-katona

Elias Najarro https://github.com/enajx

Claire Glanois https://github.com/claireaoi

Sebastian Risi [email protected], https://github.com/sebastianrisi

Citation

If you use the code for academic or commecial use, please cite the associated paper:

@inproceedings{Sudhakaran2021,
   title = {Growing 3D Artefacts and Functional Machines with Neural Cellular Automata}, 
   author = {Shyam Sudhakaran and Djordje Grbic and Siyan Li and Adam Katona and Elias Najarro and Claire Glanois and Sebastian Risi},
   booktitle = {2021 Conference on Artificial Life},
   year = {2021},
   url = {https://arxiv.org/abs/2103.08737}
}

Offical code for the paper: "Growing 3D Artefacts and Functional Machines with Neural Cellular Automata" https://arxiv.org/abs/2103.08737

Related tags

Overview

Growing 3D Artefacts and Functional Machines with Neural Cellular Automata

Video of more results: https://www.youtube.com/watch?v=-EzztzKoPeo

Requirements

Installation

For general installation

For ray tune + mlflow

Usage

Configs

Generation and Training

See generation notebook for ways to load in a pretrained nca and generate a structure in minecraft

See training notebook for ways to train an nca

CLI training

Spawning in minecraft

Example spawning the oak tree

Structures

Authors

Citation

Owner

Robotics Evolution and Art Lab

Generative Exploration and Exploitation - This is an improved version of GENE.

Code for Mesh Convolution Using a Learned Kernel Basis

Randstad Artificial Intelligence Challenge (powered by VGEN). Soluzione proposta da Stefano Fiorucci (anakin87) - primo classificato

Code for Multiple Instance Active Learning for Object Detection, CVPR 2021

PFLD pytorch Implementation

Code for paper " AdderNet: Do We Really Need Multiplications in Deep Learning?"

Chinese license plate recognition

Lbl2Vec learns jointly embedded label, document and word vectors to retrieve documents with predefined topics from an unlabeled document corpus.

Visual dialog agents with pre-trained vision-and-language encoders.

Source code for NAACL 2021 paper "TR-BERT: Dynamic Token Reduction for Accelerating BERT Inference"

Generalized Random Forests

The 2nd place solution of 2021 google landmark retrieval on kaggle.

Res2Net for Instance segmentation and Object detection using MaskRCNN

BDDM: Bilateral Denoising Diffusion Models for Fast and High-Quality Speech Synthesis

Code for ICCV 2021 paper Graph-to-3D: End-to-End Generation and Manipulation of 3D Scenes using Scene Graphs

[CVPR'22] Official PyTorch Implementation of Collaborative Transformers for Grounded Situation Recognition

List of content farm sites like g.penzai.com.

Graph Transformer Architecture. Source code for

Turning SymPy expressions into PyTorch modules.

Source codes for Improved Few-Shot Visual Classification (CVPR 2020), Enhancing Few-Shot Image Classification with Unlabelled Examples