Differentiable rasterization applied to 3D model simplification tasks

Related tags

Deep Learningnvdiffmodeling
Overview

nvdiffmodeling

Teaser image

Differentiable rasterization applied to 3D model simplification tasks, as described in the paper:

Appearance-Driven Automatic 3D Model Simplification
Jon Hasselgren, Jacob Munkberg, Jaakko Lehtinen, Miika Aittala and Samuli Laine
https://research.nvidia.com/publication/2021-04_Appearance-Driven-Automatic-3D
https://arxiv.org/abs/2104.03989

License

Copyright © 2021, NVIDIA Corporation. All rights reserved.

This work is made available under the Nvidia Source Code License.

For business inquiries, please contact [email protected]

Installation

Requires VS2019+, Cuda 10.2+ and PyTorch 1.6+

Tested in Anaconda3 with Python 3.6 and PyTorch 1.8

One time setup (Windows)

Install the Cuda toolkit (required to build the PyTorch extensions). We support Cuda 10.2 and above. Pick the appropriate version of PyTorch compatible with the installed Cuda toolkit. Below is an example with Cuda 11.1

conda create -n dmodel python=3.6
activate dmodel
conda install pytorch torchvision torchaudio cudatoolkit=11.1 -c pytorch -c conda-forge
conda install imageio
pip install PyOpenGL glfw

Install nvdiffrast: https://github.com/NVlabs/nvdiffrast in the dmodel conda env.

Every new command prompt

activate dmodel

Examples

Sphere to cow example:

python train.py --config configs/spot.json

The results will be stored in the out folder. The Spot model was created and released into the public domain by Keenan Crane.

Additional assets can be downloaded here [205MB]. Unzip and place the subfolders in the project data folder, e.g., data\skull. All assets are copyright of their respective authors, see included license files for further details.

Included examples

  • skull.json - Joint normal map and shape optimization on a skull
  • ewer.json - Ewer model from a reduced mesh as initial guess
  • gardenina.json - Aggregate geometry example
  • hibiscus.json - Aggregate geometry example
  • figure_brushed_gold_64.json - LOD example, trained against a supersampled reference
  • figure_displacement.json - Joint shape, normal map, and displacement map example

The json files that end in _paper.json are configs with the settings used for the results in the paper. They take longer and require a GPU with sufficient memory.

Server usage (through Docker)

  • Build docker image (run the command from the code root folder). docker build -f docker/Dockerfile -t diffmod:v1 . Requires a driver that supports Cuda 10.1 or newer.

  • Start an interactive docker container: docker run --gpus device=0 -it --rm -v /raid:/raid -it diffmod:v1 bash

  • Detached docker: docker run --gpus device=1 -d -v /raid:/raid -w=[path to the code] diffmod:v1 python train.py --config configs/spot.json

Comments
  • Proposition for more comprehensive Readme

    Proposition for more comprehensive Readme

    First of all, thank you for making this freely available for everyone! This is really a great move!

    I wanted to suggest a few things for the readme: 1.) Add "Microsoft Visual C++" to the requirements. It should be clear that you have to tick the C++ development environment while installing VS2019.

    2.) Install Git via "conda install git" (This makes it easier to install nvdiffrast and clone the repo)

    3.) Change "Install NVDiffRast" to something more descriptive: pip install -e git+https://github.com/NVlabs/[email protected]#egg=nvdiffrast

    4.) Add another step to the installation, detailing to move to a desired folder via the Anaconda Prompt and cloning this repo: cd YOUR/PATH/HERE git clone https://github.com/NVlabs/nvdiffmodeling.git and afterwards to move into the folder cd nvdiffmodeling

    5.) Installing CUDA via Anaconda only deploys the bare essential files. The CUDA path is not set when doing it this way. So I downloaded the CUDA toolkit from nvidia and installed it afterwards, to fix that.

    I am sure that all of these things are obvious for professionals, but it took me a while to figure all of this out, so I would like to save other people's time.

    opened by morph3us-net 7
  • About the raymarch fitting case

    About the raymarch fitting case

    Thanks for the marvelous work! I have some questions related to the shadertoy fitting case: It seems figure 22 & figure 23 use some special uv un-wrap methods, can you elaborate the algorithm under the hood? image

    opened by outlawever 4
  • RuntimeError: Error building extension 'renderutils_plugin'

    RuntimeError: Error building extension 'renderutils_plugin'

    I am trying to get this to work to test the workflow but ran into the error I have installed "Cuda" 11.1 and everything else as in the tutorial Graphics accelerator 3080t

    After trying to run a test command, it gives me the following text Do you have any ideas how to fix this? I am more of a technical artist and a cursory search did not return much

    
(dmodel) I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main>python train.py --config configs/spot.json
Using C:\Users\alexp\AppData\Local\torch_extensions\torch_extensions\Cache as PyTorch extensions root...
C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py:305: UserWarning: Error checking compiler version for cl: 'utf-8' codec can't decode byte 0x8e in position 0: invalid start byte
  warnings.warn(f'Error checking compiler version for {compiler}: {error}')
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\alexp\AppData\Local\torch_extensions\torch_extensions\Cache\renderutils_plugin\build.ninja...
Building extension module renderutils_plugin...
Allowing ninja to set a default number of workers... (overridable by setting the environment variable MAX_JOBS=N)
[1/7] cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\common.cpp /Focommon.o
FAILED: common.o
cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\common.cpp /Focommon.o
Оптимизирующий компилятор Microsoft (R) C/C++ версии 19.25.28611 для x64
(C) Корпорация Майкрософт (Microsoft Corporation).  Все права защищены.

Примечание: включение файла:  C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\cuda_runtime.h
Примечание: включение файла:   C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_config.h
Примечание: включение файла:   C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\builtin_types.h
Примечание: включение файла:    C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\device_types.h
Примечание: включение файла:     C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_defines.h
Примечание: включение файла:    C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\driver_types.h
Примечание: включение файла:     C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_defines.h
Примечание: включение файла:     C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\vector_types.h
Примечание: включение файла:      C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\crt/host_defines.h
Примечание: включение файла:     C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\limits.h
Примечание: включение файла:      C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vcruntime.h
Примечание: включение файла:       C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\sal.h
Примечание: включение файла:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\concurrencysal.h
Примечание: включение файла:       C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vadefs.h
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include\driver_types.h(78): fatal error C1083: Не удается открыть файл включение: stddef.h: No such file or directory,
[2/7] cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\torch_bindings.cpp /Fotorch_bindings.o
FAILED: torch_bindings.o
cl /showIncludes -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 /MD /wd4819 /wd4251 /wd4244 /wd4267 /wd4275 /wd4018 /wd4190 /EHsc -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\torch_bindings.cpp /Fotorch_bindings.o
Оптимизирующий компилятор Microsoft (R) C/C++ версии 19.25.28611 для x64
(C) Корпорация Майкрософт (Microsoft Corporation).  Все права защищены.

Примечание: включение файла:  C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch/extension.h
Примечание: включение файла:   C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include\torch/all.h
Примечание: включение файла:    C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include\torch/cuda.h
Примечание: включение файла:     C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch/csrc/WindowsTorchApiMacro.h
Примечание: включение файла:      C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\c10/macros/Export.h
Примечание: включение файла:       C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\c10/macros/cmake_macros.h
Примечание: включение файла:     C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\cstdint
Примечание: включение файла:      C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\yvals.h
Примечание: включение файла:       C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\yvals_core.h
Примечание: включение файла:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vcruntime.h
Примечание: включение файла:         C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\sal.h
Примечание: включение файла:          C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\concurrencysal.h
Примечание: включение файла:         C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\vadefs.h
Примечание: включение файла:        C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\xkeycheck.h
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\yvals.h(12): fatal error C1083: Не удается открыть файл включение: crtdbg.h: No such file or directory,
[3/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output mesh.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\mesh.cu -o mesh.cuda.o
FAILED: mesh.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output mesh.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\mesh.cu -o mesh.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
mesh.cu
[4/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output loss.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\loss.cu -o loss.cuda.o
FAILED: loss.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output loss.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\loss.cu -o loss.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
loss.cu
[5/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output bsdf.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\bsdf.cu -o bsdf.cuda.o
FAILED: bsdf.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output bsdf.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\bsdf.cu -o bsdf.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
bsdf.cu
[6/7] C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output normal.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\normal.cu -o normal.cuda.o
FAILED: normal.cuda.o
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\bin\nvcc --generate-dependencies-with-compile --dependency-output normal.cuda.o.d -Xcudafe --diag_suppress=dll_interface_conflict_dllexport_assumed -Xcudafe --diag_suppress=dll_interface_conflict_none_assumed -Xcudafe --diag_suppress=field_without_dll_interface -Xcudafe --diag_suppress=base_class_has_different_dll_interface -Xcompiler /EHsc -Xcompiler /wd4190 -Xcompiler /wd4018 -Xcompiler /wd4275 -Xcompiler /wd4267 -Xcompiler /wd4244 -Xcompiler /wd4251 -Xcompiler /wd4819 -Xcompiler /MD -DTORCH_EXTENSION_NAME=renderutils_plugin -DTORCH_API_INCLUDE_EXTENSION_H -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\torch\csrc\api\include -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\TH -IC:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\include\THC "-IC:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.1\include" -IC:\ProgramData\Anaconda3\envs\dmodel\Include -D_GLIBCXX_USE_CXX11_ABI=0 -D__CUDA_NO_HALF_OPERATORS__ -D__CUDA_NO_HALF_CONVERSIONS__ -D__CUDA_NO_BFLOAT16_CONVERSIONS__ -D__CUDA_NO_HALF2_OPERATORS__ --expt-relaxed-constexpr -gencode=arch=compute_86,code=compute_86 -gencode=arch=compute_86,code=sm_86 -c I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\c_src\normal.cu -o normal.cuda.o
C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\VC\Tools\MSVC\14.25.28610\include\crtdefs.h(10): fatal error C1083: ═х єфрхЄё  юЄъЁ√Є№ Їрщы тъы■ўхэшх: corecrt.h: No such file or directory,
normal.cu
ninja: build stopped: subcommand failed.
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1672, in _run_ninja_build
    env=env)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\subprocess.py", line 438, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['ninja', '-v']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "train.py", line 20, in <module>
    import src.renderutils as ru
  File "I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\__init__.py", line 9, in <module>
    from .ops import xfm_points, xfm_vectors, image_loss, prepare_shading_normal, lambert, pbr_specular, pbr_bsdf, _fresnel_shlick, _ndf_ggx, _lambda_ggx, _masking_smith
  File "I:\Python\Nvidia_Lod\nvdiffmodeling-main\nvdiffmodeling-main\src\renderutils\ops.py", line 61, in <module>
    torch.utils.cpp_extension.load(name='renderutils_plugin', sources=source_paths, extra_ldflags=ldflags, with_cuda=True, verbose=True)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1092, in load
    keep_intermediates=keep_intermediates)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1303, in _jit_compile
    is_standalone=is_standalone)
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1408, in _write_ninja_file_and_build_library
    error_prefix=f"Error building extension '{name}'")
  File "C:\ProgramData\Anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1682, in _run_ninja_build
    raise RuntimeError(message) from e
RuntimeError: Error building extension 'renderutils_plugin'
    opened by TheAlexPG 4
  • about effect Fig.22 and Fig 23

    about effect Fig.22 and Fig 23

    I read your paper and video in detail，about LEARNING MESH AND MATERIALS FROM IMPLICIT SURFACES,How to improve the effect？ Can you give me some suggestions on setting super parameters? I used my own model as the target, started from the sphere you provided, and carried out experiments. I found that the feet and face were always unsatisfactory? There is a big gap with the rendering of snails and elephants mentioned in your paper. can you give me some adivce. image

    opened by Lucklycat 2
  • Texture maps have a lot of noise

    Texture maps have a lot of noise

    Thank you very much for your work！ When I tried the ewer example, I found the following textures： image When you zoom in, you will find many red, yellow and green noises image This is still the case when I import the output obj into blender, but the contrast image rendered from nvdiffast does not show this situation. What is the reason? image

    opened by Lucklycat 2
  • How to improve the optimization effect

    How to improve the optimization effect

    Your work is great. Thank you very much for your openness. I tried to use rendering on my own model, but I encountered the following problems and would like to consult: 1.In the process of rendering, I found that the face of the character, especially the eyes, was not poorly optimized, and the color could not be reflected. Are there any parameters that need to be adjusted? image 2.During rendering, I encountered that. mtl contains a map_ The texture of d cannot be represented in the renderer. How can I solve this problem? image

    Thank you for your reply

    opened by Lucklycat 2
  • Lower training resolution => heavier texture pixelation. But why?

    Lower training resolution => heavier texture pixelation. But why?

    Hi, and thanks for great paper and comprehensive code! Could you please help me understand one thing:

    I'm running configs/dancer_displacement.json. Despite "texture_res" being 2048, these 2048 x 2048 textures "learn" huge pixels at lower "train_res":

    image

    And this isn't just poor interpolation in Blender, but these are actual large solid blocks (of different scale!) in normal_map_opt and kd_map_opt! Here is a part of texture_n.png from the top left experiment, 1:1 scale:

    image

    Why do these maps get more pixelated instead of getting more blurry? I've spent couple days searching the code for the source of this behaviour, and I believe there are only three relevant lines (for now let's consider texture only):

    Interpolation doesn't use 'nearest' method so it shouldn't be the source of big pixels. And because views are always different, rasterization should be a problem too. But probably I'm just missing some rendering subtleties?

    Thanks again!

    opened by shrubb 2
  • msvcrt.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __enclave_config

    msvcrt.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __enclave_config

    hi, Thanks for your project. When I try to run sphere to cow demo, I will report this error: msvcrt.lib(loadcfg.obj) : error LNK2001: unresolved external symbol __enclave_config renderutils_plugin.pyd : fatal error LNK1120: Unresolved external command ninja: build stopped: subcommand failed.

    Is it because of my Nanjia version? my current version is: nanjia:1.10.2 pytorch:1.8.1 torchvision:0.9.1 cudnn:7.6.5 cuda:10.2

    opened by myshiop 2
  • matrix gradients does not flow back when using renderutils xfm_points

    matrix gradients does not flow back when using renderutils xfm_points

    hi, when using renderutils.xfm_points (e.g. in render.py v_pos_clip = ru.xfm_points(mesh.v_pos[None, ...], mtx_in) - and trying to optimize camera pose in mtx_in - gradients does not flow back. Only when using use_python=True it worked.

    opened by maorp 1
  • using camera calibration matrices

    using camera calibration matrices

    Hi, first thanks for sharing this work! I'm trying to solve inverse rendering problem using a photogrammetry dataset which comes with accurate camera calibration mats. i have started from the spot example and i need to convert my intrinsic\extrinsic params to be used in the rastering and cant find an example or a guide to do it. the only hint is that I know it should be converted to openGL and NDC according to nvdiffrast, ill be happy to get an advice, thanks.

    opened by maorp 1
  • some question about the complex material, such as that ray marching snail

    some question about the complex material, such as that ray marching snail

    Hi, Thank you very much for your research. I'm trying to use your framework to approximate complex materials, but the blender complex material dataset I made with my own is not very good, because I encountered many problems with camera and light settings in my own rendering, Is the ray marching snail used in this paper directly rendered with shadertoy, or is the ray marching process written in Python?

    opened by myshiop 1
  • where can i find the Animation and Skinning demo

    where can i find the Animation and Skinning demo

    hi, i am read the papaer--Appearance-Driven Automatic 3D Model Simplification, think it's great. But in the demo,i can not find the Animation and Skinning demo, where can i find it ? I want to study how to do it. please help ! thanks

    opened by Lucklycat 2
  • CUDA out of memory.

    CUDA out of memory.

    RuntimeError: CUDA out of memory. Tried to allocate 130.00 MiB (GPU 0; 8.00 GiB total capacity; 7.11 GiB already allocated; 0 bytes free; 7.30 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

    opened by d-sharafeldeen 2
  • How to test on custom .obj file?

    How to test on custom .obj file?

    I am trying to test model simplification on my own .obj. I have created a .mtl file for the .obj but I got an error. How to export .obj file to train the model?

    opened by sanbuddhacharyas 0
  • Unable to execute the program, error during cpp_extension.py running at start

    Unable to execute the program, error during cpp_extension.py running at start 

    (dmodel) D:\NVIDIATools\nvdiffmodeling-main>python train.py --config configs/spot.json
No CUDA runtime is found, using CUDA_HOME='C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.5'
Using C:\Users\redacted\AppData\Local\torch_extensions\torch_extensions\Cache\py36_cpu as PyTorch extensions root...
Detected CUDA files, patching ldflags
Emitting ninja build file C:\Users\redacted\AppData\Local\torch_extensions\torch_extensions\Cache\py36_cpu\renderutils_plugin\build.ninja...
Traceback (most recent call last):
  File "train.py", line 20, in <module>
    import src.renderutils as ru
  File "D:\NVIDIATools\nvdiffmodeling-main\src\renderutils\__init__.py", line 9, in <module>
    from .ops import xfm_points, xfm_vectors, image_loss, prepare_shading_normal, lambert, pbr_specular, pbr_bsdf, _fresnel_shlick, _ndf_ggx, _lambda_ggx, _masking_smith
  File "D:\NVIDIATools\nvdiffmodeling-main\src\renderutils\ops.py", line 61, in <module>
    torch.utils.cpp_extension.load(name='renderutils_plugin', sources=source_paths, extra_ldflags=ldflags, with_cuda=True, verbose=True)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1136, in load
    keep_intermediates=keep_intermediates)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1347, in _jit_compile
    is_standalone=is_standalone)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1445, in _write_ninja_file_and_build_library
    is_standalone=is_standalone)
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1834, in _write_ninja_file_to_build_library
    cuda_flags = common_cflags + COMMON_NVCC_FLAGS + _get_cuda_arch_flags()
  File "C:\Users\redacted\anaconda3\envs\dmodel\lib\site-packages\torch\utils\cpp_extension.py", line 1606, in _get_cuda_arch_flags
    arch_list[-1] += '+PTX'
IndexError: list index out of range

(dmodel) D:\NVIDIATools\nvdiffmodeling-main>

    After installing the prerequisites, only difference to the instruction is CUDA 11.5 (and i changed the version to the pip install line to that same version) Getting the above when attempting to run the example spot cow training.

    Something i'm doing wrong, incompatibility with versions or something else entirely?

    opened by RiversJohn 3
Releases(v1.0)
Owner
NVIDIA Research Projects
NVIDIA Research Projects
GitHub Repository
Classification models 1D Zoo - Keras and TF.Keras

Classification models 1D Zoo - Keras and TF.Keras This repository contains 1D variants of popular CNN models for classification like ResNets, DenseNet

Roman Solovyev 12 Jan 06, 2023
Continual World is a benchmark for continual reinforcement learning

Continual World Continual World is a benchmark for continual reinforcement learning. It contains realistic robotic tasks which come from MetaWorld. Th

41 Dec 24, 2022
The official code repo of "HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound Classification and Detection"

Hierarchical Token Semantic Audio Transformer Introduction The Code Repository for "HTS-AT: A Hierarchical Token-Semantic Audio Transformer for Sound

Knut(Ke) Chen 134 Jan 01, 2023
交互式标注软件，暂定名 iann

iann 交互式标注软件，暂定名iann。 安装 按照官网介绍安装paddle。 安装其他依赖 pip install -r requirements.txt 运行 git clone https://github.com/PaddleCV-SIG/iann/ cd iann python iann

294 Dec 30, 2022
Improving Query Representations for DenseRetrieval with Pseudo Relevance Feedback:A Reproducibility Study.

APR The repo for the paper Improving Query Representations for DenseRetrieval with Pseudo Relevance Feedback:A Reproducibility Study. Environment setu

ielab 8 Nov 26, 2022
[CVPR'21] MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation

MonoRUn MonoRUn: Monocular 3D Object Detection by Reconstruction and Uncertainty Propagation. CVPR 2021. [paper] Hansheng Chen, Yuyao Huang, Wei Tian*

同济大学智能汽车研究所综合感知研究组 ( Comprehensive Perception Research Group under Institute of Intelligent Vehicles, School of Automotive Studies, Tongji University) 96 Dec 10, 2022
SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruning

SHRIMP: Sparser Random Feature Models via Iterative Magnitude Pruning This repository is the official implementation of "SHRIMP: Sparser Random Featur

Bobby Shi 0 Dec 16, 2021
Try out deep learning models online on Google Colab

Try out deep learning models online on Google Colab

Erdene-Ochir Tuguldur 1.5k Dec 27, 2022
Video2x - A lossless video/GIF/image upscaler achieved with waifu2x, Anime4K, SRMD and RealSR.

Official Discussion Group (Telegram): https://t.me/video2x A Discord server is also available. Please note that most developers are only on Telegram.

K4YT3X 5.9k Dec 31, 2022
PyTorch code for SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised DA

PyTorch Code for SENTRY: Selective Entropy Optimization via Committee Consistency for Unsupervised Domain Adaptation Viraj Prabhu, Shivam Khare, Deeks

Viraj Prabhu 46 Dec 24, 2022
A Bayesian cognition approach for belief updating of correlation judgement through uncertainty visualizations

Overview Code and supplemental materials for Karduni et al., 2020 IEEE Vis. "A Bayesian cognition approach for belief updating of correlation judgemen

Ryan Wesslen 1 Feb 08, 2022
Keras implementation of "One pixel attack for fooling deep neural networks" using differential evolution on Cifar10 and ImageNet

One Pixel Attack How simple is it to cause a deep neural network to misclassify an image if an attacker is only allowed to modify the color of one pix

Dan Kondratyuk 1.2k Dec 26, 2022
The implementation of ICASSP 2020 paper "Pixel-level self-paced learning for super-resolution"

Pixel-level Self-Paced Learning for Super-Resolution This is an official implementaion of the paper Pixel-level Self-Paced Learning for Super-Resoluti

Elon Lin 41 Dec 15, 2022
NaturalCC is a sequence modeling toolkit that allows researchers and developers to train custom models

NaturalCC NaturalCC is a sequence modeling toolkit that allows researchers and developers to train custom models for many software engineering tasks,

159 Dec 28, 2022
Making a music video with Wav2CLIP and VQGAN-CLIP

music2video Overview A repo for making a music video with Wav2CLIP and VQGAN-CLIP. The base code was derived from VQGAN-CLIP The CLIP embedding for au

Joel Jang | 장요엘 163 Dec 26, 2022
Vertical Federated Principal Component Analysis and Its Kernel Extension on Feature-wise Distributed Data based on Pytorch Framework

VFedPCA+VFedAKPCA This is the official source code for the Paper: Vertical Federated Principal Component Analysis and Its Kernel Extension on Feature-

John 9 Sep 18, 2022
Implementations of orthogonal and semi-orthogonal convolutions in the Fourier domain with applications to adversarial robustness

Orthogonalizing Convolutional Layers with the Cayley Transform This repository contains implementations and source code to reproduce experiments for t

CMU Locus Lab 36 Dec 30, 2022
Automated Melanoma Recognition in Dermoscopy Images via Very Deep Residual Networks

Introduction This repository contains the modified caffe library and network architectures for our paper "Automated Melanoma Recognition in Dermoscopy

Lequan Yu 47 Nov 24, 2022
An implementation of the methods presented in Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Data.

An implementation of the methods presented in Causal-BALD: Deep Bayesian Active Learning of Outcomes to Infer Treatment-Effects from Observational Data.

Andrew Jesson 9 Apr 04, 2022
Efficient Householder transformation in PyTorch

Efficient Householder Transformation in PyTorch This repository implements the Householder transformation algorithm for calculating orthogonal matrice

Anton Obukhov 49 Nov 20, 2022