Multi View Stereo on Internet Images

Overview

Evaluating MVS in a CPC Scenario

This repository contains the set of artficats used for the ENGN8601/8602 research project. The thesis emphasizes on the following aspects:

  • Evaluating and Analysing the performance of existing learning-based MVS networks on internet images or in a CPC scenario.
  • Proposing a novel mask estimation module and depth estimation (with depth alignment) framework to estimate depth values of foreground objects.
  • Fusing the depthmaps estimated by the proposed methodology to compute complete point clouds (including foreground objects)

Installation

Recommended: python 3.7.7, cudatoolkit 10.2.* and conda.

The python libraries required are provided in the requirements.txt file. You can install the environment and necessary modules as follows or use your own approach:

Create a new conda environment and activate it:

conda create -n mvs
conda activate mvs

Install requirements.txt and opencv & pytorch separately (make sure pip is installed):

pip install -r requirements.txt
pip install opencv-python
pip install torch==1.8.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html

Data

Download the validation dataset extracted from MegaDepth dataset here and extract the all the images from the dataset to ref_images folder as subdirectories (create ref_images if it doesn't exist).

Usage

All the results have been included in the downloaded dataset already for ease of access to the results. The directory structure is given and explained as follows. The image subdirectories include reference images, npy files containing camera information, entropies, depthmaps (monocular, estimated and ground truth) etc.

  • All the images inside the ref_images directory contain a grid_outputs subdirectory which contain the best masks estimated for the reference image. This subdirectory also contains the plots with the .npy files for visualization.
  • All the images inside the ref_images directory contain ply and ply_gt subdirectories, which contain the 3D world points and .ply files depicting the estimated point cloud of that scene reconstructed by the proposed method.
  • The final_fused_scenes folder contains the point cloud generated by fusing multiple depthmaps obtained from different views of the same scene.

NOTE: You do not need to run the following steps in a sequence since required intermediate results are already provided in the images (such as entropy maps etc.), you can run any step directly.

1. Mask Estimation

Open the terminal and run the following command:

python masking.py

The 10 best masks with lowest binary cross entropy loss and IoU for each reference image is computed and saved inside a grid_outputs subdirectory inside each image directory. You can view the mask visualizations which are saved as .png. The masks are also saved as .npy files.

2. Calculating Error Metrics

To calculate the EPE, 1px and 3px errors between the estimated depthmaps and ground truth depthmaps, run the following command:

python calc_errors.py

3. 3D Reconstruction - Individual Point Clouds

To reconstruct ground truth point clouds and the estimated point clouds with foreground objects for each individual reference image, run the following command:

python pfm2ply_aligned.py                   # Point Clouds from Estimated Depthmaps (with foreground)
python pfm2ply_aligned_gt.py                # Point Clouds from Ground Truth Depthmaps

The point clouds will be saved in the ply and ply_gt image subdirectories respectively as .ply files along with the vertices of these point clouds saved as vertices.npy. This also generates the aligned absolute depthmap and saves the visualization along with the monocular depthmap estimated via the monocular depth estimation network inside the image directories.

4. Generating Scene Point Clouds

Since step 3 comptues individual point clouds, the next task is to merge the vertices of each individual point cloud to generate the point cloud for an entire scene. Run the following command:

python merge2ply.py

You can specify the set of images to be used for reconstructing each scene by editing the merge2ply file. All the scene point clouds are saved in final_fused_scenes folder.

Visualizations of Outputs

1. Mask Estimation

masks

2. 3D Reconstruction of Invidiual Depthmaps

3dr

3. Merged Point Clouds

3dr scenes

Supporting Repositories

I would like to give credit to the following repositories for assisting me in computing intermediate results necessary for the thesis:

Thank you!

Owner
Namas Bhandari
Machine Learning/Deep Learning/AI Enthusiast
Namas Bhandari
Awesome Cheatsheet

Awesome Cheatsheet List of useful cheatsheets Inspired by @sindresorhus awesome and improved by these amazing contributors. If you see a link here is

detailyang 6.5k Jan 07, 2023
Python Control Systems Library

The Python Control Systems Library is a Python module that implements basic operations for analysis and design of feedback control systems.

Control Systems Library for Python 1.3k Jan 06, 2023
Rotazioni: a linear programming workout split optimizer

Rotazioni: a linear programming workout split optimizer Dependencies Dependencies for the frontend and backend are respectively listed in client/packa

Marco 3 Oct 13, 2022
A bot to view Dilbert comics directly from Discord and get updates of the comics automatically.

A bot to view Dilbert comics directly from Discord and get updates of the comics automatically

Raghav Sharma 3 Nov 30, 2022
Render to print for blender 2.9+

render_to_print_blender_addon ** render2print: Blender AddOn for Blender 2.90.0+ ** Calculates camera parameters to allow printing a rendered image to

5 Nov 19, 2021
A quick experiment to demonstrate Metamath formula parsing, where the grammar is embedded in a few additional 'syntax axioms'.

Warning: Hacked-up code ahead. (But it seems to work...) What it does This demonstrates an idea which I posted about several times on the Metamath mai

Marnix Klooster 1 Oct 21, 2021
A Python script to parse Fortinet products serial numbers, and detect the associated model and version.

ParseFortinetSerialNumber A Python script to parse Fortinet products serial numbers, and detect the associated model and version. Example $ ./ParseFor

Podalirius 10 Oct 28, 2022
This is a calculator of strike price distance for options.

Calculator-of-strike-price-distance-for-options This is a calculator of strike price distance for options. Options are a type of derivative. One strat

André Luís Lopes da Silva 4 Dec 30, 2022
Inverted-pendulum-with-fuzzy-control - Inverted pendulum with fuzzy control

Fuzzy Inverted Pendulum Basically, this project consists of an inverted pendulum

Mahan Ahmadvand 1 Aug 25, 2022
This is Gaurav's IP Project Completed in the year session of 2021-2022.

The Analyser by Gaurav Rayat Why this Project? Today we are continuously hearing about growth in Crime rates and the number of murders executed day by

1 Dec 30, 2021
Check a discord message and give it a percentage of scamminess

scamChecker Check a discord message and give it a percentage of scamminess Run the bot, and run the command !scamCheck and it will return a percentage

3 Sep 22, 2022
Open source book about making Python packages.

Python packages Tomas Beuzen & Tiffany Timbers Python packages are a core element of the Python programming language and are how you create organized,

Python Packages 169 Jan 06, 2023
A Lynx that manages a group that puts the federation first.

Lynx Super Federation Management Group Lynx was created to manage your groups on telegram and focuses on the Lynx Federation. I made this to root out

Unknown 2 Nov 01, 2022
This is a simple SV calling package for diploid assemblies.

dipdiff This is a simple SV calling package for diploid assemblies. It uses a modified version of svim-asm. The package includes its own version minim

Mikhail Kolmogorov 11 Jan 05, 2023
A simple solution for water overflow problem in Python

Water Overflow problem There is a stack of water glasses in a form of triangle as illustrated. Each glass has a 250ml capacity. When a liquid is poure

Kris 2 Oct 22, 2021
scap is a tool for putting code in places and for other purposes

Scap is the deployment script used by Wikimedia Foundation to publish code and configuration on production web servers.

Wikimedia 7 Nov 02, 2022
A type based dependency injection framework for Python 3.9+

Alluka A type based dependency injection framework for Python 3.9+. Installation You can install Alluka from PyPI using the following command in any P

Lucina 16 Dec 15, 2022
Daily knowledge pills to get better in Python.

Python daily pills Daily knowledge pills to get better Python code. Why Does your Python code suffers of any of this symptoms? Incorrect Indentation I

Jeferson Vaz dos Santos 35 Sep 19, 2022
An osu! cheat made in c++ rewritten in python and currently undetected.

megumi-python An osu! cheat made in c++ rewritten in python and currently undetected. Installation Guide Download python 3.9 from https://python.org C

Elaina 2 Nov 18, 2022
Learn the basics of Python. These tutorials are for Python beginners. so even if you have no prior knowledge of Python, you won’t face any difficulty understanding these tutorials.

01_Python_Introduction Introduction 👋 Python is a modern, robust, high level programming language. It is very easy to pick up even if you are complet

Milaan Parmar / Милан пармар / _米兰 帕尔马 245 Dec 30, 2022