Multi View Stereo on Internet Images

Overview

Evaluating MVS in a CPC Scenario

This repository contains the set of artficats used for the ENGN8601/8602 research project. The thesis emphasizes on the following aspects:

  • Evaluating and Analysing the performance of existing learning-based MVS networks on internet images or in a CPC scenario.
  • Proposing a novel mask estimation module and depth estimation (with depth alignment) framework to estimate depth values of foreground objects.
  • Fusing the depthmaps estimated by the proposed methodology to compute complete point clouds (including foreground objects)

Installation

Recommended: python 3.7.7, cudatoolkit 10.2.* and conda.

The python libraries required are provided in the requirements.txt file. You can install the environment and necessary modules as follows or use your own approach:

Create a new conda environment and activate it:

conda create -n mvs
conda activate mvs

Install requirements.txt and opencv & pytorch separately (make sure pip is installed):

pip install -r requirements.txt
pip install opencv-python
pip install torch==1.8.0+cu111 -f https://download.pytorch.org/whl/torch_stable.html

Data

Download the validation dataset extracted from MegaDepth dataset here and extract the all the images from the dataset to ref_images folder as subdirectories (create ref_images if it doesn't exist).

Usage

All the results have been included in the downloaded dataset already for ease of access to the results. The directory structure is given and explained as follows. The image subdirectories include reference images, npy files containing camera information, entropies, depthmaps (monocular, estimated and ground truth) etc.

  • All the images inside the ref_images directory contain a grid_outputs subdirectory which contain the best masks estimated for the reference image. This subdirectory also contains the plots with the .npy files for visualization.
  • All the images inside the ref_images directory contain ply and ply_gt subdirectories, which contain the 3D world points and .ply files depicting the estimated point cloud of that scene reconstructed by the proposed method.
  • The final_fused_scenes folder contains the point cloud generated by fusing multiple depthmaps obtained from different views of the same scene.

NOTE: You do not need to run the following steps in a sequence since required intermediate results are already provided in the images (such as entropy maps etc.), you can run any step directly.

1. Mask Estimation

Open the terminal and run the following command:

python masking.py

The 10 best masks with lowest binary cross entropy loss and IoU for each reference image is computed and saved inside a grid_outputs subdirectory inside each image directory. You can view the mask visualizations which are saved as .png. The masks are also saved as .npy files.

2. Calculating Error Metrics

To calculate the EPE, 1px and 3px errors between the estimated depthmaps and ground truth depthmaps, run the following command:

python calc_errors.py

3. 3D Reconstruction - Individual Point Clouds

To reconstruct ground truth point clouds and the estimated point clouds with foreground objects for each individual reference image, run the following command:

python pfm2ply_aligned.py                   # Point Clouds from Estimated Depthmaps (with foreground)
python pfm2ply_aligned_gt.py                # Point Clouds from Ground Truth Depthmaps

The point clouds will be saved in the ply and ply_gt image subdirectories respectively as .ply files along with the vertices of these point clouds saved as vertices.npy. This also generates the aligned absolute depthmap and saves the visualization along with the monocular depthmap estimated via the monocular depth estimation network inside the image directories.

4. Generating Scene Point Clouds

Since step 3 comptues individual point clouds, the next task is to merge the vertices of each individual point cloud to generate the point cloud for an entire scene. Run the following command:

python merge2ply.py

You can specify the set of images to be used for reconstructing each scene by editing the merge2ply file. All the scene point clouds are saved in final_fused_scenes folder.

Visualizations of Outputs

1. Mask Estimation

masks

2. 3D Reconstruction of Invidiual Depthmaps

3dr

3. Merged Point Clouds

3dr scenes

Supporting Repositories

I would like to give credit to the following repositories for assisting me in computing intermediate results necessary for the thesis:

Thank you!

Owner
Namas Bhandari
Machine Learning/Deep Learning/AI Enthusiast
Namas Bhandari
AlexaUsingPython - Alexa will pay attention to your order, as: Hello Alexa, play music, Hello Alexa

AlexaUsingPython - Alexa will pay attention to your order, as: Hello Alexa, play music, Hello Alexa, what's the time? Alexa will pay attention to your order, get it, and afterward do some activity as

Abubakar Sattar 10 Aug 18, 2022
The best free and open-source automated time tracker. Cross-platform, extensible, privacy-focused.

Records what you do so that you can know how you've spent your time. All in a secure way where you control the data. Website — Forum — Documentation —

ActivityWatch 7.8k Jan 09, 2023
BestBuy Script Designed to purchase any item when it becomes available.

prerequisites: Selnium; undetected-chromedriver. This Script is designed to order an Item provided a link from BestBuy.com only.

Bransen Smith 0 Jan 12, 2022
A dot matrix rendered using braille characters.

⣿ dotmatrix A dot matrix rendered using braille characters. Description This library provides class called Matrix which represents a dot matrix that c

Tim Fischer 25 Dec 12, 2022
This is an API to get user details for competitive coding platforms - Codeforces, Codechef, SPOJ, Interviewbit. More Platform will be Added Soon.

Competitive-Programming-Score-API An API to get user details for competitive coding platforms - Codeforces, Codechef, SPOJ, Interviewbit Platforms Ava

Aaditya Prakash 3 Jan 17, 2022
Windows symbol tables for Volatility 3

Windows Symbol Tables for Volatility 3 This repository is the Windows Symbol Table storage for Volatility 3. How to Use $ git clone https://github.com

JPCERT Coordination Center 31 Dec 25, 2022
Automator anble you to create automations on your system

WELCOME TO AUTOMATOR BETA This programm is able to create automations on your system. This programm is only an experimantal release; infact it works v

Davide 1 Jan 12, 2022
Number calculator application.

Number calculator application.

Michael J Bailey 3 Oct 08, 2021
Consulta cpf fds

Consulta-cpf Consulta cpf fds Instalação: apt-get update -y

Moleey 1 Nov 24, 2021
My programming language named JoLang. (Mainly created for fun)

JoLang status: not ready So this is my programming language which I decided to name 'JoLang' (inspired by Jonathan and GoLang). Features I implemented

Jonathan 14 Dec 22, 2022
Python calculator made with tkinter package

Python-Calculator Python calculator made with tkinter package. works both on Visual Studio Code Or Any Other Ide Or You Just Copy paste The Same Thing

Pro_Gamer_711 1 Nov 11, 2021
Deis v1, the CoreOS and Docker PaaS: Your PaaS. Your Rules.

This repository (deis/deis) is no longer developed or maintained. The Deis v1 PaaS based on CoreOS Container Linux and Fleet has been replaced by Deis

Deis 6.1k Jan 04, 2023
Get a list of the top-10 rejected libraries in your WhiteSource inventory

WhiteSource Top 10 Rejected Libraries Generate a spreadsheet listing the 10 most common libraries in your WhiteSource inventory that were rejected by

WhiteSource-PS-tools 10 Mar 23, 2022
Mdisk - 🚧 On Construction 🚧

Mdisk Install For Package pip install mdisk pip install git+https://github.com/HeimanPictures/Mdisk.git Usage You can use this as python module or via

AkKiL 6 Aug 08, 2022
A inspector to be able to view and edit Qt style sheet while an application is running

Qt Style Sheet Inspector An inspector widget to view and modify the style sheet of a Qt app at runtime. Usage In order to use the inspector widget on

ESSS 46 Dec 10, 2022
An easy FASTA object handler, reader, writer and translator for small to medium size projects without dependencies.

miniFASTA An easy FASTA object handler, reader, writer and translator for small to medium size projects without dependencies. Installation Using pip /

Jules Kreuer 3 Jun 30, 2022
RxPY - The Reactive Extensions for Python (RxPY)

The Reactive Extensions for Python (RxPY) A library for composing asynchronous and event-based programs using observable collections and query operato

ReactiveX 4.4k Dec 29, 2022
Application launcher and environment management

Application launcher and environment management for 21st century games and digital post-production, built with bleeding-rez and Qt.py News Date Releas

10 Nov 03, 2022
Huggingface package for the discrete VAE used for DALL-E.

DALL-E-Tokenizer Huggingface package for the discrete VAE used for DALL-E.

MyungHoon Jin 5 Sep 01, 2021
This is a working model for which I have used python.

Jarvis_voiceAssistance This is a working model for which I have used python. This model can: 1)Play a video or song on youtube. 2)Tell us time. 3)Tell

Hardik Jain 1 Jan 30, 2022