Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments

Overview

Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments

Paper: arXiv (ICRA 2021)

Video : https://youtu.be/CCDms7KWgI8

System figure


Shared resources


Testing / Evaluation

  1. Setup repository
    • Download test dataset, floorplans and pretrained model to <data>, <floorplan>, and <model> folders.
    • Download this repository. Copy source/sample_data_paths.json as source/data_paths.json and specify default paths.
    • For next steps, we will show example commands for one test datafile. See relevant code for more configuration options.
  2. IMU and WiFi Fusion by Optimization
    • Run source/optim/optimizer.py to geolocalize trajectory with floorplan
    • Sample command: python optimizer.py --out_dir <optimize_out_dir> --data_path <data_folder_path> --loop --no_gui --map_path <path_to_map_image> --map_latlong_path <path_to_csv_with_image_latlong_mapping>
    • E.g. : python optimizer.py --out_dir <output>/optim_s1 --data_path <data>/a001_d1_metrotown_0g --map_path <floorplan>/metrotown_0g.png --loop --no_gui
  3. Floorplan fusion by CNN
    • Run source/nn/nn_eval_full_traj.py for CNN prediction.
    • Sample command: python nn_eval_full_traj.py --floorplan_dir <directory_with_floorplan_images> --floorplan_dpi <floorplan_resolution> --input_dpi <resolution_suitable_for_network> --test_path <optimize_out_dir/data_folder> --out_dir <flow_out_dir> --model_path <fusion_dhl_cnn_checkpoint>
    • E.g. : python nn_eval_full_traj.py --floorplan_dir <floorplan> --test_path <output>/optim_s1/a001_d1_metrotown_0g --out_dir <output>/flow_s1 --model_path <model>/ckpt_fusion_dhl_unet.pt
  4. Run second iteration of optimization with prediction of 2.
    • Run source/optim/optimizer_with_flow.py
    • Sample command: python optimizer_with_flow.py --out_dir <optimize2_out_dir> --data_path <data_folder_path> --map_path <path_to_map_image> --result_dir <flow_out_dir> --loop --no_gui
    • E.g.: python optimizer_with_flow.py --out_dir <output>/optim_s2 --data_path <data>/a001_d1_metrotown_0g --map_path <floorplan>/metrotown_0g.png --result_dir <output>/flow_s1/output/full_result --loop --no_gui
  5. Repeat step 2 with results of 3 as test path --test_path <optimize2_out_dir/data_folder>
    • E.g.: python nn_eval_full_traj.py --floorplan_dir <floorplan> --test_path <output>/optim_s2/a001_d1_metrotown_0g --out_dir <output>/flow_s2 --model_path <model>/ckpt_fusion_dhl_unet.pt

Using your own dataset

The data collection, pre-processing and training steps are listed below. After completion, run testing/evaluation steps with the relevant paths

Data collection

  1. Create floorplan image according to the speicifed format and a known resolution. (Resolution must be chosen in such a way that cropped squares of size 250 by 250 pixel from the floorplan image have multiple rooms/corridors in them. The bigger the rooms, the smaller pixel/meter. We chose 2.5 pixels per meter for the shared dataset which are from shopping malls)
  2. Install Custom Maps app from apk or source and create map by aligning floorplan with google maps
    • During data collection, select map of current floorplan and manually click the current location at sparse points for evaluation.
  3. Put floorplans for training set, and floorplans for test purpose in separate folders and copy source/sample_map_info.json as map_info.json in these folders and specify the floorplan and image names.
  4. Install Sensor Data Logger app and click start service to record data
    • disable battery optimization for the app upon installation
    • location, WiFi and bluetooth needs to be switched on for data collection.
  5. Copy Sensor_Data_Logger output (in Downloads) to computer. Copy relevant Custom_Maps output files (in Downloads/mapLocalize) to a new folder named map inside the copied folder.

Data Preprocessing

  1. Download this repository. Copy source/sample_data_paths.json as source/data_paths.json and specify default paths.
  2. Download RoNIN resnet model checkpoint from the website
  3. Run source/preprocessing/compile_dataset.py to preprocess data into synced data streams and save as hdf5 files.
  4. Generate synthetic data (for training CNN)
    • Run source/gui/synthetic_data_generator.py to generate synthetic data by hand-drawing paths on a map
    • E.g. python synthetic_data_generator.py <path_to_map_image> --map_dpi <pixels_per_meter> --out_dir <path_to_directory> --add_noise
  5. For training groundtruth, run source/optim/optimizer with gui and manually specify constraints (if necessary) until the trajectory looks correct. (command in testing/evaluation)

Floorplan fusion by CNN

  1. Preprocess training data:
    • run source/nn/data_generator_train_real.py and source/nn/data_generator_train_syn.py with mode argument to generate real and synthetic dataset suitable for training the Neural Network. Please refer to the source code for the full list of command line arguments. Change _dpi to the pixel per meter resolution of your floorplan image.
    • Example command for real data generation: python3 data_generator_train_real.py --run_type 'full' --save_all_figs True --data_dir <path-to-real-data-folder> --datalist_file <path-to-list-of-real-data> --floorplans_dir <path-to-train-floorplans> --out_dir <path-to-output-real-dataset-folder>.
    • Example command for synthetic data generation: python3 data_generator_train_syn.py --save_all_figs True --data_dir <path-to-synthetic-data-folder-for-specific-floorplan> --datalist_file <path-to-list-of-synthetic-data-for-specific-floorplan> --floorplans_dir <path-to-floorplans> --out_dir <path-to-output-synthetic-dataset-folder> --which_mall <name-of-the-specific-floorplan>.
  2. Training
    • Run source/nn/nn_train.py to train or test the CNN. Please refer to the source code for the full list of command line arguments and their descriptions.
    • E.g. command for training: python3 nn_train.py --real_floorplans <path_to_real_data's_floorplans> --real_train_list <path_to_real_train_data_list> --real_val_list <path_to_real_validation_data_list> --real_dataset <path_to_real_dataset_from_previous_part> --syn_floorplans <path_to_synthetic_data's_floorplans> --syn_train_list <path_to_synthetic_train_data_list> --syn_val_list <path_to_synthetic_validation_data_list> --syn_dataset <path_to_synthetic_dataset_from_previous_part> --out_dir <path_to_outputs> --mode 'train'
    • E.g. command for testing: python3 nn_train.py --real_floorplans <path_to_real_data's_floorplans> --real_test_list <path_to_real_test_data_list> --real_dataset <path_to_real_dataset_from_previous_part> --syn_floorplans <path_to_synthetic_data's_floorplans> --syn_test_list <path_to_synthetic_test_datalist> --syn_dataset <path_to_synthetic_dataset_from_previous_part> --out_dir <path_to_outputs> --mode <'test_plot_flow'/'test_plot_traj'> --continue_from <path_to_saved_model>
    • Pretrained model

Citation

Please cite the following paper is you use the code, paper, data or any shared resources:

Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments
Sachini Herath, Saghar Irandoust, Bowen Chen, Yiming Qian, Pyojin Kim and Yasutaka Furukawa
2021 IEEE International Conference on Robotics and Automation (ICRA) 
Owner
Sachini Herath
Sachini Herath
JFB: Jacobian-Free Backpropagation for Implicit Models

JFB: Jacobian-Free Backpropagation for Implicit Models

Typal Research 28 Dec 11, 2022
[CVPR2022] Representation Compensation Networks for Continual Semantic Segmentation

RCIL [CVPR2022] Representation Compensation Networks for Continual Semantic Segmentation Chang-Bin Zhang1, Jia-Wen Xiao1, Xialei Liu1, Ying-Cong Chen2

Chang-Bin Zhang 71 Dec 28, 2022
text_recognition_toolbox: The reimplementation of a series of classical scene text recognition papers with Pytorch in a uniform way.

text recognition toolbox 1. 项目介绍 该项目是基于pytorch深度学习框架,以统一的改写方式实现了以下6篇经典的文字识别论文,论文的详情如下。该项目会持续进行更新,欢迎大家提出问题以及对代码进行贡献。 模型 论文标题 发表年份 模型方法划分 CRNN 《An End-t

168 Dec 24, 2022
Pixel-Perfect Structure-from-Motion with Featuremetric Refinement (ICCV 2021, Oral)

Pixel-Perfect Structure-from-Motion (ICCV 2021 Oral) We introduce a framework that improves the accuracy of Structure-from-Motion by refining keypoint

Computer Vision and Geometry Lab 831 Dec 29, 2022
Official repository for the paper, MidiBERT-Piano: Large-scale Pre-training for Symbolic Music Understanding.

MidiBERT-Piano Authors: Yi-Hui (Sophia) Chou, I-Chun (Bronwin) Chen Introduction This is the official repository for the paper, MidiBERT-Piano: Large-

137 Dec 15, 2022
A PyTorch implementation of Radio Transformer Networks from the paper "An Introduction to Deep Learning for the Physical Layer".

An Introduction to Deep Learning for the Physical Layer An usable PyTorch implementation of the noisy autoencoder infrastructure in the paper "An Intr

Gram.AI 120 Nov 21, 2022
Explainer for black box models that predict molecule properties

Explaining why that molecule exmol is a package to explain black-box predictions of molecules. The package uses model agnostic explanations to help us

White Laboratory 172 Dec 19, 2022
Pytorch-3dunet - 3D U-Net model for volumetric semantic segmentation written in pytorch

pytorch-3dunet PyTorch implementation 3D U-Net and its variants: Standard 3D U-Net based on 3D U-Net: Learning Dense Volumetric Segmentation from Spar

Adrian Wolny 1.3k Dec 28, 2022
Gin provides a lightweight configuration framework for Python

Gin Config Authors: Dan Holtmann-Rice, Sergio Guadarrama, Nathan Silberman Contributors: Oscar Ramirez, Marek Fiser Gin provides a lightweight configu

Google 1.7k Jan 03, 2023
Mscp jamf - Build compliance in jamf

mscp_jamf Build compliance in Jamf. This will build the following xml pieces to

Bob Gendler 3 Jul 25, 2022
Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

Point Cloud Denoising input segmentation output raw point-cloud valid/clear fog rain de-noised Abstract Lidar sensors are frequently used in environme

75 Nov 24, 2022
Implementations of LSTM: A Search Space Odyssey variants and their training results on the PTB dataset.

An LSTM Odyssey Code for training variants of "LSTM: A Search Space Odyssey" on Fomoro. Check out the blog post. Training Install TensorFlow. Clone th

Fomoro AI 95 Apr 13, 2022
FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX.

FedJAX: Federated learning with JAX What is FedJAX? FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX. FedJAX priori

Google 208 Dec 14, 2022
WTTE-RNN a framework for churn and time to event prediction

WTTE-RNN Weibull Time To Event Recurrent Neural Network A less hacky machine-learning framework for churn- and time to event prediction. Forecasting p

Egil Martinsson 727 Dec 28, 2022
This repository contains a CBIR system that uses swin transformer to extract image's feature.

Swin-transformer based CBIR This repository contains a CBIR(content-based image retrieval) system. Here we use Swin-transformer to extract query image

JsHou 12 Nov 17, 2022
Deep Crop Rotation

Deep Crop Rotation Paper (to come very soon!) We propose a deep learning approach to modelling both inter- and intra-annual patterns for parcel classi

Félix Quinton 5 Sep 23, 2022
Face2webtoon - Despite its importance, there are few previous works applying I2I translation to webtoon.

Despite its importance, there are few previous works applying I2I translation to webtoon. I collected dataset from naver webtoon 연애혁명 and tried to transfer human faces to webtoon domain.

이상윤 64 Oct 19, 2022
Localizing Visual Sounds the Hard Way

Localizing-Visual-Sounds-the-Hard-Way Code and Dataset for "Localizing Visual Sounds the Hard Way". The repo contains code and our pre-trained model.

Honglie Chen 58 Dec 07, 2022
KITTI-360 Annotation Tool is a framework that developed based on python(cherrypy + jinja2 + sqlite3) as the server end and javascript + WebGL as the front end.

KITTI-360 Annotation Tool is a framework that developed based on python(cherrypy + jinja2 + sqlite3) as the server end and javascript + WebGL as the front end.

86 Dec 12, 2022
A Broad Study on the Transferability of Visual Representations with Contrastive Learning

A Broad Study on the Transferability of Visual Representations with Contrastive Learning This repository contains code for the paper: A Broad Study on

Ashraful Islam 29 Nov 09, 2022