Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments

Overview

Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments

Paper: arXiv (ICRA 2021)

Video : https://youtu.be/CCDms7KWgI8

System figure


Shared resources


Testing / Evaluation

  1. Setup repository
    • Download test dataset, floorplans and pretrained model to <data>, <floorplan>, and <model> folders.
    • Download this repository. Copy source/sample_data_paths.json as source/data_paths.json and specify default paths.
    • For next steps, we will show example commands for one test datafile. See relevant code for more configuration options.
  2. IMU and WiFi Fusion by Optimization
    • Run source/optim/optimizer.py to geolocalize trajectory with floorplan
    • Sample command: python optimizer.py --out_dir <optimize_out_dir> --data_path <data_folder_path> --loop --no_gui --map_path <path_to_map_image> --map_latlong_path <path_to_csv_with_image_latlong_mapping>
    • E.g. : python optimizer.py --out_dir <output>/optim_s1 --data_path <data>/a001_d1_metrotown_0g --map_path <floorplan>/metrotown_0g.png --loop --no_gui
  3. Floorplan fusion by CNN
    • Run source/nn/nn_eval_full_traj.py for CNN prediction.
    • Sample command: python nn_eval_full_traj.py --floorplan_dir <directory_with_floorplan_images> --floorplan_dpi <floorplan_resolution> --input_dpi <resolution_suitable_for_network> --test_path <optimize_out_dir/data_folder> --out_dir <flow_out_dir> --model_path <fusion_dhl_cnn_checkpoint>
    • E.g. : python nn_eval_full_traj.py --floorplan_dir <floorplan> --test_path <output>/optim_s1/a001_d1_metrotown_0g --out_dir <output>/flow_s1 --model_path <model>/ckpt_fusion_dhl_unet.pt
  4. Run second iteration of optimization with prediction of 2.
    • Run source/optim/optimizer_with_flow.py
    • Sample command: python optimizer_with_flow.py --out_dir <optimize2_out_dir> --data_path <data_folder_path> --map_path <path_to_map_image> --result_dir <flow_out_dir> --loop --no_gui
    • E.g.: python optimizer_with_flow.py --out_dir <output>/optim_s2 --data_path <data>/a001_d1_metrotown_0g --map_path <floorplan>/metrotown_0g.png --result_dir <output>/flow_s1/output/full_result --loop --no_gui
  5. Repeat step 2 with results of 3 as test path --test_path <optimize2_out_dir/data_folder>
    • E.g.: python nn_eval_full_traj.py --floorplan_dir <floorplan> --test_path <output>/optim_s2/a001_d1_metrotown_0g --out_dir <output>/flow_s2 --model_path <model>/ckpt_fusion_dhl_unet.pt

Using your own dataset

The data collection, pre-processing and training steps are listed below. After completion, run testing/evaluation steps with the relevant paths

Data collection

  1. Create floorplan image according to the speicifed format and a known resolution. (Resolution must be chosen in such a way that cropped squares of size 250 by 250 pixel from the floorplan image have multiple rooms/corridors in them. The bigger the rooms, the smaller pixel/meter. We chose 2.5 pixels per meter for the shared dataset which are from shopping malls)
  2. Install Custom Maps app from apk or source and create map by aligning floorplan with google maps
    • During data collection, select map of current floorplan and manually click the current location at sparse points for evaluation.
  3. Put floorplans for training set, and floorplans for test purpose in separate folders and copy source/sample_map_info.json as map_info.json in these folders and specify the floorplan and image names.
  4. Install Sensor Data Logger app and click start service to record data
    • disable battery optimization for the app upon installation
    • location, WiFi and bluetooth needs to be switched on for data collection.
  5. Copy Sensor_Data_Logger output (in Downloads) to computer. Copy relevant Custom_Maps output files (in Downloads/mapLocalize) to a new folder named map inside the copied folder.

Data Preprocessing

  1. Download this repository. Copy source/sample_data_paths.json as source/data_paths.json and specify default paths.
  2. Download RoNIN resnet model checkpoint from the website
  3. Run source/preprocessing/compile_dataset.py to preprocess data into synced data streams and save as hdf5 files.
  4. Generate synthetic data (for training CNN)
    • Run source/gui/synthetic_data_generator.py to generate synthetic data by hand-drawing paths on a map
    • E.g. python synthetic_data_generator.py <path_to_map_image> --map_dpi <pixels_per_meter> --out_dir <path_to_directory> --add_noise
  5. For training groundtruth, run source/optim/optimizer with gui and manually specify constraints (if necessary) until the trajectory looks correct. (command in testing/evaluation)

Floorplan fusion by CNN

  1. Preprocess training data:
    • run source/nn/data_generator_train_real.py and source/nn/data_generator_train_syn.py with mode argument to generate real and synthetic dataset suitable for training the Neural Network. Please refer to the source code for the full list of command line arguments. Change _dpi to the pixel per meter resolution of your floorplan image.
    • Example command for real data generation: python3 data_generator_train_real.py --run_type 'full' --save_all_figs True --data_dir <path-to-real-data-folder> --datalist_file <path-to-list-of-real-data> --floorplans_dir <path-to-train-floorplans> --out_dir <path-to-output-real-dataset-folder>.
    • Example command for synthetic data generation: python3 data_generator_train_syn.py --save_all_figs True --data_dir <path-to-synthetic-data-folder-for-specific-floorplan> --datalist_file <path-to-list-of-synthetic-data-for-specific-floorplan> --floorplans_dir <path-to-floorplans> --out_dir <path-to-output-synthetic-dataset-folder> --which_mall <name-of-the-specific-floorplan>.
  2. Training
    • Run source/nn/nn_train.py to train or test the CNN. Please refer to the source code for the full list of command line arguments and their descriptions.
    • E.g. command for training: python3 nn_train.py --real_floorplans <path_to_real_data's_floorplans> --real_train_list <path_to_real_train_data_list> --real_val_list <path_to_real_validation_data_list> --real_dataset <path_to_real_dataset_from_previous_part> --syn_floorplans <path_to_synthetic_data's_floorplans> --syn_train_list <path_to_synthetic_train_data_list> --syn_val_list <path_to_synthetic_validation_data_list> --syn_dataset <path_to_synthetic_dataset_from_previous_part> --out_dir <path_to_outputs> --mode 'train'
    • E.g. command for testing: python3 nn_train.py --real_floorplans <path_to_real_data's_floorplans> --real_test_list <path_to_real_test_data_list> --real_dataset <path_to_real_dataset_from_previous_part> --syn_floorplans <path_to_synthetic_data's_floorplans> --syn_test_list <path_to_synthetic_test_datalist> --syn_dataset <path_to_synthetic_dataset_from_previous_part> --out_dir <path_to_outputs> --mode <'test_plot_flow'/'test_plot_traj'> --continue_from <path_to_saved_model>
    • Pretrained model

Citation

Please cite the following paper is you use the code, paper, data or any shared resources:

Fusion-DHL: WiFi, IMU, and Floorplan Fusion for Dense History of Locations in Indoor Environments
Sachini Herath, Saghar Irandoust, Bowen Chen, Yiming Qian, Pyojin Kim and Yasutaka Furukawa
2021 IEEE International Conference on Robotics and Automation (ICRA) 
Owner
Sachini Herath
Sachini Herath
The project covers common metrics for super-resolution performance evaluation.

Super-Resolution Performance Evaluation Code The project covers common metrics for super-resolution performance evaluation. Metrics support The script

xmy 10 Aug 03, 2022
Cancer-and-Tumor-Detection-Using-Inception-model - In this repo i am gonna show you how i did cancer/tumor detection in lungs using deep neural networks, specifically here the Inception model by google.

Cancer-and-Tumor-Detection-Using-Inception-model In this repo i am gonna show you how i did cancer/tumor detection in lungs using deep neural networks

Deepak Nandwani 1 Jan 01, 2022
Multi-query Video Retreival

Multi-query Video Retreival

Princeton Visual AI Lab 17 Nov 22, 2022
A large-scale video dataset for the training and evaluation of 3D human pose estimation models

ASPset-510 (Australian Sports Pose Dataset) is a large-scale video dataset for the training and evaluation of 3D human pose estimation models. It contains 17 different amateur subjects performing 30

Aiden Nibali 25 Jun 20, 2021
Jupyter notebooks for using & learning Keras

deep-learning-with-keras-notebooks 這個github的repository主要是個人在學習Keras的一些記錄及練習。希望在學習過程中發現到一些好的資訊與範例也可以對想要學習使用 Keras來解決問題的同好,或是對深度學習有興趣的在學學生可以有一些方便理解與上手範例

ErhWen Kuo 2.1k Dec 27, 2022
[CVPR'22] Official PyTorch Implementation of Collaborative Transformers for Grounded Situation Recognition

[CVPR'22] Collaborative Transformers for Grounded Situation Recognition Paper | Model Checkpoint This is the official PyTorch implementation of Collab

Junhyeong Cho 29 Dec 10, 2022
Realtime YOLO Monster Detection With Non Maximum Supression

Realtime-YOLO-Monster-Detection-With-Non-Maximum-Supression Table of Contents In

5 Oct 07, 2022
Sdf sparse conv - Deep Learning on SDF for Classifying Brain Biomarkers

Deep Learning on SDF for Classifying Brain Biomarkers To reproduce the results f

1 Jan 25, 2022
Official code for Next Check-ins Prediction via History and Friendship on Location-Based Social Networks (MDM 2018)

MUC Next Check-ins Prediction via History and Friendship on Location-Based Social Networks (MDM 2018) Performance Details for Accuracy: | Dataset

Yijun Su 3 Oct 09, 2022
Neural Scene Flow Fields using pytorch-lightning, with potential improvements

nsff_pl Neural Scene Flow Fields using pytorch-lightning. This repo reimplements the NSFF idea, but modifies several operations based on observation o

AI葵 178 Dec 21, 2022
Model of an AI powered sign language interpreter.

TEXT AND SPEECH TO SIGN LANGUAGE. A web application which takes in text or live audio speech recording as input, converts and displays the relevant Si

Mark Gatere 4 Mar 30, 2022
This is a collection of simple PyTorch implementations of neural networks and related algorithms. These implementations are documented with explanations,

labml.ai Deep Learning Paper Implementations This is a collection of simple PyTorch implementations of neural networks and related algorithms. These i

labml.ai 16.4k Jan 09, 2023
Transformers based fully on MLPs

Awesome MLP-based Transformers papers An up-to-date list of Transformers based fully on MLPs without attention! Why this repo? After transformers and

Fawaz Sammani 35 Dec 30, 2022
This tutorial repository is to introduce the functionality of KGTK to first-time users

Welcome to the KGTK notebook tutorial The goal of this tutorial repository is to introduce the functionality of KGTK to first-time users. The Knowledg

USC ISI I2 58 Dec 21, 2022
Implementation of Vaswani, Ashish, et al. "Attention is all you need."

Attention Is All You Need Paper Implementation This is my from-scratch implementation of the original transformer architecture from the following pape

Brando Koch 195 Dec 30, 2022
Stochastic Scene-Aware Motion Prediction

Stochastic Scene-Aware Motion Prediction [Project Page] [Paper] Description This repository contains the training code for MotionNet and GoalNet of SA

Mohamed Hassan 31 Dec 09, 2022
This repository is for the preprint "A generative nonparametric Bayesian model for whole genomes"

BEAR Overview This repository contains code associated with the preprint A generative nonparametric Bayesian model for whole genomes (2021), which pro

Debora Marks Lab 10 Sep 18, 2022
RNN Predict Street Commercial Vitality

RNN-for-Predicting-Street-Vitality Code and dataset for Predicting the Vitality of Stores along the Street based on Business Type Sequence via Recurre

Zidong LIU 1 Dec 15, 2021
Project page for our ICCV 2021 paper "The Way to my Heart is through Contrastive Learning"

The Way to my Heart is through Contrastive Learning: Remote Photoplethysmography from Unlabelled Video This is the official project page of our ICCV 2

36 Jan 06, 2023
Continual learning with sketched Jacobian approximations

Continual learning with sketched Jacobian approximations This repository contains the code for reproducing figures and results in the paper ``Provable

Machine Learning and Information Processing Laboratory 1 Jun 30, 2022