LinkNet - This repository contains our Torch7 implementation of the network developed by us at e-Lab.

Related tags

Deep LearningLinkNet
Overview

LinkNet

This repository contains our Torch7 implementation of the network developed by us at e-Lab. You can go to our blogpost or read the article LinkNet: Exploiting Encoder Representations for Efficient Semantic Segmentation for further details.

Dependencies:

  • Torch7 : you can follow our installation step specified here
  • VideoDecoder : video decoder for torch that utilizes avcodec library.
  • Profiler : use it to calculate # of paramaters, operations and forward pass time of any network trained using torch.

Currently the network can be trained on two datasets:

Datasets Input Resolution # of classes
CamVid (cv) 768x576 11
Cityscapes (cs) 1024x512 19

To download both datasets, follow the link provided above. Both the datasets are first of all resized by the training script and if you want then you can cache this resized data using --cachepath option. In case of CamVid dataset, the available video data is first split into train/validate/test set. This is done using prepCamVid.lua file. dataDistributionCV.txt contains the detail about splitting of CamVid dataset. These things are automatically run before training of the network.

LinkNet performance on both of the above dataset:

Datasets Best IoU Best iIoU
Cityscapes 76.44 60.78
CamVid 69.10 55.83

Pretrained models and confusion matrices for both datasets can be found in the latest release.

Files/folders and their usage:

  • run.lua : main file
  • opts.lua : contains all the input options used by the tranining script
  • data : data loaders for loading datasets
  • [models] : all the model architectures are defined here
  • train.lua : loading of models and error calculation
  • test.lua : calculate testing error and save confusion matrices

There are three model files present in models folder:

  • model.lua : our LinkNet architecture
  • model-res-dec.lua : LinkNet with residual connection in each of the decoder blocks. This slightly improves the result but we had to use bilinear interpolation in residual connection because of which we were not able to run our trained model on TX1.
  • nobypass.lua : this architecture does not use any link between encoder and decoder. You can use this model to verify if connecting encoder and decoder modules actually improve performance.

A sample command to train network is given below:

th main.lua --datapath /Datasets/Cityscapes/ --cachepath /dataCache/cityscapes/ --dataset cs --model models/model.lua --save /Models/cityscapes/ --saveTrainConf --saveAll --plot

License

This software is released under a creative commons license which allows for personal and research use only. For a commercial license please contact the authors. You can view a license summary here: http://creativecommons.org/licenses/by-nc/4.0/

Comments
  • memory consuming

    memory consuming

    The model read all the dataset into the momory, this method is too memory consuming. Maybe it is better to read the dataset list and iterate the list when training .

    opened by mingminzhen 7
  • Training on camvid dataset

    Training on camvid dataset

    Hi. I can't reproduce your result on camvid dataset. What is the learning rate and number of training epoch you used in your training, is your published result on validate or test set?.

    opened by vietdoan 4
  • Torch: not enough memory (17GB)

    Torch: not enough memory (17GB)

    Hi, all

    When I run : th main.lua --datapath /data2/cityscapes_dataset/leftImg8bit/all_train_images/ --cachepath /data2/cityscapes_dataset/leftImg8bit/dataCache/ --dataset cs --model models/model.lua --save save_models/cityscapes/ --saveTrainConf --saveAll --plot

    I got "Torch: not enough memory: you tried to allocate 17GB" error (details)

    It's strange because the paper mentioned it is trained using Titan X which has 12GB memory. Why the network consumes 17GB in running?

    Any suggestion to fix this issue?

    Thanks!

    opened by amiltonwong 3
  • Fine Tuning

    Fine Tuning

    Hi,

    is there any possibility to fine-tune this model on a custom datase with different number of classes? The pre-trained weights must be exist also, as I know.

    opened by MyVanitar 3
  • Model input/output details?

    Model input/output details?

    Hi,

    I'm having a hell of a time trying to understand what the model is expecting in terms of input and output. I'm trying to use this model in an iOS project, so I need to convert the model to Apple's CoreML format.

    Image input questions:

    • For image pixel values: 0-255, 0-1, -1-1?
    • RGB or BGR?
    • Color bias?

    Prediction output:

    • Looks like the shape is # of classes, width, height?
    • Predictions are positive floats from 0-100?

    So far I'm having the best luck with these specifications:

    import torch
    from torch2coreml import convert
    from torch.utils.serialization import load_lua
    
    model = load_lua("model-cs-IoU-cpu.net")
    
    input_shape = (3, 512, 1024)
    coreml_model = convert(
            model,
            [input_shape],
            input_names=['inputImage'],
            output_names=['outputImage'],
            image_input_names=['inputImage'],
            preprocessing_args={
                'image_scale': 2/255.0
            }
        )
    coreml_model.save("/home/sean/Downloads/Final/model-cs-IoU.mlmodel")
    
    opened by seantempesta 2
  • About IoU

    About IoU

    Hi, @codeAC29
    I cannot obtain the high IoU in my training. I looked into your code and found that, the IoU is computed via averageValid. But this is actually computing the mean of class accuracy. The IoU should be the value of averageUnionValid. Do you notice the difference and obtain 76% IoU by averageUnionValid ?

    Sorry for the trouble. For convenience, I refer the definition of averageValid and averageUnionValid here.

    opened by qqning 2
  • Error while running linknet main file

    Error while running linknet main file

    Hii, I am getting this error while running main.py RuntimeError: Expected object of type torch.cuda.LongTensor but found type torch.cuda.FloatTensor for argument 2 'target'. Please help me out. Also when i try to run the trained models i am running into error. I am using pytorch to run .net files. I am not able to load them as it is showing error: name cs is not defined. It is a model. Why does it have a variable named cs(here cs represents cityscapes) in it?

    opened by Tharun98 0
  • Model fails for input size other than multiples of 32(for depth of 4)

    Model fails for input size other than multiples of 32(for depth of 4)

    Hi, If we give the input image size other than 32 multiples there is a size mismatch error when adding the output from encoder3 and decoder4. For example input image size is 1000x2000 output of encoder3 is 63x125 and decoder4 output size is 64x126. We need adjust parameters for spatialfullconvolution layer only if input image size is multiple of 2^(n+1) where n is encoder depth. For other image sizes adjust parameter depends on the image size. In this example network works if adjust parameter is zero in decoders 3 and 4. Please clarify if this network works only for 2^(n+1) sizes. Thanks.

    opened by Tharun98 1
  • How about the image resolution?

    How about the image resolution?

    Hi, I am reproducing the LinkNet. I have a doubt about the input image resolution and the output image resolution when you compute the FLOPS. I find my FLOPS and running speed are different your results reported on your paper.

    opened by ycszen 5
  • linknet  architecture

    linknet architecture

    iam trying to build linknet in caffe. Could you please help me in below qns: 1)Found that there are 5 downsampling and 6 updsampling by 2. if we have different no of up sampling and down sampling(6,5) how can we get the same output shape as input. Referred:https://arxiv.org/pdf/1707.03718.pdf 2)how many iterations you ran to get the proper results. 3)To match the encoder and decoder output shape i used crop layer before Eltwise instead of adding extra row or column. Will it make any difference?

    opened by vishnureghu007 7
  • Error while training

    Error while training

    I got the camVid dataset as specified in the in the read me file and installed video-decoder

    Ientered the following command to start training: th main.lua --datapath ./data/CamVid/ --cachepath ./dataCache/CamV/ --dataset cv --model ./models/model.lua --save ./Models/CamV/ --saveTrainConf --saveAll --plot

    And I got the following error,

    Preparing CamVid dataset for data loader Filenames and their role found in: ./misc/dataDistributionCV.txt

    Getting input images and labels for: 01TP_extract.avi /home/jayp/torch/install/bin/luajit: /home/jayp/torch/install/share/lua/5.1/trepl/init.lua:389: /home/jayp/torch/install/share/lua/5.1/trepl/init.lua:389: error loading module 'libvideo_decoder' from file '/home/jayp/torch/install/lib/lua/5.1/libvideo_decoder.so': /home/jayp/torch/install/lib/lua/5.1/libvideo_decoder.so: undefined symbol: avcodec_get_frame_defaults stack traceback: [C]: in function 'error' /home/jayp/torch/install/share/lua/5.1/trepl/init.lua:389: in function 'require' main.lua:34: in main chunk [C]: in function 'dofile' ...jayp/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:150: in main chunk

    I would really appreciate if anyone would help me with this.

    Thank You!

    opened by jay98 4
Releases(v1.0)
Owner
e-Lab
e-Lab
《Dual-Resolution Correspondence Network》(NeurIPS 2020)

Dual-Resolution Correspondence Network Dual-Resolution Correspondence Network, NeurIPS 2020 Dependency All dependencies are included in asset/dualrcne

Active Vision Laboratory 45 Nov 21, 2022
FlingBot: The Unreasonable Effectiveness of Dynamic Manipulations for Cloth Unfolding

This repository contains code for training and evaluating FlingBot in both simulation and real-world settings on a dual-UR5 robot arm setup for Ubuntu 18.04

Columbia Artificial Intelligence and Robotics Lab 70 Dec 06, 2022
Multi-task Multi-agent Soft Actor Critic for SMAC

Multi-task Multi-agent Soft Actor Critic for SMAC Overview The CARE formulti-task: Multi-Task Reinforcement Learning with Context-based Representation

RuanJingqing 8 Sep 30, 2022
NCVX (NonConVeX): A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning.

The source code is temporariy removed, as we are solving potential copyright and license issues with GRANSO (http://www.timmitchell.com/software/GRANS

SUN Group @ UMN 28 Aug 03, 2022
UT-Sarulab MOS prediction system using SSL models

UTMOS: UTokyo-SaruLab MOS Prediction System Official implementation of "UTMOS: UTokyo-SaruLab System for VoiceMOS Challenge 2022" submitted to INTERSP

sarulab-speech 58 Nov 22, 2022
Code of the paper "Multi-Task Meta-Learning Modification with Stochastic Approximation".

Multi-Task Meta-Learning Modification with Stochastic Approximation This repository contains the code for the paper "Multi-Task Meta-Learning Modifica

Andrew 3 Jan 05, 2022
City Surfaces: City-scale Semantic Segmentation of Sidewalk Surfaces

City Surfaces: City-scale Semantic Segmentation of Sidewalk Surfaces Paper Temporary GitHub page for City Surfaces paper. More soon! While designing s

14 Nov 10, 2022
Implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021).

[PDF] | [Slides] The official implementation of Learning Gradient Fields for Molecular Conformation Generation (ICML 2021 Long talk) Installation Inst

MilaGraph 117 Dec 09, 2022
Face Identity Disentanglement via Latent Space Mapping [SIGGRAPH ASIA 2020]

Face Identity Disentanglement via Latent Space Mapping Description Official Implementation of the paper Face Identity Disentanglement via Latent Space

150 Dec 07, 2022
Official Implementation of DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation

DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation [Arxiv] [Paper] As acquiring pixel-wise an

Lukas Hoyer 305 Dec 29, 2022
Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields.

This repository contains the code release for Mip-NeRF: A Multiscale Representation for Anti-Aliasing Neural Radiance Fields. This implementation is written in JAX, and is a fork of Google's JaxNeRF

Google 625 Dec 30, 2022
A Python module for the generation and training of an entry-level feedforward neural network.

ff-neural-network A Python module for the generation and training of an entry-level feedforward neural network. This repository serves as a repurposin

Riadh 2 Jan 31, 2022
Open CV - Convert a picture to look like a cartoon sketch in python

Use the video https://www.youtube.com/watch?v=k7cVPGpnels for initial learning.

Sammith S Bharadwaj 3 Jan 29, 2022
Axel - 3D printed robotic hands and they controll with Raspberry Pi and Arduino combo

Axel It's our graduation project about 3D printed robotic hands and they control

0 Feb 14, 2022
An open-source online reverse dictionary.

An open-source online reverse dictionary.

THUNLP 6.3k Jan 09, 2023
WORD: Revisiting Organs Segmentation in the Whole Abdominal Region

WORD: Revisiting Organs Segmentation in the Whole Abdominal Region. This repository provides the codebase and dataset for our work WORD: Revisiting Or

Healthcare Intelligence Laboratory 71 Jan 07, 2023
Tool for live presentations using manim

manim-presentation Tool for live presentations using manim Install pip install manim-presentation opencv-python Usage Use the class Slide as your sce

Federico Galatolo 146 Jan 06, 2023
Use your Philips Hue lights as Racing Flags. Works with Assetto Corsa, Assetto Corsa Competizione and iRacing.

phue-racing-flags Use your Philips Hue lights as Racing Flags. Explore the docs » Report Bug · Request Feature Table of Contents About The Project Bui

50 Sep 03, 2022
DECAF: Generating Fair Synthetic Data Using Causally-Aware Generative Networks

DECAF (DEbiasing CAusal Fairness) Code Author: Trent Kyono This repository contains the code used for the "DECAF: Generating Fair Synthetic Data Using

van_der_Schaar \LAB 7 Nov 24, 2022
Code repository for the paper "Doubly-Trained Adversarial Data Augmentation for Neural Machine Translation" with instructions to reproduce the results.

Doubly Trained Neural Machine Translation System for Adversarial Attack and Data Augmentation Languages Experimented: Data Overview: Source Target Tra

Steven Tan 1 Aug 18, 2022