Evaluating deep transfer learning for whole-brain cognitive decoding

Overview

Evaluating deep transfer learning for whole-brain cognitive decoding

This README file contains the following sections:

Project description

This project provides two main packages (see src/) that allow to apply DeepLight (see below) to the task-fMRI data of the Human Connectome Project (HCP):

  • deeplight is a simple python package that provides easy access to two pre-trained DeepLight architectures (2D-DeepLight and 3D-DeepLight; see below), designed for cognitive decoding of whole-brain fMRI data. Both architecturs were pre-trained with the fMRI data of 400 individuals in six of the seven HCP experimental tasks (all tasks except for the working memory task, which we left out for testing purposes; click here for details on the HCP data).
  • hcprepis a simple python package that allows to easily download the HCP task-fMRI data in a preprocessed format via the Amazon Web Services (AWS) S3 storage system and to transform these data into the tensorflow records data format.

Repository organization

├── poetry.lock         <- Overview of project dependencies
├── pyproject.toml      <- Lists details of installed dependencies
├── README.md           <- This README file
├── .gitignore          <- Specifies files that git should ignore
|
├── scrips/
|    ├── decode.py      <- An example of how to decode fMRI data with `deeplight`
|    ├── download.py    <- An example of how to download the preprocessed HCP fMRI data with `hcprep`
|    ├── interpret.py   <- An example of how to interpret fMRI data with `deeplight`
|    └── preprocess.sh  <- An example of how to preprocess fMRI data with `hcprep`
|    └── train.py       <- An example of how to train with `hcprep`
|
└── src/
|    ├── deeplight/
|    |    └──           <- `deeplight` package
|    ├── hcprep/
|    |    └──           <- 'hcprep' package
|    ├── modules/
|    |    └──           <- 'modules' package
|    └── setup.py       <- Makes 'deeplight', `hcprep`, and `modules` pip-installable (pip install -e .)  

Installation

deeplight and hcprep are written for python 3.6 and require a working python environment running on your computer (we generally recommend pyenv for python version management).

First, clone and switch to this repository:

git clone https://github.com/athms/evaluating-deeplight-transfer.git
cd evaluating-deeplight-transfer

This project uses python poetry for dependency management. To install all required dependencies with poetry, run:

poetry install

To then install deeplight, hcprep, and modules in your poetry environment, run:

cd src/
poetry run pip3 install -e .

Packages

HCPrep

hcprep stores the HCP task-fMRI data data locally in the Brain Imaging Data Structure (BIDS) format.

To make fMRI data usable for DL analyses with TensorFlow, hcprep can clean the downloaded fMRI data and store these in the TFRecords data format.

Getting data access: To download the HCP task-fMRI data, you will need AWS access to the HCP public data directory. A detailed instruction can be found here. Make sure to safely store the ACCESS_KEY and SECRET_KEY; they are required to access the data via the AWS S3 storage system.

AWS configuration: Setup your local AWS client (as described here) and add the following profile to '~/.aws/config'

[profile hcp]
region=eu-central-1

Choose the region based on your location.

TFR data storage: hcprep stores the preprocessed fMRI data locally in TFRecords format, with one entry for each input fMRI volume of the data, each containing the following features:

  • volume: the flattened voxel activations with shape 91x109x91 (flattened over the X (91), Y (109), and Z (91) dimensions)
  • task_id, subject_id, run_id: numerical id of task, subject, and run
  • tr: TR of the volume in the underlying experimental task
  • state: numerical label of the cognive state associated with the volume within its task (e.g., [0,1,2,3] for the four cognitive states of the working memory task)
  • onehot: one-hot encoding of the state across all experimental tasks that are used for training (e.g., there are 20 cognitive tasks across the seven experimental tasks of the HCP; the four cognitive states of the working memory task could thus be mapped to the last four positions of the one-hot encoding, with indices [16: 0, 17: 1, 18: 2, 19: 3])

Note that hcprep also provides basic descriptive information about the HCP task-fMRI data in info.basics:

hcp_info = hcprep.info.basics()

basics contains the following information:

  • tasks: names of all HCP experimental tasks ('EMOTION', 'GAMBLING', 'LANGUAGE', 'MOTOR', 'RELATIONAL', 'SOCIAL', 'WM')
  • subjects: dictionary containing 1000 subject IDs for each task
  • runs: run IDs ('LR', 'RL')
  • t_r: repetition time of the fMRI data in seconds (0.72)
  • states_per_task: dictionary containing the label of each cognitive state of each task
  • onehot_idx_per_task: index that is used to assign cognitive states of each task to the onehotencoding of the TFR-files (see onehot above)

For further details on the experimental tasks and their cognitive states, click here.

DeepLight

deeplight implements two DeepLight architectures ("2D" and "3D"), which can be accessed as deeplight.two (2D) and deeplight.three (3D).

Importantly, both DeepLight architectures operate on the level of individual whole-brain fMRI volumes (e.g., individual TRs).

2D-DeepLight: A whole-brain fMRI volume is first sliced into a sequence of axial 2D-images (from bottom-to-top). These images are passed to a DL model, consisting of a 2D-convolutional feature extractor as well as an LSTM unit and output layer. First, the 2D-convolutional feature extractor reduces the dimensionality of the axial brain images through a sequence of 2D-convolution layers. The resulting sequence of higher-level slice representations is then fed to a bi-directional LSTM, modeling the spatial dependencies of brain activity within and across brain slices. Lastly, 2D-DeepLight outputs a decoding decision about the cognitive state underlying the fMRI volume, through a softmax output layer with one output unit per cognitive state in the data.

3D-DeepLight: The whole-brain fMRI volume is passed to a 3D-convolutional feature extractor, consisting of a sequence of twelve 3D-convolution layers. The 3D-convolutional feature extractor directly projects the fMRI volume into a higher-level, but lower dimensional, representation of whole-brain activity, without the need of an LSTM. To make a decoding decision, 3D-DeepLight utilizes an output layer that is composed of a 1D- convolution and global average pooling layer as well as a softmax activation function. The 1D-convolution layer maps the higher-level representation of whole-brain activity of the 3D-convolutional feature extractor to one representation for each cognitive state in the data, while the global average pooling layer and softmax function then reduce these to a decoding decision.

To interpret the decoding decisions of the two DeepLight architectures, relating their decoding decisions to the fMRI data, deeplight makes use of the LRP technique. The LRP technique decomposes individual decoding decisions of a DL model into the contributions of the individual input features (here individual voxel activities) to these decisions.

Both deeplight architectures implement basic fit, decode, and interpret methods, next to other functionalities. For details on how to {train, decode, interpret} with deeplight, see scripts/.

For further methdological details regarding the two DeepLight architectures, see the upcoming preprint.

Note that we currently recommend to run any applications of interpret with 2D-DeepLight on CPU instead of GPU, due to its high memory demand (assuming that your available CPU memory is larger than your available GPU memory). This switch can be made by setting the environment variable export CUDA_VISIBLE_DEVICES="". We are currently working on reducing the overall memory demand of interpret with 2D-DeepLight and will push a code update soon.

Modules

modules is a fork of the modules module from interprettensor, which deeplight uses to build the 2D-DeepLight architecture. Note that modules is licensed differently from the other python packages in this repository (see modules/LICENSE).

Basic usage

You can find a set of example python scripts in scripts/, which illustrate how to download and preprocess task-fMRI data from the Human Connectome Project with hcprep and how to {train on, decode, interpret} fMRI data with the two DeepLight architectures of deeplight.

You can run individual scripts in your poetryenvironment with:

cd scripts/
poetry run python <SCRIPT NAME>
Owner
Armin Thomas
Ram and Vijay Shriram Data Science Fellow at Stanford Data Science
Armin Thomas
Official implementation of "UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspective with Transformer"

[AAAI2022] UCTransNet This repo is the official implementation of "UCTransNet: Rethinking the Skip Connections in U-Net from a Channel-wise Perspectiv

Haonan Wang 199 Jan 03, 2023
Finite Element Analysis

FElupe - Finite Element Analysis FElupe is a Python 3.6+ finite element analysis package focussing on the formulation and numerical solution of nonlin

Andreas D. 20 Jan 09, 2023
Fast EMD for Python: a wrapper for Pele and Werman's C++ implementation of the Earth Mover's Distance metric

PyEMD: Fast EMD for Python PyEMD is a Python wrapper for Ofir Pele and Michael Werman's implementation of the Earth Mover's Distance that allows it to

William Mayner 433 Dec 31, 2022
Pytorch implementation of "Neural Wireframe Renderer: Learning Wireframe to Image Translations"

Neural Wireframe Renderer: Learning Wireframe to Image Translations Pytorch implementation of ideas from the paper Neural Wireframe Renderer: Learning

Yuan Xue 7 Nov 14, 2022
EDCNN: Edge enhancement-based Densely Connected Network with Compound Loss for Low-Dose CT Denoising

EDCNN: Edge enhancement-based Densely Connected Network with Compound Loss for Low-Dose CT Denoising By Tengfei Liang, Yi Jin, Yidong Li, Tao Wang. Th

workingcoder 115 Jan 05, 2023
Individual Treatment Effect Estimation

CAPE Individual Treatment Effect Estimation Run CAPE python train_causal.py --loop 10 -m cape_cau -d NI --i_t 1 Run a baseline model python train_cau

S. Deng 4 Sep 02, 2022
Pytorch implementation of face attention network

Face Attention Network Pytorch implementation of face attention network as described in Face Attention Network: An Effective Face Detector for the Occ

Hooks 312 Dec 09, 2022
This is a Deep Leaning API for classifying emotions from human face and human audios.

Emotion AI This is a Deep Leaning API for classifying emotions from human face and human audios. Starting the server To start the server first you nee

crispengari 5 Oct 02, 2022
An End-to-End Machine Learning Library to Optimize AUC (AUROC, AUPRC).

Logo by Zhuoning Yuan LibAUC: A Machine Learning Library for AUC Optimization Website | Updates | Installation | Tutorial | Research | Github LibAUC a

Optimization for AI 176 Jan 07, 2023
MusicYOLO framework uses the object detection model, YOLOx, to locate notes in the spectrogram.

MusicYOLO MusicYOLO framework uses the object detection model, YOLOX, to locate notes in the spectrogram. Its performance on the ISMIR2014 dataset, MI

Xianke Wang 2 Aug 02, 2022
Classification Modeling: Probability of Default

Credit Risk Modeling in Python Introduction: If you've ever applied for a credit card or loan, you know that financial firms process your information

Aktham Momani 2 Nov 07, 2022
Determined: Deep Learning Training Platform

Determined: Deep Learning Training Platform Determined is an open-source deep learning training platform that makes building models fast and easy. Det

Determined AI 2k Dec 31, 2022
Supervised Classification from Text (P)

MSc-Thesis Module: Masters Research Thesis Language: Python Grade: 75 Title: An investigation of supervised classification of therapeutic process from

Matthew Laws 1 Nov 22, 2021
RefineGNN - Iterative refinement graph neural network for antibody sequence-structure co-design (RefineGNN)

Iterative refinement graph neural network for antibody sequence-structure co-des

Wengong Jin 83 Dec 31, 2022
FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX.

FedJAX: Federated learning with JAX What is FedJAX? FedJAX is a library for developing custom Federated Learning (FL) algorithms in JAX. FedJAX priori

Google 208 Dec 14, 2022
PyTorch Implementations for DeeplabV3 and PSPNet

Pytorch-segmentation-toolbox DOC Pytorch code for semantic segmentation. This is a minimal code to run PSPnet and Deeplabv3 on Cityscape dataset. Shor

Zilong Huang 746 Dec 15, 2022
A Pytorch implementation of "Manifold Matching via Deep Metric Learning for Generative Modeling" (ICCV 2021)

Manifold Matching via Deep Metric Learning for Generative Modeling A Pytorch implementation of "Manifold Matching via Deep Metric Learning for Generat

69 Dec 10, 2022
U^2-Net - Portrait matting This repository explores possibilities of using the original u^2-net model for portrait matting.

U^2-Net - Portrait matting This repository explores possibilities of using the original u^2-net model for portrait matting.

Dennis Bappert 104 Nov 25, 2022
Code for "Unsupervised Source Separation via Bayesian inference in the latent domain"

LQVAE-separation Code for "Unsupervised Source Separation via Bayesian inference in the latent domain" Paper Samples GT Compressed Separated Drums GT

Michele Mancusi 30 Oct 25, 2022
Machine Learning From Scratch. Bare bones NumPy implementations of machine learning models and algorithms with a focus on accessibility. Aims to cover everything from linear regression to deep learning.

Machine Learning From Scratch About Python implementations of some of the fundamental Machine Learning models and algorithms from scratch. The purpose

Erik Linder-Norén 21.8k Jan 09, 2023