Code to produce syntactic representations that can be used to study syntax processing in the human brain

Overview

Can fMRI reveal the representation of syntactic structure in the brain?

The code base for our paper on understanding syntactic representations in the human brain using naturalistic fMRI data. We explain how to reproduce our results in detail and also point to our preprocessed data. Please cite this work if you use our code:

@inproceedings{reddywehbe2021,
  title={Can fMRI reveal the representation of syntactic structure in the brain?},
  author={Reddy, Aniketh Janardhan and Wehbe, Leila},
  booktitle={Thirty-Fifth Conference on Neural Information Processing Systems},
  year={2021}
}

Dependencies

This work used the Bridges system, which is supported by NSF award number ACI-1445606, at the Pittsburgh Supercomputing Center (PSC). All of the results were obtained using a machine with 14 CPU cores, 128 GB RAM and a CUDA-capable GPU. Our analyses were performed using iPython notebooks with Python3.6 kernels. We have tested this code on CentOS Linux 8. We recommend using a Linux-based environment to run our code. The analysis pipeline is fairly compute-intensive and it took us about 4 days to run it. Expect the runtime to be significantly longer if you are using a system with less than 8 cores. Our code does not make very heavy use of a GPU. Thus, an entry level graphics card such as an Nvidia RTX 2060 should be sufficient. It is possible to run the code even without a GPU but it might take longer to generate some features.

The Python packages needed to run our code can be installed by running the install_python_dependencies.sh script:

bash install_python_dependencies.sh

You will also have to install FSL (https://fsl.fmrib.ox.ac.uk/fsl/fslwiki/FSL) which is needed to transform results to MNI space.

The graph embeddings-based features used in our paper are computed using sub2vec [1]. Please download the code written by the original authors for this algorithm from here http://people.cs.vt.edu/~bijaya/codes/sub2vec.zip and extract the code to a folder called sub2vec.

Clone the code for the incremental top-down parser [2, 3] from this repo - https://github.com/roarkbr/incremental-top-down-parser to the folder containing our code. This is needed to generate the syntactic surprisal and InConTreGE features. This code must be in a folder called incremental-top-down-parser.

Finally, download the ROIs created by Fedorenko et al. (2010) [4] from here - https://osf.io/2gaw3/ and extract the files to the folder containing our code. These are required to create the image of the ROIs.

Reproducing Our Results

The preprocessed fMRI data we use have been uploaded here - https://drive.google.com/file/d/1aYEZZSyrlo0UqswDBUiGzE3kGl3RcCn8/view?usp=sharing. Please download the file and extract it to the directory in which the code has been cloned. The data should be saved in a folder called sub_space_data.

Note that we cannot provide the anatomical data needed to visualize subject space results to protect the anonymity of the subjects. However, we provide the binary masks and transforms needed to transform subject space results to MNI space. These were obtained using pycortex.

Also, we provide all of the main files needed to generate our figures and tables since running our full pipeline can take a long time. These include the features we generate (in the features folder), the R^2 scores and the significance testing results (in the predictions and predictions_mni folders) among others.

Please follow these steps to reproduce our results using this codebase:

  1. Upon extracting the aforementioned file, the preprocessed fMRI data can be found in the folder called sub_space_data.

  2. The text which is presented to the subjects is in the chapter9.txt file. The string on each line of the file is sequentially presented (there are 5176 lines). The + symbol is a fixation cross that is periodically shown to the subjects. Since we use word-level features, all of the files which contain these features are numpy arrays of the form (5176, number of feature dimensions). The rows which correspond to the presentation of a + are filled with zeros.

  3. Generate the complexity metrics - Node Count (NC), Syntactic Surprisal (SS), Word Frequency (WF) and Word Length (WL), for every presented word by running the generate_node_count.ipynb, generate_syntactic_surprisal.ipynb, generate_word_frequencies_and_word_lengths.ipynb notebooks respectively. The outputs are stored in the features folder as node_count.npy, syntactic_surprisal.npy, word_frequency.npy and word_length.npy.

  4. Generate the POS tags of the presented words using the generate_pos_tags.ipynb notebook. The output is stored in the features folder as pos_tags.npy.

  5. Generate the DEP tags of the presented words using the generate_dep_tags.ipynb notebook. The output is stored in the features folder as dep_tags.npy.

  6. Generate the punctuation-based feature space by running the generate_punct.ipynb notebook. This feature space is extracted from POS and DEP tags since it is just a subset of these features. The output is stored in the features folder as punct_final.npy.

  7. In order to generate the ConTreGE Comp vectors, we first need to generate the subtrees to be encoded. This is done by running the generate_contrege_comp_subtrees.ipynb notebook. These subtrees are stored in the contrege_comp_subtrees folder. Then, we run the generate_contrege_comp_vectors_using_sub2vec.ipynb notebook to generate 5 sets of ConTreGE Comp vectors using sub2vec. These are stored in the features folder (called as contrege_comp_set_0.npy, contrege_comp_set_1.npy, contrege_comp_set_2.npy, contrege_comp_set_3.npy, contrege_comp_set_4.npy). We include all of the sets we generated and used in our analyses since these vectors are stochastic and can vary from run to run.

  8. We need to follow steps similar to those used to generate ConTreGE Comp so as to generate the ConTreGE Incomp vectors. We first need to generate the subtrees to be encoded by running the generate_contrege_incomp_subtrees.ipynb notebook. These subtrees are stored in the contrege_incomp_subtrees folder. Then, we run the generate_contrege_incomp_vectors_using_sub2vec.ipynb notebook to generate 5 sets of ConTreGE Incomp vectors using sub2vec. These are stored in the features folder (called as contrege_incomp_set_0.npy, contrege_incomp_set_1.npy, contrege_incomp_set_2.npy, contrege_incomp_set_3.npy, contrege_incomp_set_4.npy). Again, we include all of the sets we generated and used in our analyses since these vectors are also stochastic and can vary from run to run.

  9. The InConTreGE vectors are generated using the partial parses output by the aforementioned incremental top-down parser. To get the subtrees which are representative of these partial parses, run the generate_incontrege_subtrees.ipynb notebook. Then, run the generate_incontrege_vectors_using_sub2vec.ipynb notebook to generate 5 sets of InConTreGE vectors using sub2vec. These are stored in the features folder (called as incontrege_set_0.npy, incontrege_set_1.npy, incontrege_set_2.npy, incontrege_set_3.npy, incontrege_set_4.npy). We include all of the sets we generated and used in our analyses since these vectors are stochastic and can vary from run to run.

  10. To generate the BERT embeddings-based semantic features, run the generate_incremental_bert_embeddings.ipynb notebook. The output is stored in the features folder as incremental_bert_embeddings_layer12_PCA_dims_15.npy.

  11. Now that all of the individual features are ready, we can build the hierarchical feature groups used in the paper. Run the generate_hierarchical_feature_groups.ipynb notebook to build them. Note that the punctuation-based feature is not explicitly added to feature groups that contain POS and DEP tags. This is because POS and DEP tags already contain the punctuation feature in them. This step generates the following important files in the features folder:

    1. node_count_punct.npy = {NC, PU}
    2. syntactic_surprisal_punct.npy = {SS, PU}
    3. word_frequency_punct.npy = {WF, PU}
    4. word_length_punct.npy = {WL, PU}
    5. all_complexity_metrics_punct.npy = {CM, PU}
    6. pos_dep_tags_all_complexity_metrics.npy = {PD, CM, PU}
    7. contrege_comp_set_X_pos_dep_tags_all_complexity_metrics.npy = {CC, PD, CM, PU}
    8. contrege_incomp_set_X_pos_dep_tags_all_complexity_metrics.npy = {CI, PD, CM, PU}
    9. incontrege_set_X_pos_dep_tags_all_complexity_metrics.npy = {INC, PD, CM, PU}
    10. bert_PCA_dims_15_contrege_incomp_set_X_pos_dep_tags_node_count.npy = {BERT, CI, PD, CM, PU}
  12. We can then start training Ridge regression models and using these trained models to make predictions (training and prediction is done in a cross validated fashion as described in the paper). Run the predictions_master_script.ipynb notebook in order to generate all of the predictions. Predictions made using each feature group will be stored in separate subfolders in the predictions folder and will be in subject space (these subfolders will be named after the numpy files used to make the predictions). The R^2 scores for each subject and feature group are stored in the files of the form SubjectName_r2s.npy.

  13. The prediction results obtained using the ConTreGE Comp, ConTreGE Incomp and InConTreGE vectors need to be averaged across the 5 sets. Run the aggregate_contrege_results_across_sets.ipynb notebook to do this. The script outputs the averaged results to subfolders that start with the aggregated prefix in the predictions folder.

  14. After obtaining the predictions, we can start testing our results for significance. First, we test the significance of the R^2 scores obtained using punctuations only by performing a permutation test. This test is run by executing the significance_testing_permutation.ipynb notebook. Running this notebook will generate files of the form SubjectName_sig.npy in the punct_final subfolder of predictions. These subject space files indicate voxels for which the R^2 scores produced using punctuations are significant.

  15. Then, we test for the significance of the differences in R^2 scores between consecutive hierarchical feature groups by running the difference_significance_testing_bootstrap.ipynb notebook. This generates subfolders of the form {features in group 1}_diff_{features in group 2} that contain files of the form SubjectName_sig_boot.npy in the predictions folder. These subject space files indicate voxels for which the difference in R^2 scores between group 1 and group 2 (= R^2_group1 - R^2_group2) are significant. Note that:

    1. node_count_punct_diff_punct_final = {NC, PU} - {PU}
    2. syntactic_surprisal_punct_diff_punct_final = {SS, PU} - {PU}
    3. word_frequency_punct_diff_punct_final = {WF, PU} - {PU}
    4. word_length_punct_diff_punct_final = {WL, PU} - {PU}
    5. all_complexity_metrics_punct_diff_punct_final = {CM, PU} - {PU}
    6. pos_dep_tags_all_complexity_metrics_diff_all_complexity_metrics_punct = {PD, CM, PU} - {CM, PU}
    7. aggregated_contrege_comp_pos_dep_tags_all_complexity_metrics_diff_pos_dep_tags_all_complexity_metrics = {CC, PD, CM, PU} - {PD, CM, PU}
    8. aggregated_contrege_incomp_pos_dep_tags_all_complexity_metrics_diff_pos_dep_tags_all_complexity_metrics = {CI, PD, CM, PU} - {PD, CM, PU}
    9. aggregated_incontrege_pos_dep_tags_all_complexity_metrics_diff_pos_dep_tags_all_complexity_metrics = {INC, PD, CM, PU} - {PD, CM, PU}
    10. aggregated_bert_PCA_dims_15_contrege_incomp_pos_dep_tags_node_count_diff_aggregated_contrege_incomp_pos_dep_tags_node_count = {BERT, CI, PD, CM, PU} - {CI, PD, CM, PU}
  16. False Discovery Rate correction is then performed for all of the significance tests as described in the paper by running the perform_FDR_correction.ipynb notebook. For the punctuation feature, we obtain files of the form SubjectName_sig_group_corrected.npy and for all the other tests, files of the form SubjectName_sig_bootstrap_group_corrected.npy are obtained. These files are stored in the same subfolders of the predictions folder that contain the uncorrected p-val files.

  17. To generate the brain maps shown in the paper, we need to transform the significance testing results and R^2 scores that are in subject space to MNI space. Run the mni_transform.ipynb notebook to perform this transformation. The transformed files are saved in the predictions_mni folder. Note that running the aforementioned notebook requires FSL to be installed.

  18. Finally, running the create_figures.ipynb notebook generates the figures in our paper in a folder called figures. R^2+ figures are stored in r2plus_figures subfolder and the significance testing results are stored in the sig_figures subfolder. blank_plot_of_rois.png is a plot showing the ROIs and the ROI analysis figures are saved in the roi_figures subfolder.

The syntactic information analysis can be carried out by following these steps:

  1. Run the generate_ancestor_data_for_information_analysis.ipynb notebook to generate numpy files that encode the ancestor information. These files are stored in the ancestor_information_analysis folder.

  2. Then run the syntactic_information_analysis.ipynb notebook to perform the prediction analysis. The notebook generates a CSV file called final_syntactic_information_analysis_results.csv that contains the prediction accuracies and the associated p-vals. The last cell of the notebook shows the label distribution for each level.

To test that the BERT embeddings are better predictors of GloVe-based semantic vectors (extracted from spaCy) than the ConTreGE vectors, we first need to extract the GloVe-based semantic vectors by running the generate_spacy_embeddings.ipynb notebook. Then run the compare_glove_vectors_predictivity.ipynb notebook to train and test RidgeCV models that predict the GloVe-based semantic vectors. The outputs of the last two cells show that BERT embeddings are much predictors of these GloVe-based semantic vectors when compared to the ConTreGE vectors.

References

  1. Adhikari, Bijaya, et al. "Sub2vec: Feature learning for subgraphs." Pacific-Asia Conference on Knowledge Discovery and Data Mining. Springer, Cham, 2018.

  2. Roark, Brian. "Probabilistic top-down parsing and language modeling." Computational linguistics 27.2 (2001): 249-276.

  3. Roark, Brian, et al. "Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing." Proceedings of the 2009 conference on empirical methods in natural language processing. 2009.

  4. Fedorenko, Evelina, et al. "New method for fMRI investigations of language: defining ROIs functionally in individual subjects." Journal of neurophysiology 104.2 (2010): 1177-1194.

Owner
Aniketh Janardhan Reddy
Computer Science PhD Student, UC Berkeley
Aniketh Janardhan Reddy
Lucid Sonic Dreams syncs GAN-generated visuals to music.

Lucid Sonic Dreams Lucid Sonic Dreams syncs GAN-generated visuals to music. By default, it uses NVLabs StyleGAN2, with pre-trained models lifted from

731 Jan 02, 2023
A transformer model to predict pathogenic mutations

MutFormer MutFormer is an application of the BERT (Bidirectional Encoder Representations from Transformers) NLP (Natural Language Processing) model wi

Wang Genomics Lab 2 Nov 29, 2022
Personalized Federated Learning using Pytorch (pFedMe)

Personalized Federated Learning with Moreau Envelopes (NeurIPS 2020) This repository implements all experiments in the paper Personalized Federated Le

Charlie Dinh 226 Dec 30, 2022
Unofficial PyTorch Implementation of Multi-Singer

Multi-Singer Unofficial PyTorch Implementation of Multi-Singer: Fast Multi-Singer Singing Voice Vocoder With A Large-Scale Corpus. Requirements See re

SunMail-hub 123 Dec 28, 2022
Learning-Augmented Dynamic Power Management

Learning-Augmented Dynamic Power Management This repository contains source code accompanying paper Learning-Augmented Dynamic Power Management with M

Adam 0 Feb 22, 2022
Official repository for the paper, MidiBERT-Piano: Large-scale Pre-training for Symbolic Music Understanding.

MidiBERT-Piano Authors: Yi-Hui (Sophia) Chou, I-Chun (Bronwin) Chen Introduction This is the official repository for the paper, MidiBERT-Piano: Large-

137 Dec 15, 2022
Start-to-finish tutorial for interactive music co-creation in PyTorch and Tensorflow.js

Start-to-finish tutorial for interactive music co-creation in PyTorch and Tensorflow.js

Chris Donahue 98 Dec 14, 2022
social humanoid robots with GPGPU and IoT

Social humanoid robots with GPGPU and IoT Social humanoid robots with GPGPU and IoT Paper Authors Mohsen Jafarzadeh, Stephen Brooks, Shimeng Yu, Balak

0 Jan 07, 2022
HNECV: Heterogeneous Network Embedding via Cloud model and Variational inference

HNECV This repository provides a reference implementation of HNECV as described in the paper: HNECV: Heterogeneous Network Embedding via Cloud model a

4 Jun 28, 2022
WebUAV-3M: A Benchmark Unveiling the Power of Million-Scale Deep UAV Tracking

WebUAV-3M: A Benchmark Unveiling the Power of Million-Scale Deep UAV Tracking [Paper Link] Abstract In this work, we contribute a new million-scale Un

25 Jan 01, 2023
NeurIPS 2021 Datasets and Benchmarks Track

AP-10K: A Benchmark for Animal Pose Estimation in the Wild Introduction | Updates | Overview | Download | Training Code | Key Questions | License Intr

AP-10K 82 Dec 11, 2022
A high-performance anchor-free YOLO. Exceeding yolov3~v5 with ONNX, TensorRT, NCNN, and Openvino supported.

YOLOX is an anchor-free version of YOLO, with a simpler design but better performance! It aims to bridge the gap between research and industrial communities. For more details, please refer to our rep

7.7k Jan 06, 2023
Minimisation of a negative log likelihood fit to extract the lifetime of the D^0 meson (MNLL2ELDM)

Minimisation of a negative log likelihood fit to extract the lifetime of the D^0 meson (MNLL2ELDM) Introduction The average lifetime of the $D^{0}$ me

Son Gyo Jung 1 Dec 17, 2021
Application of K-means algorithm on a music dataset after a dimensionality reduction with PCA

PCA for dimensionality reduction combined with Kmeans Goal The Goal of this notebook is to apply a dimensionality reduction on a big dataset in order

Arturo Ghinassi 0 Sep 17, 2022
Repo 4 basic seminar §How to make human machine readable"

WORK IN PROGRESS... Notebooks from the Seminar: Human Machine Readable WS21/22 Introduction into programming Georg Trogemann, Christian Heck, Mattis

experimental-informatics 3 May 29, 2022
i-RevNet Pytorch Code

i-RevNet: Deep Invertible Networks Pytorch implementation of i-RevNets. i-RevNets define a family of fully invertible deep networks, built from a succ

Jörn Jacobsen 378 Dec 06, 2022
A PyTorch Implementation of Single Shot MultiBox Detector

SSD: Single Shot MultiBox Object Detector, in PyTorch A PyTorch implementation of Single Shot MultiBox Detector from the 2016 paper by Wei Liu, Dragom

Max deGroot 4.8k Jan 07, 2023
This repository is an open-source implementation of the ICRA 2021 paper: Locus: LiDAR-based Place Recognition using Spatiotemporal Higher-Order Pooling.

Locus This repository is an open-source implementation of the ICRA 2021 paper: Locus: LiDAR-based Place Recognition using Spatiotemporal Higher-Order

Robotics and Autonomous Systems Group 96 Dec 15, 2022
MazeRL is an application oriented Deep Reinforcement Learning (RL) framework

MazeRL is an application oriented Deep Reinforcement Learning (RL) framework, addressing real-world decision problems. Our vision is to cover the complete development life cycle of RL applications ra

EnliteAI GmbH 222 Dec 24, 2022
Semi-supervised Domain Adaptation via Minimax Entropy

Semi-supervised Domain Adaptation via Minimax Entropy (ICCV 2019) Install pip install -r requirements.txt The code is written for Pytorch 0.4.0, but s

Vision and Learning Group 243 Jan 09, 2023