Tutorial in Python targeted at Epidemiologists. Will discuss the basics of analysis in Python 3

Overview

Python-for-Epidemiologists

Join the chat at https://gitter.im/zEpid/community DOI

This repository is an introduction to epidemiology analyses in Python. Additionally, the tutorials for my library zEpid are hosted here. For more information on zEpid, see GitHub or ReadTheDocs.

The directory of this guide is

  1. Python Basics
  2. Basics of pandas (data management library)
  3. Epidemiology analyses in Python
    1. Basics
    2. Missing data
    3. Causal inference
      1. Time-fixed treatments
      2. Time-varying treatments
    4. Sensitivity analyses

Required packages for tutorial

To complete the tutorial, user must have the following packages installed: numpy, pandas, zepid, matplotlib, statsmodels, lifelines, and sklearn

IDE (Integrated Development Environment)

No IDE is required to complete the tutorial. All files are available in ipynb also known as jupyter notebooks. Code can be either downloaded or copied from the notebooks.

Here are some IDEs I have used in the past (and what I believe to be their advantages and disadvantages

Rodeo

This is the IDE I used for a long time. It is set up like RStudio

Advantages:

Basically RStudio but for Python, decent interface, easy to run line-by-line, easy to visualize plots (although it encourage bad habits)

Disadvantages:

Does not have all the features of RStudio (will delete changes if closed without saving), sucks up a lot of memory, sometimes the auto-complete would stop working if I hit more than 300+ lines of code, the environment tab is not great (don't expect it to open anything like RStudio)

Aside: their website has great tutorials how to run some basic stuff in Python if you are new to analysis in Python https://rodeo.yhat.com/

jupyter notebooks

Designed to be like a lab notebook, or like R markdown. Supports a pseudo-line-by-line concept Good for writing, since it allows for MarkDown. While I know a lot of people like jupyter, I only really use it for examples of code, not my personal programming. I never liked how it had to open via a Web Browser. I would rather have it be separate program. However, all guides were made using this IDE

PyCharm

This is the IDE I currently use

Advantages:

Easily set up virtual environments, interacts natively with Git, supports different file formats with plug-ins (e.g. .md), enforces certain coding conventions, better debug code features, organization of files under the project tab are convenient

Disadvantages:

Not great for running line-by-line code (it can do it, just not as elegantly), little more hardcore (I wouldn't really consider it a beginner's IDE. It requires some knowledge of set-up of Python)

IDLE

Ships with the basic Python 3.x installation. It is very basic and does not support line-by-line. Wouldn't recommend unless you are just starting with Python and don't want to commit to an IDE yet

Spyder

Ships with conda. Not bad but I didn't use it that much (I couldn't get the hang of it). Similarly it is an RStudio copy. Can't say too much since I haven't used it extensively

Basic Introduction to Python

If you have never used Python before, I have created some introductory materials to Python and the data management library I use, pandas. These are basic guides, but they also point to other resources. Please READ ALL OF THE BELOW BEFORE PROCEEDING.

Installing Python

To install, Python 3.x, we can download it directly from: https://www.python.org/downloads/

The installer provides an option to add Python3 to PATH, it is highly recommended you do this, since it allows you to avoid having to do it manually

Open Command Prompt / Terminal. When opened, type python and this should open Python in the same window. From here, you can quit by typing 'quit()' or closing the window. If this does NOT work, make sure your environmental variable was created properly

Installing Python Packages

Packages are what stores Python functions that we will use. These packages are contributed by various members of the community (including me)) and there is a wide array. To be able to download packages, we need to make sure we have an environmental variable created for python. We will discuss how to install packages

Python 3.x conveniently comes with a package manager. Basically it stores all the packages and we can use it to download new ones or update already downloaded ones.

To download a new package: Open Command Prompt/Terminal and use the following code (we will be installing pandas)

pip install pandas

To update a Python package, type the following command into Command Prompt. For example, we will update our pandas package

pip install pandas --upgrade

That concludes the basics. Please review parts 1 and 2 of the tutorials next

Comments
  • Cochran-Mantel-Haenszel

    Cochran-Mantel-Haenszel

    Thank you @pzivich for this amazing resource. Having the Hernan/Robbins causal model code in python is super helpful... g-estimation!

    I have a request... do you have a Cochran-Mantel-Haenszel script? If you get the chance, please, it would be useful to us to have in this repo. Thank you in advance!

    opened by opioiddatalab 2
  • Slight changes in Incidence Rate Ratio

    Slight changes in Incidence Rate Ratio

    Incidence Ratio Rate Paragraph

    • Fixed repetition
    • T1 & T0 are defined the same way. I believe that T0 is the person-time contributed by people NOT treated with ART
    opened by jaimiles23 0
  • Updates for v0.8.0

    Updates for v0.8.0

    Checklist for various notebooks to update with v0.8.0 release (hasn't released yet)

    • [x] IPTW update. Lots of major changes, so notebook needs to be completely overhauled

    • [x] Demonstrate new diagnostic functions for IPTW, g-formula, AIPW, TMLE

    • [x] Demonstrate g-bound argument

    • [x] Remove TMLE machine learning custom models. This is being removed in favor of cross-fitting. Can leave how to apply for now, but add the warning and mention will be cut in v0.9.0

    opened by pzivich 0
  • Notebooks not rendering in GitHub

    Notebooks not rendering in GitHub

    Sometimes GitHub has trouble rendering the notebooks. AFAIK the rendering system is behind the scenes at GitHub. Others have this same problem across repos and it sometimes occurs to me as well.

    If the notebook won't render in GitHub, you can copy the URL to the notebook you want to view and use the following site to view the notebook: https://nbviewer.jupyter.org/

    opened by pzivich 0
  • Replicate

    Replicate "Causal Inference"

    Issue to track progress on implementation of Hernan and Robins "Causal Inference" chapters

    • [x] Chapter 12: Inverse probability weights

    • [x] Chapter 13: Parametric g-formula

    • [x] Chapter 14: G-estimation of structural nested models

    • [x] ~Chapter 16: G-estimation for IV analysis~

    • [ ] Chapter 17: Causal survival analysis

    • [ ] Part III: Time-varying treatments

    ~G-estimation is not currently implemented. I will need to implement these before chapter 14 can be done.~

    Currently there are no plans to replicate Chapter 15 (propensity scores and regression) or Chapter 16 (instrumental variables) since the first method does not require zEpid and I am unfamiliar with the second. Maybe instrumental variables will be added in the future?

    For Chapter 16, I am considering demonstrating the usage of g-estimation instead of two-stage least-squares. Specifically, using the same data as done in Chapter 16 but following Technical Point 16.3

    enhancement 
    opened by pzivich 0
  • Tutorials

    Tutorials

    On the website, create quick tutorials demonstrating each of the implemented estimators, descriptions of how they work, and why you might want to use them. Might be more digestible than the current docs (also better justify why to choose one over the other)

    Reference to base on https://lifelines.readthedocs.io/en/latest/jupyter_notebooks/Proportional%20hazard%20assumption.html https://github.com/CamDavidsonPilon/lifelines/blob/master/docs/jupyter_notebooks/Proportional%20hazard%20assumption.ipynb

    TODO

    • [x] Basic measures

    • [x] splines

    • [x] IPTW: time-fixed treatment

    • [ ] IPTW: stochastic treatment

    • [ ] IPTW: time-varying treatment

    • [x] IPCW

    • [x] IPMW: single variable

    • [ ] IPMW: monotone

    • [ ] IPMW: nonmonotone (to add after implemented)

    • [x] G-formula: time-fixed binary treatment, binary outcome

    • [x] G-formula: time-fixed categorical treatment, binary outcome

    • [ ] G-formula: time-fixed continuous treatment, binary outcome (to add after implemented)

    • [x] G-formula: time-fixed binary treatment, continuous outcome

    • [x] G-formula: Monte Carlo

    • [x] G-formula: Iterative Conditional

    • [x] G-estimation of SNM

    • [x] AIPTW

    • [ ] AIPMW

    • [x] TMLE

    • [x] TMLE: stochastic treatment

    • [ ] LTMLE (to add after implemented)

    • [x] Quantitative bias analysis

    • [x] Functional form assessment

    • [x] Generalizability

    • [ ] Transportability (IPSW, g-transport, AIPSW)

    • [x] Monte Carlo g-formula by-hand (helps to explain underlying process)

    opened by pzivich 1
Releases(v0.8.0)
Owner
Paul Zivich
Epidemiology post-doc working in epidemiologic methods and infectious diseases.
Paul Zivich
OCR Streamlit App is used to extract text from images using python's easyocr, pytorch and streamlit packages

OCR-Streamlit-App OCR Streamlit App is used to extract text from images using python's easyocr, pytorch and streamlit packages OCR app gets an image a

Siva Prakash 5 Apr 05, 2022
Pytorch code for ICRA'21 paper: "Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation"

Hierarchical Cross-Modal Agent for Robotics Vision-and-Language Navigation This repository is the pytorch implementation of our paper: Hierarchical Cr

43 Nov 21, 2022
ML-based medical imaging using Azure

Disclaimer This code is provided for research and development use only. This code is not intended for use in clinical decision-making or for any other

Microsoft Azure 68 Dec 23, 2022
Convolutional Neural Network for Text Classification in Tensorflow

This code belongs to the "Implementing a CNN for Text Classification in Tensorflow" blog post. It is slightly simplified implementation of Kim's Convo

Denny Britz 5.5k Jan 02, 2023
Code for the ICCV2021 paper "Personalized Image Semantic Segmentation"

PSS: Personalized Image Semantic Segmentation Paper PSS: Personalized Image Semantic Segmentation Yu Zhang, Chang-Bin Zhang, Peng-Tao Jiang, Ming-Ming

张宇 15 Jul 09, 2022
Code for "FGR: Frustum-Aware Geometric Reasoning for Weakly Supervised 3D Vehicle Detection", ICRA 2021

FGR This repository contains the python implementation for paper "FGR: Frustum-Aware Geometric Reasoning for Weakly Supervised 3D Vehicle Detection"(I

Yi Wei 31 Dec 08, 2022
git《FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding》(CVPR 2021) GitHub: [fig8]

FSCE: Few-Shot Object Detection via Contrastive Proposal Encoding (CVPR 2021) This repo contains the implementation of our state-of-the-art fewshot ob

233 Dec 29, 2022
Crosslingual Segmental Language Model

Crosslingual Segmental Language Model This repository contains the code from Multilingual unsupervised sequence segmentation transfers to extremely lo

C.M. Downey 1 Jun 13, 2022
Code of our paper "Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning"

CCOP Code of our paper Contrastive Object-level Pre-training with Spatial Noise Curriculum Learning Requirement Install OpenSelfSup Install Detectron2

Chenhongyi Yang 21 Dec 13, 2022
Code for CVPR2021 "Visualizing Adapted Knowledge in Domain Transfer". Visualization for domain adaptation. #explainable-ai

Visualizing Adapted Knowledge in Domain Transfer @inproceedings{hou2021visualizing, title={Visualizing Adapted Knowledge in Domain Transfer}, auth

Yunzhong Hou 80 Dec 25, 2022
A NSFW content filter.

Project_Nfilter A NSFW content filter. With a motive of minimizing the spreads and leakage of NSFW contents on internet and access to others devices ,

1 Jan 20, 2022
Awesome Long-Tailed Learning

Awesome Long-Tailed Learning This repo pays specially attention to the long-tailed distribution, where labels follow a long-tailed or power-law distri

Stomach_ache 284 Jan 06, 2023
This repository is an open-source implementation of the ICRA 2021 paper: Locus: LiDAR-based Place Recognition using Spatiotemporal Higher-Order Pooling.

Locus This repository is an open-source implementation of the ICRA 2021 paper: Locus: LiDAR-based Place Recognition using Spatiotemporal Higher-Order

Robotics and Autonomous Systems Group 96 Dec 15, 2022
Few-Shot Graph Learning for Molecular Property Prediction

Few-shot Graph Learning for Molecular Property Prediction Introduction This is the source code and dataset for the following paper: Few-shot Graph Lea

Zhichun Guo 94 Dec 12, 2022
Beyond imagenet attack (accepted by ICLR 2022) towards crafting adversarial examples for black-box domains.

Beyond ImageNet Attack: Towards Crafting Adversarial Examples for Black-box Domains (ICLR'2022) This is the Pytorch code for our paper Beyond ImageNet

Alibaba-AAIG 37 Nov 23, 2022
S2s2net - Sentinel-2 Super-Resolution Segmentation Network

S2S2Net Sentinel-2 Super-Resolution Segmentation Network Getting started Install

Wei Ji 10 Nov 10, 2022
Optical machine for senses sensing using speckle and deep learning

# Senses-speckle [Remote Photonic Detection of Human Senses Using Secondary Speckle Patterns](https://doi.org/10.21203/rs.3.rs-724587/v1) paper Python

Zeev Kalyuzhner 0 Sep 26, 2021
[ICCV'21] NEAT: Neural Attention Fields for End-to-End Autonomous Driving

NEAT: Neural Attention Fields for End-to-End Autonomous Driving Paper | Supplementary | Video | Poster | Blog This repository is for the ICCV 2021 pap

254 Jan 02, 2023
[ICCV 2021] Deep Hough Voting for Robust Global Registration

Deep Hough Voting for Robust Global Registration, ICCV, 2021 Project Page | Paper | Video Deep Hough Voting for Robust Global Registration Junha Lee1,

Junha Lee 10 Dec 02, 2022
🔪 Elimination based Lightweight Neural Net with Pretrained Weights

ELimNet ELimNet: Eliminating Layers in a Neural Network Pretrained with Large Dataset for Downstream Task Removed top layers from pretrained Efficient

snoop2head 4 Jul 12, 2022