Efficient Deep Learning Systems

This repository contains materials for the Efficient Deep Learning Systems course taught at the Faculty of Computer Science of HSE University and Yandex School of Data Analysis.

Syllabus

Week 1: Introduction
- Lecture: Course overview and organizational details. Core concepts of the GPU architecture and CUDA API.
- Seminar: CUDA operations in PyTorch. Introduction to benchmarking.
Week 2: Basics of distributed ML
- Lecture: Introduction to distributed training. Process-based communication. Parameter Server architecture.
- Seminar: Multiprocessing basics. Parallel GloVe training.
Week 3: Data-parallel training and All-Reduce
- Lecture: Data-parallel training of neural networks. All-Reduce and its efficient implementations.
- Seminar: Introduction to PyTorch Distributed. Data-parallel training primitives.
Week 4: Memory-efficient and model-parallel training
Week 5: Profiling DL code, training-time optimizations
Week 6: Basics of Python application deployment
Week 7: Software for serving neural networks
Week 8: Optimizing models for faster inference
Week 9: Experiment tracking, model and data versioning
Week 10: Testing, debugging and monitoring of models

Grading

There will be a total of 4 home assignments (some of them spread over several weeks). The final grade is a weighted sum of per-assignment grades. Please refer to the course page of your institution for details.

Efficient Deep Learning Systems course

Related tags

Overview

Efficient Deep Learning Systems

Syllabus

Grading

Staff

Owner

Max Ryabinin

A very simple tool for situations where optimization with onnx-simplifier would exceed the Protocol Buffers upper file size limit of 2GB, or simply to separate onnx files to any size you want.

Evaluation framework for testing segmentation networks in PyTorch

Code for AutoNL on ImageNet (CVPR2020)

Benchmark tools for Compressive LiDAR-to-map registration

Local Multi-Head Channel Self-Attention for FER2013

PyTorch implementation of Wide Residual Networks with 1-bit weights by McDonnell (ICLR 2018)

Vision Transformer and MLP-Mixer Architectures

Weakly-Supervised Semantic Segmentation Network with Deep Seeded Region Growing (CVPR 2018).

Disentangled Face Attribute Editing via Instance-Aware Latent Space Search, accepted by IJCAI 2021.

Keras community contributions

STMTrack: Template-free Visual Tracking with Space-time Memory Networks

A simple code to convert image format and channel as well as resizing and renaming multiple images.

Tf alloc - Simplication of GPU allocation for Tensorflow2

Using VapourSynth with super resolution models and speeding them up with TensorRT.

An open source library for face detection in images. The face detection speed can reach 1000FPS.

Code and data (Incidents Dataset) for ECCV 2020 Paper "Detecting natural disasters, damage, and incidents in the wild".

Tools for computational pathology

Geometry-Aware Learning of Maps for Camera Localization (CVPR2018)

PyTorch implementation of popular datasets and models in remote sensing

arxiv-sanity, but very lite, simply providing the core value proposition of the ability to tag arxiv papers of interest and have the program recommend similar papers.