Deep Learning Theory

Overview

Deep Learning Theory

整理了一些深度学习的理论相关内容,持续更新。

Overview

  1. Recent advances in deep learning theory 总结了目前深度学习理论研究的六个方向的一些结果,概述型,没做深入探讨(2021)。

    • 1.1 complexity and capacity-basedapproaches for analyzing the generalizability of deep learning;

    • 1.2 stochastic differential equations andtheir dynamic systems for modelling stochastic gradient descent and its variants, which characterizethe optimization and generalization of deep learning, partially inspired by Bayesian inference;

    • 1.3 thegeometrical structures of the loss landscape that drives the trajectories of the dynamic systems;

    • 1.4 theroles of over-parameterization of deep neural networks from both positive and negative perspectives;

    • 1.5 theoretical foundations of several special structures in network architectures;

    • 1.6 the increasinglyintensive concerns in ethics and security and their relationships with generalizability

Course

  1. Theory of Deep LearningTTIC,西北大学等组织的一系列课程和讲座,基础课程涉及DL的基础(符号化,简化后的数学问题和结论),信息论和学习,统计和计算,信息论,统计学习和强化学习(2020)。

  2. MathsDL-spring19,MathDL系列,18,19,20年均有。

    • 3.1 Geometry of Data

      • Euclidean Geometry: transportation metrics, CNNs , scattering.
      • Non-Euclidean Geometry: Graph Neural Networks.
      • Unsupervised Learning under Geometric Priors (Implicit vs explicit models, microcanonical, transportation metrics).
      • Applications and Open Problems: adversarial examples, graph inference, inverse problems.
    • 3.2 Geometry of Optimization and Generalization

      • Stochastic Optimization (Robbins & Munro, Convergence of SGD)
      • Stochastic Differential Equations (Fokker-Plank, Gradient Flow, Langevin + + Dynamics, links with SGD; open problems) Dynamics of Neural Network Optimization (Mean Field Models using Optimal Transport, Kernel Methods)
      • Landscape of Deep Learning Optimization (Tensor/Matrix factorization, Deep Nets; open problems).
      • Generalization in Deep Learning.
    • 3.3 Open qustions on Reinforcement Learning

Architecture

  1. Partial Differential Equations is All You Need for Generating Neural Architectures -- A Theory for Physical Artificial Intelligence Systems 将统计物理的反应扩散方程,量子力学中的薛定谔方程,傍轴光学中的亥姆霍兹方程统一整合到神经网络偏微分方程中(NPDE),利用有限元方法找到数值解,从离散过程中,构造了多层感知,卷积网络,和循环网络,并提供了优化方法L-BFGS等,主要是建立了经典物理模型和经典神经网络的联系(2021)。

Approximation

  1. NN Approximation Theory

Optimization

  1. SGD

  2. offconvex几个学术工作者维护的AI博客。

Geometry

  1. Optima transmission

Book

  1. Theory of Deep Learning(draft)Rong Ge 等(2019)。

  2. Spectral Learning on Matrices and TensorsMajid Janzamin等(2020)

  3. Deep Learning Architectures A Mathematical Approach(2020),你可以libgen获取,内容如其名字,大概包含:工业问题,DL基础(激活,结构,优化等),函数逼近,万有逼近,RELU等逼近新研究,函数表示,以及两大方向,信息角度,几何角度等相关知识,实际场景中的卷积,池化,循环,生成,随机网络等具体实用内容的数学化,另外附录集合论,测度论,概率论,泛函,实分析等基础知识。

  4. The Principles of Deep Learning Theory(2021)Daniel A. Roberts and Sho Yaida(mit),Beginning from a first-principles component-level picture of networks,本书解释了如何通过求解层到层迭代方程和非线性学习动力学来确定训练网络输出的准确描述。一个主要的结果是网络的预测是由近高斯分布描述的,网络的深度与宽度的纵横比控制着与无限宽度高斯描述的偏差。本书解释了这些有效深度网络如何从训练中学习非平凡的表示,并更广泛地分析非线性模型的表示学习机制。从近内核方法的角度来看,发现这些模型的预测对底层学习算法的依赖可以用一种简单而通用的方式来表达。为了获得这些结果,作者开发了表示组流(RG 流)的概念来表征信号通过网络的传播。通过将网络调整到临界状态,他们为梯度爆炸和消失问题提供了一个实用的解决方案。作者进一步解释了 RG 流如何导致近乎普遍的行为,从而可以将由不同激活函数构建的网络做类别划分。Altogether, they show that the depth-to-width ratio governs the effective model complexity of the ensemble of trained networks。利用信息理论,作者估计了模型性能最好的最佳深宽比,并证明了残差连接能将深度推向任意深度。利用以上理论工具,就可以更加细致的研究架构的归纳偏差,超参数,优化。

  5. Physics-based Deep Learning(2021)N. Thuerey, P. Holl,etc.github resources深度学习与物理学的联系。比如基于物理的损失函数,可微流体模拟,逆问题的求解,Navier-Stokes方程的前向模拟,Controlling Burgers’ Equation和强化学习的关系等。

Session

  1. Foundations of Deep Learning(2019),西蒙研究中心会议。
  2. Deep Learning Theory 4(2021, ICML)Claire Monteleoni主持...,深度学习理论会议4,包含论文和视频。
  3. Deep Learning Theory 5 (2021,ICML)MaYi主持...,深度学习理论会议5,包含论文和视频。

Others

  1. Theoretical issues in deep networks 表明指数型损失函数中存在隐式的正则化,其优化的结果和一般损失函数优化结果一致,优化收敛结果和梯度流的迹有关,目前还不能证明哪个结果最优(2020)。
  2. The Dawning of a New Erain Applied MathematicsWeinan E关于在DL的新处境下结合历史的工作范式给出的指导性总结(2021)。
  3. Mathematics of deep learning from Newton Institute
  4. DEEP NETWORKS FROM THE PRINCIPLE OF RATE REDUCTION,白盒神经网络。
  5. redunet_paper白盒神经网络代码。
  6. Theory of Deep Convolutional Neural Networks:Downsampling下采样的数学分析Ding-Xuan Zhou(2020)
  7. Theory of deep convolutional neural networks II: Spherical analysis还有III:radial functions 逼近,(2020)。不过这些工作到底如何,只是用数学转换了一下,理论上没做过多贡献,或者和实际结合没难么紧密,还不得而知。
  8. The Modern Mathematics of Deep Learning(2021)主要是deep laerning的数学分析描述,涉及的问题包括:超参数网络的通用能力,深度在深度模型中的核心作用,深度学习对维度灾难的克服,优化在非凸优化问题的成功,学习的表示特征的数学分析,为何深度模型在物理问题上有超常表现,模型架构中的哪些因素以何种方式影响不同任务的学习中的不同方面。
Tensorflow 2.x based implementation of EDSR, WDSR and SRGAN for single image super-resolution

Single Image Super-Resolution with EDSR, WDSR and SRGAN A Tensorflow 2.x based implementation of Enhanced Deep Residual Networks for Single Image Supe

Martin Krasser 1.3k Jan 06, 2023
The repository offers the official implementation of our BMVC 2021 paper in PyTorch.

CrossMLP Cascaded Cross MLP-Mixer GANs for Cross-View Image Translation Bin Ren1, Hao Tang2, Nicu Sebe1. 1University of Trento, Italy, 2ETH, Switzerla

Bingoren 16 Jul 27, 2022
Experiments with Fourier layers on simulation data.

Factorized Fourier Neural Operators This repository contains the code to reproduce the results in our NeurIPS 2021 ML4PS workshop paper, Factorized Fo

Alasdair Tran 57 Dec 25, 2022
6D Grasping Policy for Point Clouds

GA-DDPG [website, paper] Installation git clone https://github.com/liruiw/GA-DDPG.git --recursive Setup: Ubuntu 16.04 or above, CUDA 10.0 or above, py

Lirui Wang 48 Dec 21, 2022
Image Segmentation Evaluation

Image Segmentation Evaluation Martin Keršner, [email protected] Evaluation

Martin Kersner 273 Oct 28, 2022
This repository contains an overview of important follow-up works based on the original Vision Transformer (ViT) by Google.

This repository contains an overview of important follow-up works based on the original Vision Transformer (ViT) by Google.

75 Dec 02, 2022
Bottom-up Human Pose Estimation

Introduction This is the official code of Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation. This paper has been accepted to CVPR2

108 Dec 01, 2022
FSL-Mate: A collection of resources for few-shot learning (FSL).

FSL-Mate is a collection of resources for few-shot learning (FSL). In particular, FSL-Mate currently contains FewShotPapers: a paper list which tracks

Yaqing Wang 1.5k Jan 08, 2023
A Large Scale Benchmark for Individual Treatment Effect Prediction and Uplift Modeling

large-scale-ITE-UM-benchmark This repository contains code and data to reproduce the results of the paper "A Large Scale Benchmark for Individual Trea

10 Nov 19, 2022
The official repository for "Revealing unforeseen diagnostic image features with deep learning by detecting cardiovascular diseases from apical four-chamber ultrasounds"

Revealing unforeseen diagnostic image features with deep learning by detecting cardiovascular diseases from apical four-chamber ultrasounds The why Im

3 Mar 29, 2022
The official PyTorch implementation for the paper "sMGC: A Complex-Valued Graph Convolutional Network via Magnetic Laplacian for Directed Graphs".

Magnetic Graph Convolutional Networks About The official PyTorch implementation for the paper sMGC: A Complex-Valued Graph Convolutional Network via M

3 Feb 25, 2022
Google Brain - Ventilator Pressure Prediction

Google Brain - Ventilator Pressure Prediction https://www.kaggle.com/c/ventilator-pressure-prediction The ventilator data used in this competition was

Samuele Cucchi 1 Feb 11, 2022
A multi-entity Transformer for multi-agent spatiotemporal modeling.

baller2vec This is the repository for the paper: Michael A. Alcorn and Anh Nguyen. baller2vec: A Multi-Entity Transformer For Multi-Agent Spatiotempor

Michael A. Alcorn 56 Nov 15, 2022
Fader Networks: Manipulating Images by Sliding Attributes - NIPS 2017

FaderNetworks PyTorch implementation of Fader Networks (NIPS 2017). Fader Networks can generate different realistic versions of images by modifying at

Facebook Research 753 Dec 23, 2022
Learning Temporal Consistency for Low Light Video Enhancement from Single Images (CVPR2021)

StableLLVE This is a Pytorch implementation of "Learning Temporal Consistency for Low Light Video Enhancement from Single Images" in CVPR 2021, by Fan

99 Dec 19, 2022
CLIP (Contrastive Language–Image Pre-training) trained on Indonesian data

CLIP-Indonesian CLIP (Radford et al., 2021) is a multimodal model that can connect images and text by training a vision encoder and a text encoder joi

Galuh 17 Mar 10, 2022
Monocular Depth Estimation - Weighted-average prediction from multiple pre-trained depth estimation models

merged_depth runs (1) AdaBins, (2) DiverseDepth, (3) MiDaS, (4) SGDepth, and (5) Monodepth2, and calculates a weighted-average per-pixel absolute dept

Pranav 39 Nov 21, 2022
PyTorch implementation for the visual prior component (i.e. perception module) of the Visually Grounded Physics Learner [Li et al., 2020].

VGPL-Visual-Prior PyTorch implementation for the visual prior component (i.e. perception module) of the Visually Grounded Physics Learner (VGPL). Give

Toru 8 Dec 29, 2022
Gym environments used in the paper: "Developmental Reinforcement Learning of Control Policy of a Quadcopter UAV with Thrust Vectoring Rotors"

gym_multirotor Gym to train reinforcement learning agents on UAV platforms Quadrotor Tiltrotor Requirements This package has been tested on Ubuntu 18.

Aditya M. Deshpande 19 Dec 29, 2022