PyGRANSO: A PyTorch-enabled port of GRANSO with auto-differentiation

Overview

PyGRANSO

PyGRANSO

PyGRANSO: A PyTorch-enabled port of GRANSO with auto-differentiation

Please check https://ncvx.org/PyGRANSO for detailed instructions (introduction, installation, settings, examples...).

PyGRANSO is AGPL v3.0 licensed, but it also contains a small portion of GPL code. Please see the LICENSE_INFO folder for more information.

Brief Introduction

Optimizing nonconvex (NCVX) problems, especially nonsmooth and constrained ones, is an essential part of machine learning. However, it can be hard to reliably solve such problems without optimization expertise. Existing general-purpose NCVX optimization packages are powerful but typically cannot handle nonsmoothness. GRANSO is among the first optimization solvers targeting general nonsmooth NCVX problems with nonsmooth constraints, but, as it is implemented in MATLAB and requires the user to provide analytical gradients, GRANSO is often not a convenient choice in machine learning (especially deep learning) applications. To greatly lower the technical barrier, we introduce a new software package called NCVX, whose initial release contains the solver PyGRANSO, a PyTorch-enabled port of GRANSO incorporating auto-differentiation, GPU acceleration, tensor input, and support for new QP solvers. NCVX is built on freely available and widely used open-source frameworks, and as a highlight, can solve general constrained deep learning problems, the first of its kind.

Installation

NOTE: The installation process is tested on Ubuntu 20.04

Installing PyGRANSO is simple. Here is a step-by-step instruction:

  1. Install Python >= 3.9

  2. Get the most recent PyGRANSO package (including examples and requirements file):

     git clone https://github.com/sun-umn/PyGRANSO.git
     cd PyGRANSO
    
  3. Install PyGRANSO solver from PyPI:

    pip install pygranso==1.0.0.dev2
    
  4. Install Dependencies from PyPI:

    OS: Linux OR Windows; Compute Platform: CUDA:

     pip install -r requirements.txt -f https://download.pytorch.org/whl/cu111/torch_stable.html
    

    OS: Linux; Compute Platform: CPU:

     pip install -r requirements_linux_cpu.txt -f https://download.pytorch.org/whl/cpu/torch_stable.html
    

    OS: Mac OR Windows; Compute Platform: CPU:

     pip install -r requirements_cpu.txt
    
  5. (CUDA) Run test to make sure the dependency installation is correct:

     python test_cuda.py
    

    (CPU) Run test to make sure the dependency installation is correct:

     python test_cpu.py
    
  6. Check the example folder in the source code or example section on the documentation website to get started.

Dependencies

Python-3.9.7

numpy-1.20.3

scipy-1.7.1

pytorch-1.9.0

osqp-0.6.2

Jupyter Notebook-6.4.5

Change Logs

Please check CHANGELOG.md in the main folder.

Notes on Documentation

PyGRANSO is a PyTorch-enabled port of GRANSO with auto-differentiation, and some of its documentation uses MATLAB conventions. In the PyGRANSO source code docstrings, please note that:

  • struct refers to pygransoStruct, which is a simple class that users need to use for specifying their problems and options.
  • Vector and matrix refer to PyTorch tensor with (n,1) and (m,n) dimension, respectively.

Acknowledgements

We would like to thank Frank E. Curtis and Michael L. Overton for their involvement in creating the BFGS-SQP algorithm that is implemented in the software package GRANSO. This work was supported by UMII Seed Grant Program and NSF CMMI 2038403.

Citation

If you publish work that uses or refers to PyGRANSO, please cite the following two papers, which respectively introduced PyGRANSO and GRANSO:

[1] Buyun Liang, Tim Mitchell, and Ju Sun, NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning, arXiv preprint arXiv:2111.13984 (2021). Available at https://arxiv.org/abs/2111.13984

[2] Frank E. Curtis, Tim Mitchell, and Michael L. Overton, A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles, Optimization Methods and Software, 32(1):148-181, 2017. Available at https://dx.doi.org/10.1080/10556788.2016.1208749

BibTex:

@article{liang2021ncvx,
    title={{NCVX}: {A} User-Friendly and Scalable Package for Nonconvex 
    Optimization in Machine Learning}, 
    author={Buyun Liang, Tim Mitchell, and Ju Sun},
    year={2021},
    eprint={2111.13984},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

@article{curtis2017bfgssqp,
    title={A {BFGS-SQP} method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles},
    author={Frank E. Curtis, Tim Mitchell, and Michael L. Overton},
    journal={Optimization Methods and Software},
    volume={32},
    number={1},
    pages={148--181},
    year={2017},
    publisher={Taylor \& Francis}
}

Contact

For questions or bug reports, please either:

  • raise issues in the PyGRANSO repository (https://github.com/sun-umn/PyGRANSO/) or
  • send an email to:
    • Buyun Liang (liang664 an_at_symbol umn a_dot_symbol edu)
    • Tim Mitchell (tim an_at_symbol timmitchell a_dot_symbol com)
    • Ju Sun (jusun an_at_symbol umn a_dot_symbol edu)

Thanks to other contributors and bug reporters:

Chen Jiang: Tested perceptual attack example (ex6). Tested PyGRANSO on Win10. Debug updatePenaltyParameter function.

Comments
  • Eigenvalue opt example

    Eigenvalue opt example

    Need to change:

    • [x] \lambda to \Lambda (lowercase to uppercase) (Typically, \Lambda uppercase denotes the spectrum, while \lambda lowercase denotes an eigenvalue.)
    • [x] "$\lambda(\cdot)$ is the eigenvalue of $\cdot$," to "$\Lambda(\cdot)$ is the spectrum of a square matrix,"
    • [x] "$\alpha$ is the maximum real part of eigenvalues of $M$." to "$\alpha(\cdot)$ is the spectral abscissa of a square matrix, i.e., the maximum real part of its eigenvalues."
    • [x] For imaginary part, \Im, in papers, I generally redefine this from the standard symbol to just "Im", e.g., "Im(A+BXC)", as I think this is usually much clearer. But this is a debatable suggestion.
    opened by t-mitchell 13
  • Posting announcement for PyGRANSO

    Posting announcement for PyGRANSO

    Machine Learning News: https://groups.google.com/g/ml-news

    SIAM: https://www.siam.org/membership/activity-groups/detail/computational-science-and-engineering https://www.siam.org/membership/activity-groups/detail/data-science https://www.siam.org/membership/activity-groups/detail/imaging-science https://www.siam.org/membership/activity-groups/detail/optimization

    opened by Buyun-Liang 11
  • Discussion: code and algorithms

    Discussion: code and algorithms

    General things:

    • [x] #13
    • [x] #20
    • [x] What is the deal with X_struct and the separation of variables as opposed to just an input vector?
    • [x] #14

    Gradient stuff:

    • [x] Does the AD enabling have to happen in comb_fn that the user provides?
    • [x] #15
    • [x] Is adding support for the user to choose between explicit and AD gradients useful? Might be a bit of work to support this though!

    Line search stuff

    • [x] What is the deal with disabling code 6 termination and how is the BFGS update then ensured to not lose positive definiteness?
    • [x] What is the deal with the backtracking line search? Same issue with positive definiteness and this may not work well for nonsmooth problems. The GRANSO line search is a key component of the BFGS-SQP algorithm.
    • [x] #16
    Code TODO 
    opened by t-mitchell 8
  • update license and copyright

    update license and copyright

    I went through the private folder and saw that we have four different types of files to update. I did some sample updates so you (a) can review to see if you agree and like my changes and (b) have templates to work from. The types are:

    1. direct port from GRANSO (see bfgsDamping.py)
    2. direct port from URTM / GRANSO (see isAnInteger.py)
    3. port from GRANSO with changes (see linesearchWeakWolfe.py)
    4. entirely new file and code (see getCiGradVec.py)

    Note that I added a GPL header block linesearchWeakWolfe.py (and bfgsHessianInverse.py) as these individual routines were derived from HANSO which is GPL. But don't add this headers to any other files.

    Overall, I tried to make things as compact as possible and only use one AGPL license block. I also eliminated some lines I thought were unnecessary or would be annoying to continually update for future releases. For example, maintaining version numbers in every file is tedious work, so I only put version numbers in the main user-facing routines, i.e., granso, gransoOptions, and gransoOptionsAdvanced.

    p.s. Let me know what you think and if you have suggestions! I definitely went back and forth on how to clearly do all of this.

    opened by t-mitchell 6
  • authors file

    authors file

    Regarding AUTHORS.md:

    1. Do you think we need or want such a detailed file? I was thinking we just I wrote GRANSO and you ported it to Python and added PyTorch support and more example.
    2. I wasn't sure how you determined the file groups, as they look different from what I see marked as direct port or not in the private folder.
    opened by t-mitchell 4
  • docstring indentation

    docstring indentation

    Probably we should remove a level of indentation in the docstring for each file, since as is, we have double the normal amount of indentation. Obviously, this is a super minor point and not critical.

    opened by t-mitchell 4
  • Regarding dummy class in `pygransoStruct.py`

    Regarding dummy class in `pygransoStruct.py`

    I'm going to replace in the dummy class with dict for the v1.0.0 release.

    How should I define the dict for the opts so that the values in dict can have different data type?

    Code TODO 
    opened by Buyun-Liang 3
  • "Please see the notes for the data structures used in PyGRANSO"?

    What does "Please see the notes for the data structures used in PyGRANSO" refer to? Is it another document somewhere? Or is this self-referential, i.e., referring to the "Notes on Documentation" section in the readme that this phrase appears in?

    question 
    opened by t-mitchell 2
  • Examples Updated

    Examples Updated

    I just updated all examples in the ./example folder. Maybe we can find a time to carefully review the README.md tomorrow

    Please let me know if there is any other thing to be updated in the source code before the initial public release.

    opened by Buyun-Liang 2
  • TODO in JMLR manuscript

    TODO in JMLR manuscript

    1. "zero dependency on proprietary packages": we should discuss this since GRANSO doesn't really have this because it could be run using OCTAVE
    2. Role of the NCVX name in the PyGRANSO package. NCVX and PyGRANSO in the JMLR paper
    Paper TODO 
    opened by Buyun-Liang 2
  • Discussion: naming and titles

    Discussion: naming and titles

    • [x] #9

    • [x] #10 optimization in machine learning" be for NCVX or PyGRANSO?

    • [x] Relatedly, is the title of the paper going to change?

    • [x] #12

    Depending on the answers to these questions, we'll need to change the docstrings in all the files.

    opened by t-mitchell 2
  • Minor bug in qpTerminationCondition.py#L226

    Minor bug in qpTerminationCondition.py#L226

    https://github.com/sun-umn/PyGRANSO/blob/main/pygranso/private/qpTerminationCondition.py#L226

    Use Hreg = torch.tensor(V) @ torch.diag(dvec) @ torch.tensor(V).T instead

    opened by Buyun-Liang 0
  • TODO in future release

    TODO in future release

    1. For avoiding unnecessary AD gradients, the GRANSO line search could be modified so that the AD gradient is only computed on demand, i.e., only compute the gradient when sufficient decrease holds

    2. Adding the MPI parallel full BFGS Python code Tim has

    Future Release 
    opened by Buyun-Liang 2
Releases(v1.2.0)
  • v1.2.0(Jul 27, 2022)

    Version: 1.2.0 --- 2022-07-26

    Description: major fixes and improvements on LBFGS.

    Fixed

    • Reducing memory usage for LBFGS. Now PyGRANSO can solve problem with ~15k parameters by using 14 GB memory.
    • Update example: ortho RNN with max folding and orthonormal initialization.
    • Allow high precision for QP solver.
    • Allow part of optimization variables not showing up in objective (see SVM example).
    • Fixed Code 12: terminated with steering failure.
    • Fixed stationary failure: try different stationarity calculation, or set stationarity measure to be inf if encounter numerical issue

    Added

    • Reorganize and add examples: perceptual/lp norm attack on ImageNet images. trace optimization with orthogonal constraints; unconstrained deep learning with LeNet5; logistic regression.
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Feb 20, 2022)

    Version: 1.1.0 --- 2022-02-20

    Description: major fixes and improvements.

    Fixed

    • Avoid gradient accumulating in deep learning problem;
    • Prevent memory leak problem when using torch tensor. See ex6 perceptual attack.

    Changed

    • Update format of user-defined variables when using pygranso interface.

    Packaging

    • Publish pygranso package on Pypi.

    Added

    • Add examples: ex 10 dictionary learning with torch.nn module; ex 11 orthogonal recurrent neural networks.
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Dec 31, 2021)

    Description: initial public release of PyGRANSO.

    Main features: auto-differentiation, GPU acceleration, tensor input, scalable QP solver, and zero dependency on proprietary packages. Multiple new examples added.

    Source code(tar.gz)
    Source code(zip)
A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks

A framework that constructs deep neural networks, autoencoders, logistic regressors, and linear networks without the use of any outside machine learning libraries - all from scratch.

Kordel K. France 2 Nov 14, 2022
This repo is a C++ version of yolov5_deepsort_tensorrt. Packing all C++ programs into .so files, using Python script to call C++ programs further.

yolov5_deepsort_tensorrt_cpp Introduction This repo is a C++ version of yolov5_deepsort_tensorrt. And packing all C++ programs into .so files, using P

41 Dec 27, 2022
[Preprint] ConvMLP: Hierarchical Convolutional MLPs for Vision, 2021

Convolutional MLP ConvMLP: Hierarchical Convolutional MLPs for Vision Preprint link: ConvMLP: Hierarchical Convolutional MLPs for Vision By Jiachen Li

SHI Lab 143 Jan 03, 2023
Video-Music Transformer

VMT Video-Music Transformer (VMT) is an attention-based multi-modal model, which generates piano music for a given video. Paper https://arxiv.org/abs/

Chin-Tung Lin 5 Jul 13, 2022
SeqAttack: a framework for adversarial attacks on token classification models

A framework for adversarial attacks against token classification models

Walter 23 Nov 25, 2022
End-to-end speech secognition toolkit

End-to-end speech secognition toolkit This is an E2E ASR toolkit modified from Espnet1 (version 0.9.9). This is the official implementation of paper:

Jinchuan Tian 147 Dec 28, 2022
Iranian Cars Detection using Yolov5s, PyTorch

Iranian Cars Detection using Yolov5 Train 1- git clone https://github.com/ultralytics/yolov5 cd yolov5 pip install -r requirements.txt 2- Dataset ../

Nahid Ebrahimian 22 Dec 05, 2022
MIRACLE (Missing data Imputation Refinement And Causal LEarning)

MIRACLE (Missing data Imputation Refinement And Causal LEarning) Code Author: Trent Kyono This repository contains the code used for the "MIRACLE: Cau

van_der_Schaar \LAB 15 Dec 29, 2022
[CVPR 2021] MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognition

MetaSAug: Meta Semantic Augmentation for Long-Tailed Visual Recognition (CVPR 2021) arXiv Prerequisite PyTorch = 1.2.0 Python3 torchvision PIL argpar

51 Nov 11, 2022
A collection of IPython notebooks covering various topics.

ipython-notebooks This repo contains various IPython notebooks I've created to experiment with libraries and work through exercises, and explore subje

John Wittenauer 2.6k Jan 01, 2023
Official Implementation of Few-shot Visual Relationship Co-localization

VRC Official implementation of the Few-shot Visual Relationship Co-localization (ICCV 2021) paper project page | paper Requirements Use python = 3.8.

22 Oct 13, 2022
This is an implementation of PIFuhd based on Pytorch

Open-PIFuhd This is a unofficial implementation of PIFuhd PIFuHD: Multi-Level Pixel-Aligned Implicit Function forHigh-Resolution 3D Human Digitization

Lingteng Qiu 235 Dec 19, 2022
TyXe: Pyro-based BNNs for Pytorch users

TyXe: Pyro-based BNNs for Pytorch users TyXe aims to simplify the process of turning Pytorch neural networks into Bayesian neural networks by leveragi

87 Jan 03, 2023
Unsupervised Video Interpolation using Cycle Consistency

Unsupervised Video Interpolation using Cycle Consistency Project | Paper | YouTube Unsupervised Video Interpolation using Cycle Consistency Fitsum A.

NVIDIA Corporation 100 Nov 30, 2022
MMFlow is an open source optical flow toolbox based on PyTorch

Documentation: https://mmflow.readthedocs.io/ Introduction English | 简体中文 MMFlow is an open source optical flow toolbox based on PyTorch. It is a part

OpenMMLab 688 Jan 06, 2023
KakaoBrain KoGPT (Korean Generative Pre-trained Transformer)

KoGPT KoGPT (Korean Generative Pre-trained Transformer) https://github.com/kakaobrain/kogpt https://huggingface.co/kakaobrain/kogpt Model Descriptions

Kakao Brain 799 Dec 28, 2022
Implementation of Self-supervised Graph-level Representation Learning with Local and Global Structure (ICML 2021).

Self-supervised Graph-level Representation Learning with Local and Global Structure Introduction This project is an implementation of ``Self-supervise

MilaGraph 50 Dec 09, 2022
Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

CoProtector Code for the prototype tool in our paper "CoProtector: Protect Open-Source Code against Unauthorized Training Usage with Data Poisoning".

Zhensu Sun 1 Oct 26, 2021
Official repo for AutoInt: Automatic Integration for Fast Neural Volume Rendering in CVPR 2021

AutoInt: Automatic Integration for Fast Neural Volume Rendering CVPR 2021 Project Page | Video | Paper PyTorch implementation of automatic integration

Stanford Computational Imaging Lab 149 Dec 22, 2022
Pytorch implementation of the AAAI 2022 paper "Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification"

[AAAI22] Cross-Domain Empirical Risk Minimization for Unbiased Long-tailed Classification We point out the overlooked unbiasedness in long-tailed clas

PatatiPatata 28 Oct 18, 2022