PyGRANSO: A PyTorch-enabled port of GRANSO with auto-differentiation

Overview

PyGRANSO

PyGRANSO

PyGRANSO: A PyTorch-enabled port of GRANSO with auto-differentiation

Please check https://ncvx.org/PyGRANSO for detailed instructions (introduction, installation, settings, examples...).

PyGRANSO is AGPL v3.0 licensed, but it also contains a small portion of GPL code. Please see the LICENSE_INFO folder for more information.

Brief Introduction

Optimizing nonconvex (NCVX) problems, especially nonsmooth and constrained ones, is an essential part of machine learning. However, it can be hard to reliably solve such problems without optimization expertise. Existing general-purpose NCVX optimization packages are powerful but typically cannot handle nonsmoothness. GRANSO is among the first optimization solvers targeting general nonsmooth NCVX problems with nonsmooth constraints, but, as it is implemented in MATLAB and requires the user to provide analytical gradients, GRANSO is often not a convenient choice in machine learning (especially deep learning) applications. To greatly lower the technical barrier, we introduce a new software package called NCVX, whose initial release contains the solver PyGRANSO, a PyTorch-enabled port of GRANSO incorporating auto-differentiation, GPU acceleration, tensor input, and support for new QP solvers. NCVX is built on freely available and widely used open-source frameworks, and as a highlight, can solve general constrained deep learning problems, the first of its kind.

Installation

NOTE: The installation process is tested on Ubuntu 20.04

Installing PyGRANSO is simple. Here is a step-by-step instruction:

  1. Install Python >= 3.9

  2. Get the most recent PyGRANSO package (including examples and requirements file):

     git clone https://github.com/sun-umn/PyGRANSO.git
     cd PyGRANSO
    
  3. Install PyGRANSO solver from PyPI:

    pip install pygranso==1.0.0.dev2
    
  4. Install Dependencies from PyPI:

    OS: Linux OR Windows; Compute Platform: CUDA:

     pip install -r requirements.txt -f https://download.pytorch.org/whl/cu111/torch_stable.html
    

    OS: Linux; Compute Platform: CPU:

     pip install -r requirements_linux_cpu.txt -f https://download.pytorch.org/whl/cpu/torch_stable.html
    

    OS: Mac OR Windows; Compute Platform: CPU:

     pip install -r requirements_cpu.txt
    
  5. (CUDA) Run test to make sure the dependency installation is correct:

     python test_cuda.py
    

    (CPU) Run test to make sure the dependency installation is correct:

     python test_cpu.py
    
  6. Check the example folder in the source code or example section on the documentation website to get started.

Dependencies

Python-3.9.7

numpy-1.20.3

scipy-1.7.1

pytorch-1.9.0

osqp-0.6.2

Jupyter Notebook-6.4.5

Change Logs

Please check CHANGELOG.md in the main folder.

Notes on Documentation

PyGRANSO is a PyTorch-enabled port of GRANSO with auto-differentiation, and some of its documentation uses MATLAB conventions. In the PyGRANSO source code docstrings, please note that:

  • struct refers to pygransoStruct, which is a simple class that users need to use for specifying their problems and options.
  • Vector and matrix refer to PyTorch tensor with (n,1) and (m,n) dimension, respectively.

Acknowledgements

We would like to thank Frank E. Curtis and Michael L. Overton for their involvement in creating the BFGS-SQP algorithm that is implemented in the software package GRANSO. This work was supported by UMII Seed Grant Program and NSF CMMI 2038403.

Citation

If you publish work that uses or refers to PyGRANSO, please cite the following two papers, which respectively introduced PyGRANSO and GRANSO:

[1] Buyun Liang, Tim Mitchell, and Ju Sun, NCVX: A User-Friendly and Scalable Package for Nonconvex Optimization in Machine Learning, arXiv preprint arXiv:2111.13984 (2021). Available at https://arxiv.org/abs/2111.13984

[2] Frank E. Curtis, Tim Mitchell, and Michael L. Overton, A BFGS-SQP method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles, Optimization Methods and Software, 32(1):148-181, 2017. Available at https://dx.doi.org/10.1080/10556788.2016.1208749

BibTex:

@article{liang2021ncvx,
    title={{NCVX}: {A} User-Friendly and Scalable Package for Nonconvex 
    Optimization in Machine Learning}, 
    author={Buyun Liang, Tim Mitchell, and Ju Sun},
    year={2021},
    eprint={2111.13984},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

@article{curtis2017bfgssqp,
    title={A {BFGS-SQP} method for nonsmooth, nonconvex, constrained optimization and its evaluation using relative minimization profiles},
    author={Frank E. Curtis, Tim Mitchell, and Michael L. Overton},
    journal={Optimization Methods and Software},
    volume={32},
    number={1},
    pages={148--181},
    year={2017},
    publisher={Taylor \& Francis}
}

Contact

For questions or bug reports, please either:

  • raise issues in the PyGRANSO repository (https://github.com/sun-umn/PyGRANSO/) or
  • send an email to:
    • Buyun Liang (liang664 an_at_symbol umn a_dot_symbol edu)
    • Tim Mitchell (tim an_at_symbol timmitchell a_dot_symbol com)
    • Ju Sun (jusun an_at_symbol umn a_dot_symbol edu)

Thanks to other contributors and bug reporters:

Chen Jiang: Tested perceptual attack example (ex6). Tested PyGRANSO on Win10. Debug updatePenaltyParameter function.

Comments
  • Eigenvalue opt example

    Eigenvalue opt example

    Need to change:

    • [x] \lambda to \Lambda (lowercase to uppercase) (Typically, \Lambda uppercase denotes the spectrum, while \lambda lowercase denotes an eigenvalue.)
    • [x] "$\lambda(\cdot)$ is the eigenvalue of $\cdot$," to "$\Lambda(\cdot)$ is the spectrum of a square matrix,"
    • [x] "$\alpha$ is the maximum real part of eigenvalues of $M$." to "$\alpha(\cdot)$ is the spectral abscissa of a square matrix, i.e., the maximum real part of its eigenvalues."
    • [x] For imaginary part, \Im, in papers, I generally redefine this from the standard symbol to just "Im", e.g., "Im(A+BXC)", as I think this is usually much clearer. But this is a debatable suggestion.
    opened by t-mitchell 13
  • Posting announcement for PyGRANSO

    Posting announcement for PyGRANSO

    Machine Learning News: https://groups.google.com/g/ml-news

    SIAM: https://www.siam.org/membership/activity-groups/detail/computational-science-and-engineering https://www.siam.org/membership/activity-groups/detail/data-science https://www.siam.org/membership/activity-groups/detail/imaging-science https://www.siam.org/membership/activity-groups/detail/optimization

    opened by Buyun-Liang 11
  • Discussion: code and algorithms

    Discussion: code and algorithms

    General things:

    • [x] #13
    • [x] #20
    • [x] What is the deal with X_struct and the separation of variables as opposed to just an input vector?
    • [x] #14

    Gradient stuff:

    • [x] Does the AD enabling have to happen in comb_fn that the user provides?
    • [x] #15
    • [x] Is adding support for the user to choose between explicit and AD gradients useful? Might be a bit of work to support this though!

    Line search stuff

    • [x] What is the deal with disabling code 6 termination and how is the BFGS update then ensured to not lose positive definiteness?
    • [x] What is the deal with the backtracking line search? Same issue with positive definiteness and this may not work well for nonsmooth problems. The GRANSO line search is a key component of the BFGS-SQP algorithm.
    • [x] #16
    Code TODO 
    opened by t-mitchell 8
  • update license and copyright

    update license and copyright

    I went through the private folder and saw that we have four different types of files to update. I did some sample updates so you (a) can review to see if you agree and like my changes and (b) have templates to work from. The types are:

    1. direct port from GRANSO (see bfgsDamping.py)
    2. direct port from URTM / GRANSO (see isAnInteger.py)
    3. port from GRANSO with changes (see linesearchWeakWolfe.py)
    4. entirely new file and code (see getCiGradVec.py)

    Note that I added a GPL header block linesearchWeakWolfe.py (and bfgsHessianInverse.py) as these individual routines were derived from HANSO which is GPL. But don't add this headers to any other files.

    Overall, I tried to make things as compact as possible and only use one AGPL license block. I also eliminated some lines I thought were unnecessary or would be annoying to continually update for future releases. For example, maintaining version numbers in every file is tedious work, so I only put version numbers in the main user-facing routines, i.e., granso, gransoOptions, and gransoOptionsAdvanced.

    p.s. Let me know what you think and if you have suggestions! I definitely went back and forth on how to clearly do all of this.

    opened by t-mitchell 6
  • authors file

    authors file

    Regarding AUTHORS.md:

    1. Do you think we need or want such a detailed file? I was thinking we just I wrote GRANSO and you ported it to Python and added PyTorch support and more example.
    2. I wasn't sure how you determined the file groups, as they look different from what I see marked as direct port or not in the private folder.
    opened by t-mitchell 4
  • docstring indentation

    docstring indentation

    Probably we should remove a level of indentation in the docstring for each file, since as is, we have double the normal amount of indentation. Obviously, this is a super minor point and not critical.

    opened by t-mitchell 4
  • Regarding dummy class in `pygransoStruct.py`

    Regarding dummy class in `pygransoStruct.py`

    I'm going to replace in the dummy class with dict for the v1.0.0 release.

    How should I define the dict for the opts so that the values in dict can have different data type?

    Code TODO 
    opened by Buyun-Liang 3
  • "Please see the notes for the data structures used in PyGRANSO"?

    What does "Please see the notes for the data structures used in PyGRANSO" refer to? Is it another document somewhere? Or is this self-referential, i.e., referring to the "Notes on Documentation" section in the readme that this phrase appears in?

    question 
    opened by t-mitchell 2
  • Examples Updated

    Examples Updated

    I just updated all examples in the ./example folder. Maybe we can find a time to carefully review the README.md tomorrow

    Please let me know if there is any other thing to be updated in the source code before the initial public release.

    opened by Buyun-Liang 2
  • TODO in JMLR manuscript

    TODO in JMLR manuscript

    1. "zero dependency on proprietary packages": we should discuss this since GRANSO doesn't really have this because it could be run using OCTAVE
    2. Role of the NCVX name in the PyGRANSO package. NCVX and PyGRANSO in the JMLR paper
    Paper TODO 
    opened by Buyun-Liang 2
  • Discussion: naming and titles

    Discussion: naming and titles

    • [x] #9

    • [x] #10 optimization in machine learning" be for NCVX or PyGRANSO?

    • [x] Relatedly, is the title of the paper going to change?

    • [x] #12

    Depending on the answers to these questions, we'll need to change the docstrings in all the files.

    opened by t-mitchell 2
  • Minor bug in qpTerminationCondition.py#L226

    Minor bug in qpTerminationCondition.py#L226

    https://github.com/sun-umn/PyGRANSO/blob/main/pygranso/private/qpTerminationCondition.py#L226

    Use Hreg = torch.tensor(V) @ torch.diag(dvec) @ torch.tensor(V).T instead

    opened by Buyun-Liang 0
  • TODO in future release

    TODO in future release

    1. For avoiding unnecessary AD gradients, the GRANSO line search could be modified so that the AD gradient is only computed on demand, i.e., only compute the gradient when sufficient decrease holds

    2. Adding the MPI parallel full BFGS Python code Tim has

    Future Release 
    opened by Buyun-Liang 2
Releases(v1.2.0)
  • v1.2.0(Jul 27, 2022)

    Version: 1.2.0 --- 2022-07-26

    Description: major fixes and improvements on LBFGS.

    Fixed

    • Reducing memory usage for LBFGS. Now PyGRANSO can solve problem with ~15k parameters by using 14 GB memory.
    • Update example: ortho RNN with max folding and orthonormal initialization.
    • Allow high precision for QP solver.
    • Allow part of optimization variables not showing up in objective (see SVM example).
    • Fixed Code 12: terminated with steering failure.
    • Fixed stationary failure: try different stationarity calculation, or set stationarity measure to be inf if encounter numerical issue

    Added

    • Reorganize and add examples: perceptual/lp norm attack on ImageNet images. trace optimization with orthogonal constraints; unconstrained deep learning with LeNet5; logistic regression.
    Source code(tar.gz)
    Source code(zip)
  • v1.1.0(Feb 20, 2022)

    Version: 1.1.0 --- 2022-02-20

    Description: major fixes and improvements.

    Fixed

    • Avoid gradient accumulating in deep learning problem;
    • Prevent memory leak problem when using torch tensor. See ex6 perceptual attack.

    Changed

    • Update format of user-defined variables when using pygranso interface.

    Packaging

    • Publish pygranso package on Pypi.

    Added

    • Add examples: ex 10 dictionary learning with torch.nn module; ex 11 orthogonal recurrent neural networks.
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(Dec 31, 2021)

    Description: initial public release of PyGRANSO.

    Main features: auto-differentiation, GPU acceleration, tensor input, scalable QP solver, and zero dependency on proprietary packages. Multiple new examples added.

    Source code(tar.gz)
    Source code(zip)
Implementation of experiments in the paper Clockwork Variational Autoencoders (project website) using JAX and Flax

Clockwork VAEs in JAX/Flax Implementation of experiments in the paper Clockwork Variational Autoencoders (project website) using JAX and Flax, ported

Julius Kunze 26 Oct 05, 2022
The final project of "Applying AI to 3D Medical Imaging Data" from "AI for Healthcare" nanodegree - Udacity.

Quantifying Hippocampus Volume for Alzheimer's Progression Background Alzheimer's disease (AD) is a progressive neurodegenerative disorder that result

Omar Laham 1 Jan 14, 2022
A Python Package for Convex Regression and Frontier Estimation

pyStoNED pyStoNED is a Python package that provides functions for estimating multivariate convex regression, convex quantile regression, convex expect

Sheng Dai 17 Jan 08, 2023
Voice control for Garry's Mod

WIP: Talonvoice GMod integrations Very work in progress voice control demo for Garry's Mod. HOWTO Install https://talonvoice.com/ Press https://i.imgu

Meta Construct 5 Nov 15, 2022
PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition, CVPR 2018

PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place Recognition PointNetVLAD: Deep Point Cloud Based Retrieval for Large-Scale Place

Mikaela Uy 294 Dec 12, 2022
[CoRL 2021] A robotics benchmark for cross-embodiment imitation.

x-magical x-magical is a benchmark extension of MAGICAL specifically geared towards cross-embodiment imitation. The tasks still provide the Demo/Test

Kevin Zakka 36 Nov 26, 2022
An executor that performs image segmentation on fashion items

ClothingSegmenter U2NET fashion image/clothing segmenter based on https://github.com/levindabhi/cloth-segmentation Overview The ClothingSegmenter exec

Jina AI 5 Mar 30, 2022
Novel and high-performance medical image classification pipelines are heavily utilizing ensemble learning strategies

An Analysis on Ensemble Learning optimized Medical Image Classification with Deep Convolutional Neural Networks Novel and high-performance medical ima

14 Dec 18, 2022
Split Variational AutoEncoder

Split-VAE Split Variational AutoEncoder Introduction This repository contains and implemementation of a Split Variational AutoEncoder (SVAE). In a SVA

Andrea Asperti 2 Sep 02, 2022
This is the code for CVPR 2021 oral paper: Jigsaw Clustering for Unsupervised Visual Representation Learning

JigsawClustering Jigsaw Clustering for Unsupervised Visual Representation Learning Pengguang Chen, Shu Liu, Jiaya Jia Introduction This project provid

DV Lab 73 Sep 18, 2022
Internship Assessment Task for BaggageAI.

BaggageAI Internship Task Problem Statement: You are given two sets of images:- background and threat objects. Background images are the background x-

Arya Shah 10 Nov 14, 2022
Convert weight file.pth to weight file.blob

CONVERT YOUR MODEL TO IR FORMAT INSTALLATION OpenVino Toolkit Download openvinotoolkit 2021.3 version : Link Instruction of installation : Link Pytorc

Tran Anh Tuan 3 Nov 18, 2021
Learning to Estimate Hidden Motions with Global Motion Aggregation

Learning to Estimate Hidden Motions with Global Motion Aggregation (GMA) This repository contains the source code for our paper: Learning to Estimate

Shihao Jiang (Zac) 221 Dec 18, 2022
PiRank: Learning to Rank via Differentiable Sorting

PiRank: Learning to Rank via Differentiable Sorting This repository provides a reference implementation for learning PiRank-based models as described

54 Dec 17, 2022
Py4fi2nd - Jupyter Notebooks and code for Python for Finance (2nd ed., O'Reilly) by Yves Hilpisch.

Python for Finance (2nd ed., O'Reilly) This repository provides all Python codes and Jupyter Notebooks of the book Python for Finance -- Mastering Dat

Yves Hilpisch 1k Jan 05, 2023
A rule learning algorithm for the deduction of syndrome definitions from time series data.

README This project provides a rule learning algorithm for the deduction of syndrome definitions from time series data. Large parts of the algorithm a

0 Sep 24, 2021
OpenMMLab Detection Toolbox and Benchmark

MMDetection is an open source object detection toolbox based on PyTorch. It is a part of the OpenMMLab project.

OpenMMLab 22.5k Jan 05, 2023
Joint project of the duo Hacker Ninjas

Project Smoothie Společný projekt dua Hacker Ninjas. První pokus o hříčku po třech týdnech učení se programování. Jakub Kolář e:\

Jakub Kolář 2 Jan 07, 2022
KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite

KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite

80 Dec 27, 2022
Lexical Substitution Framework

LexSubGen Lexical Substitution Framework This repository contains the code to reproduce the results from the paper: Arefyev Nikolay, Sheludko Boris, P

Samsung 37 Sep 15, 2022