pydock - Docker-based environment manager for Python

Related tags

Miscellaneouspydock
Overview

pydock - Docker-based environment manager for Python

GitHub GitHub tag (latest SemVer pre-release) GitHub Repo stars GitHub forks GitHub watchers

⚠️ pydock is still in beta mode, and very unstable. It is not recommended for anything serious.

pydock is a poor man's Python environment manager fully based on Docker. You can think of it as a replacement for virtualenv. In reality, pydock is just a very thing wrapper around Docker, so everything you can do with pydock you can also do it yourself just with Docker.

The purpose of pydock is to avoid having to install anything at all in your system, and instead manage everything using Docker. In short, pydock gives you an interface similar to most Python environment managers, but uses Docker under the hood, creating dockerfiles, images, and containers as necessary. This creates a bunch of additional headaches, that's for sure, but it has some nice conveniences.

With pydock you can create "virtual" environments, which are actually Docker images, and manage them similarly as with virtualenv and any other Python environment manager. Every environment you create has associated dockerfile and requirements.txt files which provide a completely platform-independent description of that environment. Thus, if at any point you want to migrate those environments to another computer, you just need to copy these files, and run pydock build there.

Design

pydock's mantra is zero-dependencies and absolute freedom. This means it will never create an environment that requires you to install anything to use, not even pydock (outside of Docker, that is, but everyone is already using Docker, right?) In particular, these are some principles we abide to:

  • Use of open standards for decribing environments: Right now the definition of an environment is just a dockerfile and a requirement.txt. There is not and will never be any pydock-specific file there. This means you completely control what goes into an environment, and will never be locked into using pydock for runing or modifying an environment.

  • Depend only on the standard library: Since pydock is supposed to remove your need to install things in your system's Python, it cannot depend on anything that is not bundled in the standard Python distribution that comes with most operating systems (we're talking real operating systems here 😛 ).

Installation (sort of...)

pydock is a single Python file with no dependencies outside the Python standard library and Docker. So you can just download it, give it execution permisions, and add it to your path.

In Linux one way to do this is with this convenience script:

curl https://raw.githubusercontent.com/apiad/pydock/main/install/linux.sh | sudo bash

If you only want to use pydock inside a specific project, then you can just download the pydock.py file into your codebase and commit it to your repository. Then you can use it locally as (provided you gave it execution permisions):

./pydock.py --local  [args...]

Usage

Run pydock to see all available commands, and run pydock to see a small help for that command.

pydock can run in global or local mode, the difference being where it will store the environments configuration. In global mode, everything will be stored in ~/.pydock, at the /home of the current user. In local mode, everything is stored inside a .pydock folder at the current working directory. The rules to decide whether to run in global or local mode are:

  • If you explicitely type pydock --local it will be local. Likewise, if you explicitely type pydock --global it will be global.
  • If no explicit flag is used, then if there is a .pydock folder already created in the current folder (i.e., you ran pydock --local sometime before), it will default to local mode.
  • Otherwise, it will run in global mode.

We recommend global mode when you're creating an environment for interactive coding, e.g., for notebooks, one-off scripts, etc. They are stored in your home folder and can be accessed from anywhere.

Use local mode when you're creating one or more environments for a specific project. Store them with the project source code and probably even commit them to version control, so that all developers share the same environments.

In any moment, you can type pydock config and it will tell you whether it is running in local or global mode.

Creating an environment

Run pydock [--local/--global] create to create a new environment with a given name and Python version. For example:

pydock create datascience 3.8

This command will do the following:

  • Create a new folder datascience inside .pydock (wherever that folder is depends on the local vs global mode).
  • Create a dockerfile and requirements.txt files inside that folder.
  • Run docker build in that context, effectively creating a new image with your desired Python version.

By default, that image will have a user named like the user who run pydock create (this can be customized via configuration).

Executing a shell in an environment

After creating an environment, if you run docker images you'll see a pydock-:latest image, which corresponds to your environment. You can easily start it with (continuing with the previous example):

pydock shell datascience

This will execute a docker run ... datascience bash command tailored to that environment with some additional tidbits. One is that your current working directory will be mounted inside the newly created container's /home/, which will be the starting working directory. Thus, inside the container, whatever you do will be reflected back in your host filesystem, hopefully with the right permissions.

Managing dependencies in an environment

In any existing environment pydock can help you install new dependencies while keeping updated the Docker image and tracking all packages. For example:

pydock install datascience pandas

This will launch a fresh container in the datascience environment and install pandas. pydock will commit the container and re-tag the new image such that it replaces the existing one for this environment, effectively saving the changes you did to the environment. Additionally, the requirements.txt will be updated with the contents of pip freeze, such that next time you call build you'll have the same environment.

Likewise, you can use pydock update and pydock uninstall to update / uninstall dependencies in an environment, with the same syntax and it will do what you expect.

Rebuilding an environment

At any moment, the pydock- images that correspond to each environment should be up-to-date but, if you manually modify the dockerfile or requirements.txt (which you are absolutely free to do), you can run this command to rebuild and tag the corresponding image.

pydock build <name>

This command is also useful if you want to move environments around. For example, by commiting your local .pydock folder into source control for a given project, other developers can easily run pydock build ... after checkout and the corresponding environment(s) will be created.

If you run build manually, pydock will not delete the old image for that container, which will appear labelled . Make sure to either delete it manually with docker rmi or run docker system prune periodically to remove any accumulated waste.

Roadmap

Planned

  • Add a docker-compose.yml file to environments to handle port bindings, volumes, etc.
  • Change dockerfile template such that user and repository are args, inserted during build instead of when generating the file.
  • Generate unique environment image names for envs that have the same name but are located in different local folders

v0.0.5

  • Add /home/user/.local/bin to $PATH so that installed scripts work.

v0.0.4

  • Automatically deletes untagged images when managing dependencies.
  • Added commands to remove and update dependencies.
  • Added a bunch of exception handling when Docker commands fail.

v0.0.3

  • Improved install script to make it robust to different paths for the python command.

v0.0.2

  • Added a command to install dependencies inside the environment and commit/rebuild the image.

v0.0.1

  • Added commands to create, list, and run a shell inside of environments.

License and Contribution

Code is MIT, and all contributions are appreciated 👋 !

To use pydock in development mode, after you fork and clone, run:

sudo make dev

This will create a soft link in /usr/bin/pydock to your working src/pydock.py file, so that when you type pydock you'll be using your development version.

Owner
Alejandro Piad
Professor (Instructor) at @matcom, University of Havana and Ph.D. student jointly at U.Alicante. Democratizing ML via @autogoal, working on cNLP at @ehealthkd.
Alejandro Piad
NasaApod - Astronomy Picture of the Day

Astronomy Picture of the Day Get interesting Astronomical pictures with a brief

Shripad Rao 1 Feb 15, 2022
Twikoo自定义表情列表 | HexoPlusPlus自定义表情列表(其实基于OwO的项目都可以用的啦)

Twikoo-Magic 更新说明 2021/1/15 基于2021/1/14 Twikoo 更新1.1.0-beta,所有表情都将以缩写形式(如:[ text ]:)输出。1/14之前本仓库有部分表情text缺失及重复, 导致无法正常使用表情 1/14后的所有表情json列表已全部更新

noionion 90 Jan 05, 2023
Your self-hosted bookmark archive. Free and open source.

Your self-hosted bookmark archive. Free and open source. Contents About LinkAce Support Setup Contribution About LinkAce LinkAce is a self-hosted arch

Kevin Woblick 1.7k Jan 03, 2023
Tools I'm building in order to help my investments decisions

b3-tools Tools I'm building in order to help my investments decisions. Based in the REITs I've in my personal portifolio I ran a script that scrapy th

Rafael Cassau 2 Jan 21, 2022
Standalone PyQGIS application for executing custom scripts without a QGIS GUI.

PyQGIS Standalone Script Executer Standalone PyQGIS application that is able to run a custom script, in this case Proximity.py without the need of a G

6 Sep 23, 2022
Powering up Apache JMeter with Streamlit and opening the door for machine learning.

Powering up Apache JMeter with Streamlit Overview Apache JMeter is an open source load testing tool written in 100% pure Java. JMeter supports umpteen

NaveenKumar Namachivayam ⚡ 16 Aug 24, 2022
A (hopefully) considerably copious collection of classical cipher crackers

ClassicalCipherCracker A (hopefully) considerably copious collection of classical cipher crackers Written in Python3 (and run with PyPy) TODOs Write a

Stanley Zhong 2 Feb 22, 2022
Sailwind Mod Manager

Sailwind Mod Manager The Sailwind Mod Manager is an open source mod manager for the Sailwind community. It currently allows you to browse and download

Max 3 Jul 15, 2022
Generate Azure Blob Storage account authentication headers for Munki

Azure Blob Storage Authentication for Munki The Azure Blob Storage Middleware allows munki clients to connect securely, and directly to a munki repo h

Oliver Kieselbach 10 Apr 12, 2022
Allows you to purge all reply comments left by a user on a YouTube channel or video.

YouTube Spammer Purge Allows you to purge all reply comments left by a user on a YouTube channel or video. Purpose Recently, there has been a massive

4.3k Jan 09, 2023
:snake: Complete C99 parser in pure Python

pycparser v2.20 Contents 1 Introduction 1.1 What is pycparser? 1.2 What is it good for? 1.3 Which version of C does pycparser support? 1.4 What gramma

Eli Bendersky 2.8k Dec 29, 2022
A visidata plugin for parsing f5 ltm/gtm/audit logs

F5 Log Visidata Plugin This plugin supports the default log format for: /var/log/ltm* /var/log/gtm* /var/log/apm* /var/log/audit* It extracts common l

James Deucker 1 Jan 06, 2022
📜Generate poetry with gcc diagnostics

gado (gcc awesome diagnostics orchestrator) is a wrapper of gcc that outputs its errors and warnings in a more poetic format.

Dikson Santos 19 Jun 25, 2022
Simple card retirement plugin for Anki

Anki Retirement Addon Allow users to suspend, tag, delete, or move cards that reach a specific retirement interval Supports Anki version 2.1.45 Licens

3 Dec 23, 2022
A TODO-list tool written in Python

PyTD A TODO-list tool written in Python. Its goal is to provide a stable posibility to get a good view over all your TODOs motivate you to actually fi

1 Feb 12, 2022
This is a program for Carbon Emission calculator.

Summary This is a program for Carbon Emission calculator. Usage This will calculate the carbon emission by each person on various factors. Contributor

Ankit Rane 2 Feb 18, 2022
Python implementation of the Learning Time-Series Shapelets method, that learns a shapelet-based time-series classifier with gradient descent.

shaplets Python implementation of the Learning Time-Series Shapelets method by Josif Grabocka et al., that learns a shapelet-based time-series classif

Mohamed Haseeb 187 Dec 14, 2022
Purge all transformation orientations addon for Blender 2.8 and newer versions

CTO Purge This add-on adds a new button to Blender's Transformation Orientation panel which empowers the user to purge all of his/her custom transform

MMMrqs 10 Dec 29, 2022
Team Hash Brown Science4Cast Submission

Team Hash Brown Science4Cast Submission This code reproduces Team Hash Brown's (@princengoc, @Xieyangxinyu) best submission (ee5a) for the competition

3 Feb 02, 2022
Abilian Core: an enterprise application development platform based on the Flask micro-framework, the SQLAlchemy ORM

About Abilian Core is an enterprise application development platform based on the Flask micro-framework, the SQLAlchemy ORM, good intentions and best

Abilian open source projects 47 Apr 14, 2022