Hot reloading for Python

Overview

jurigged

Jurigged lets you update your code while it runs. Using it is trivial:

  1. jurigged your_script.py
  2. Change some function or method with your favorite editor and save the file
  3. Jurigged will hot patch the new function into the running script

Jurigged updates live code smartly: changing a function or method will fudge code pointers so that all existing instances are simultaneously modified to implement the new behavior. When modifying a module, only changed lines will be re-run.

demo

Install

pip install jurigged

Command line

The simplest way to use jurigged is to add -m jurigged to your script invocation, or to use jurigged instead of python. You can use -v to get feedback about what files are watched and what happens when you change a file.

python -m jurigged -v script.py

OR

jurigged -v script.py

With no arguments given, it will start a live REPL:

python -m jurigged

OR

jurigged

Full help:

usage: jurigged [-h] [--verbose] [--watch PATH] [-m MODULE] [PATH] ...

Run a Python script so that it is live-editable.

positional arguments:
  PATH                  Path to the script to run
  ...                   Script arguments

optional arguments:
  -h, --help            show this help message and exit
  --verbose, -v         Show watched files and changes as they happen
  --watch PATH, -w PATH
                        Wildcard path/directory for which files to watch
  -m MODULE             Module or module:function to run

Troubleshooting

First, if there's a problem, use the verbose flag (jurigged -v) to get more information. It will output a Watch statement for every file that it watches and Update/Add/Delete statements when you update, add or delete a function in the original file and then save it.

The file is not being watched.

By default, scripts are watched in the current working directory. Try jurigged -w to watch a specific file, or jurigged -w / to watch all files.

The file is watched, but nothing happens when I change the function.

It's possibly because you are using an editor that saves into a temporary swap file and moves it into place (vi does this). The watchdog library that Jurigged uses loses track of the file when that happens. Pending a better solution, you can try to configure your editor so that it writes to the file directly. For example, in vi, :set nowritebackup seems to do the trick (either put it in your .vimrc or execute it before you save for the first time).

Jurigged said it updated the function but it's still running the old code.

If you are editing the body of a for loop inside a function that's currently running, the changes will only be in effect the next time that function is called. A workaround is to extract the body of the for loop into its own helper function, which you can then edit. Alternatively, you can use reloading alongside Jurigged.

Similarly, updating a generator or async function will not change the behavior of generators or async functions that are already running.

I can update some functions but not others.

There may be issues updating some functions when they are decorated or stashed in some data structure that Jurigged does not understand. Jurigged does have to find them to update them, unfortunately.

API

You can call jurigged.watch() to programmatically start watching for changes. This should also work within IPython or Jupyter as an alternative to the %autoreload magic.

import jurigged
jurigged.watch()

By default all files in the current directory will be watched, but you can use jurigged.watch("script.py") to only watch a single file, or jurigged.watch("/") to watch all modules.

Recoders

Functions can be programmatically changed using a Recoder. Make one with jurigged.make_recoder. This can be used to implement hot patching or mocking. The changes can also be written back to the filesystem.

from jurigged import make_recoder

def f(x):
    return x * x

assert f(2) == 4

# Change the behavior of the function, but not in the original file
recoder = make_recoder(f)
recoder.patch("def f(x): return x * x * x")
assert f(2) == 8

# Revert changes
recoder.revert()
assert f(2) == 4

# OR: write the patch to the original file itself
recoder.commit()

revert will only revert up to the last commit, or to the original contents if there was no commit.

A recoder also allows you to add imports, helper functions and the like to a patch, but you have to use recoder.patch_module(...) in that case.

Caveats

Jurigged works in a surprisingly large number of situations, but there are several cases where it won't work, or where problems may arise:

  • Functions that are already running will keep running with the existing code. Only the next invocations will use the new code.
    • When debugging with a breakpoint, functions currently on the stack can't be changed.
    • A running generator or async function won't change.
    • You can use reloading in addition to Jurigged if you want to be able to modify a running for loop.
  • Changing initializers or attribute names may cause errors on existing instances.
    • Jurigged modifies all existing instances of a class, but it will not re-run __init__ or rename attributes on existing instances, so you can easily end up with broken objects (new methods, but old data).
  • Updating the code of a decorator or a closure may or may not work. Jurigged will do its best, but it is possible that some closures will be updated but not others.
  • Decorators that look at/tweak function code will probably not update properly.
    • Wrappers that try to compile/JIT Python code won't know about jurigged and won't be able to redo their work for the new code.
    • They can be made to work if they set the (jurigged-specific) __conform__ attribute on the old function. __conform__ takes a reference to the function that should replace it, or None if it is to be deleted.

How it works

In a nutshell, jurigged works as follows:

  1. Insert an import hook that collects and watches source files.
  2. Parse a source file into a set of definitions.
  3. Crawl through a module to find function objects and match them to definitions.
    • It will go through class members, follow functions' __wrapped__ and __closure__ pointers, and so on.
  4. When a file is modified, re-parse it into a set of definitions and match them against the original, yielding a set of changes, additions and deletions.
  5. For a change, exec the new code (with the decorators stripped out, if they haven't changed), then take the resulting function's internal __code__ pointer and shove it into the old one. If the change fails, it will be reinterpreted as a deletion of the old code followed by the addition of the new code.
  6. New additions are run in the module namespace.

Comparison

The two most comparable implementations of Jurigged's feature set that I could find (but it can be a bit difficult to find everything comparable) are %autoreload in IPython and limeade. Here are the key differences:

  • They both re-execute the entire module when its code is changed. Jurigged, by contrast, surgically extracts changed functions from the parse tree and only replaces these. It only executes new or changed statements in a module.

    Which is better is somewhat situation-dependent: on one hand, re-executing the module will pick up more changes. On the other hand, it will reinitialize module variables and state, so certain things might break. Jurigged's approach is more conservative and will only pick up on modified functions, but it will not touch anything else, so I believe it is less likely to have unintended side effects. It also tells you what it is doing :)

  • They will re-execute decorators, whereas Jurigged will instead dig into them and update the functions it finds inside.

    Again, there's no objectively superior approach. %autoreload will properly re-execute changed decorators, but these decorators will return new objects, so if a module imports an already decorated function, it won't update to the new version. If you only modify the function's code and not the decorators, however, Jurigged will usually be able to change it inside the decorator, so all the old instances will use the new behavior.

  • They rely on synchronization points, whereas Jurigged can be run in its own thread.

    This is a double-edged sword, because even though Jurigged can add live updates to existing scripts with zero lines of additional code, it is not thread safe at all (code could be executed in the middle of an update, which is possibly an inconsistent state).

Other similar efforts:

  • reloading can wrap an iterator to make modifiable for loops. Jurigged cannot do that, but you can use both packages at the same time.
Owner
Olivier Breuleux
Olivier Breuleux
Raganarok X: Next Generation Data Dump

Raganarok X Data Dump Raganarok X: Next Generation Data Dump More interesting Files File Name Contains en_langs All the variables you need in English

14 Jul 15, 2022
A Random Password Generator made from Python

Things you need Python Step 1 Download the python file from Releases Step 2 Go to the directory where the python file is and run it Step 3 Type the le

Kavindu Nimsara 3 May 30, 2022
A python lib for generate random string and digits and special characters or A combination of them

A python lib for generate random string and digits and special characters or A combination of them

Torham 4 Nov 15, 2022
An awesome tool to save articles from RSS feed to Pocket automatically.

RSS2Pocket An awesome tool to save articles from RSS feed to Pocket automatically. About the Project I used to use IFTTT to save articles from RSS fee

Hank Liao 10 Nov 12, 2022
This is Cool Utility tools that you can use in python.

This is Cool Utility tools that you can use in python. There are a few tools that you might find very useful, you can use this on pretty much any project and some utils might help you a lot and save

Senarc Studios 6 Apr 18, 2022
Program to extract signatures from documents.

Extracting Signatures from Bank Checks Introduction Ahmed et al. [1] suggest a connected components-based method for segmenting signatures in document

Muhammad Saif Ullah Khan 9 Jan 26, 2022
Customized python validations.

A customized python validations.

Wilfred V. Pine 2 Apr 20, 2022
kawadi is a versatile tool that used as a form of weapon and is used to cut, shape and split wood.

kawadi kawadi (કવાડિ in Gujarati) (Axe in English) is a versatile tool that used as a form of weapon and is used to cut, shape and split wood. kawadi

Jay Vala 2 Jan 10, 2022
A pythonic dependency injection library.

Pinject Pinject is a dependency injection library for python. The primary goal of Pinject is to help you assemble objects into graphs in an easy, main

Google 1.3k Dec 30, 2022
A python module to update the console without flashing.

A python module to update the console without flashing.

Matthias 112 Dec 19, 2022
Lock files using python and cmd

Python_Lock_Files Lock files using python and cmd license feel free to do whatever you want to with these files, i dont take any responsibility tho, u

1 Nov 01, 2021
The producer-consumer problem implemented with threads in Python

This was developed using a Python virtual environment, I would strongly recommend to do the same if you want to clone this repository. How to run this

Omar Beltran 1 Oct 30, 2021
Cardano Stakepools: Check for scheduled blocks in current epoch.

ReLeaderLogs For Cardano Stakepool Operators: Lightweight Scheduled Blocks Checker for Current Epoch. No cardano-node Required, data is taken from blo

SNAKE (Cardano Stakepool) 2 Oct 19, 2021
SH-PUBLIC is a python based cloning script. You can clone unlimited UID facebook accounts by using this tool.

SH-PUBLIC is a python based cloning script. You can clone unlimited UID facebook accounts by using this tool. This tool works on any Android devices without root.

(Md. Tanvir Ahmed) 5 Mar 09, 2022
Hot reloading for Python

Hot reloading for Python

Olivier Breuleux 769 Jan 03, 2023
python script to generate color coded resistor images

Resistor image generator I got nerdsniped into making this. It's not finished at all, and the code is messy. The end goal it generate a whole E-series

MichD 1 Nov 12, 2021
Python Yeelight YLKG07YL/YLKG08YL dimmer handler

With this class you can receive, decrypt and handle Yeelight YLKG07YL/YLKG08YL dimmer bluetooth notifications in your python code.

12 Dec 26, 2022
Just some scripts to export vector tiles to geojson.

Vector tiles to GeoJSON Nowadays modern web maps are usually based on vector tiles. The great thing about vector tiles is, that they are not just imag

Lilith Wittmann 77 Jul 26, 2022
Plone Interface contracts, plus basic features and utilities

plone.base This package is the base package of the CMS Plone https://plone.org. It contains only interface contracts and basic features and utilitie

Plone Foundation 1 Oct 03, 2022
SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance .

SysInfo SysInfo is an app developed in python which gives Basic System Info , and some detailed graphs of system performance . Installation Download t

5 Nov 08, 2021