Import Python modules from dicts and JSON formatted documents.

Overview

Paker

Build Version Version

Paker is module for importing Python packages/modules from dictionaries and JSON formatted documents. It was inspired by httpimporter.

Important: Since v0.6.0 paker supports importing .pyd and .dll modules directly from memory. This was achieved by using _memimporter from py2exe project. Importing .so files on Linux still requires writing them to disk.

Installation

From PyPI

pip install paker -U

From source

git clone https://github.com/desty2k/paker.git
cd paker
pip install .

Usage

In Python script

You can import Python modules directly from string, dict or bytes (without disk IO).

import paker
import logging

MODULE = {"somemodule": {"type": "module", "extension": "py", "code": "fun = lambda x: x**2"}}
logging.basicConfig(level=logging.NOTSET)

if __name__ == '__main__':
    with paker.loads(MODULE) as loader:
        # somemodule will be available only in this context
        from somemodule import fun
        assert fun(2), 4
        assert fun(5), 25
        print("6**2 is {}".format(fun(6)))
        print("It works!")

To import modules from .json files use load function. In this example paker will serialize and import mss package.

import paker
import logging

file = "mss.json"
logging.basicConfig(level=logging.NOTSET)

# install mss using `pip install mss`
# serialize module
with open(file, "w+") as f:
    paker.dump("mss", f, indent=4)

# now you can uninstall mss using `pip uninstall mss -y`
# load package back from dump file
with open(file, "r") as f:
    loader = paker.load(f)

import mss
with mss.mss() as sct:
    sct.shot()

# remove loader and clean the cache
loader.unload()

try:
    # this will throw error
    import mss
except ImportError:
    print("mss unloaded successfully!")

CLI

Paker can also work as a standalone script. To dump module to JSON dict use dump command:

paker dump mss

To recreate module from JSON dict use load:

paker load mss.json

Show all modules and packages in .json file

paker list mss.json

How it works

When importing modules or packages Python iterates over importers in sys.meta_path and calls find_module method on each object. If the importer returns self, it means that the module can be imported and None means that importer did not find searched package. If any importer has confirmed the ability to import module, Python executes another method on it - load_module. Paker implements its own importer called jsonimporter, which instead of searching for modules in directories, looks for them in Python dictionaries

To dump module or package to JSON document, Paker recursively iterates over modules and creates dict with code and type of each module and submodules if object is package.

You might also like...
An executor that loads ONNX models and embeds documents using the ONNX runtime.

ONNXEncoder An executor that loads ONNX models and embeds documents using the ONNX runtime. Usage via Docker image (recommended) from jina import Flow

Implementation of self-attention mechanisms for general purpose. Focused on computer vision modules. Ongoing repository.
Implementation of self-attention mechanisms for general purpose. Focused on computer vision modules. Ongoing repository.

Self-attention building blocks for computer vision applications in PyTorch Implementation of self attention mechanisms for computer vision in PyTorch

Turning SymPy expressions into PyTorch modules.

sympytorch A micro-library as a convenience for turning SymPy expressions into PyTorch Modules. All SymPy floats become trainable parameters. All SymP

DI-HPC is an acceleration operator component for general algorithm modules in reinforcement learning algorithms

DI-HPC: Decision Intelligence - High Performance Computation DI-HPC is an acceleration operator component for general algorithm modules in reinforceme

Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules
Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

DCSR: Dual Camera Super-Resolution Implementation for our ICCV 2021 oral paper: Dual-Camera Super-Resolution with Aligned Attention Modules paper | pr

Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules
Implementation for our ICCV 2021 paper: Dual-Camera Super-Resolution with Aligned Attention Modules

DCSR: Dual Camera Super-Resolution Implementation for our ICCV 2021 oral paper: Dual-Camera Super-Resolution with Aligned Attention Modules paper | pr

Weight initialization schemes for PyTorch nn.Modules

nninit Weight initialization schemes for PyTorch nn.Modules. This is a port of the popular nninit for Torch7 by @kaixhin. ##Update This repo has been

Pytorch modules for paralel models with same architecture. Ideal for multi agent-based systems
Pytorch modules for paralel models with same architecture. Ideal for multi agent-based systems

WideLinears Pytorch parallel Neural Networks A package of pytorch modules for fast paralellization of separate deep neural networks. Ideal for agent-b

Stacs-ci - A set of modules to enable integration of STACS with commonly used CI / CD systems
Stacs-ci - A set of modules to enable integration of STACS with commonly used CI / CD systems

Static Token And Credential Scanner CI Integrations What is it? STACS is a YARA

Comments
  • psutil example exits with module not found when using _memimporter

    psutil example exits with module not found when using _memimporter

    I pulled latest releases zip file, ran python setup.py build and attempted to run the psutil example with the compiled pyd. This resulted in the following error:

    DEBUG:jsonimporter:searching for pwd
    DEBUG:jsonimporter:searching for psutil._common
    INFO:jsonimporter:psutil._common has been imported successfully
    DEBUG:jsonimporter:searching for psutil._compat
    INFO:jsonimporter:psutil._compat has been imported successfully
    DEBUG:jsonimporter:searching for psutil._pswindows
    DEBUG:jsonimporter:searching for psutil._psutil_windows
    DEBUG:jsonimporter:searching for psutil._psutil_windows
    INFO:jsonimporter:using _memimporter to load '.pyd' file
    INFO:jsonimporter:unloaded all modules
    Traceback (most recent call last):
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\psutil_example.py", line 20, in <module>
        import psutil
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\paker\importers\jsonimporter.py", line 115, in load_module
        exec(jsonmod["code"], mod.__dict__)
      File "<string>", line 107, in <module>
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\paker\importers\jsonimporter.py", line 115, in load_module
        exec(jsonmod["code"], mod.__dict__)
      File "<string>", line 35, in <module>
      File "c:\Users\User\Desktop\paker-0.7.1\paker-0.7.1\build\lib.win-amd64-cpython-310\paker\importers\jsonimporter.py", line 134, in load_module
        mod = _memimporter.import_module(fullname, path, initname, self._get_data, spec)
    ImportError: MemoryLoadLibrary failed loading psutil\_psutil_windows.pyd: The specified module could not be found. (126)
    

    Is this an issue with how I compiled memimporter, or something else?

    opened by rkbennett 1
Releases(v0.7.1)
Owner
Wojciech Wentland
Wojciech Wentland
TANL: Structured Prediction as Translation between Augmented Natural Languages

TANL: Structured Prediction as Translation between Augmented Natural Languages Code for the paper "Structured Prediction as Translation between Augmen

98 Dec 15, 2022
SOLOv2 on onnx & tensorRT

SOLOv2.tensorRT: NOTE: code based on WXinlong/SOLO add support to TensorRT inference onnxruntime tensorRT full_dims and dynamic shape postprocess with

47 Nov 26, 2022
AI virtual gym is an AI program which can be used to exercise and can be used to see if we are doing the exercises

AI virtual gym is an AI program which can be used to exercise and can be used to see if we are doing the exercises

4 Feb 13, 2022
Modification of convolutional neural net "UNET" for image segmentation in Keras framework

ZF_UNET_224 Pretrained Model Modification of convolutional neural net "UNET" for image segmentation in Keras framework Requirements Python 3.*, Keras

209 Nov 02, 2022
GeDML is an easy-to-use generalized deep metric learning library

GeDML is an easy-to-use generalized deep metric learning library

Borui Zhang 32 Dec 05, 2022
Fuzzification helps developers protect the released, binary-only software from attackers who are capable of applying state-of-the-art fuzzing techniques

About Fuzzification Fuzzification helps developers protect the released, binary-only software from attackers who are capable of applying state-of-the-

gts3.org (<a href=[email protected])"> 55 Oct 25, 2022
Code and data (Incidents Dataset) for ECCV 2020 Paper "Detecting natural disasters, damage, and incidents in the wild".

Incidents Dataset See the following pages for more details: Project page: IncidentsDataset.csail.mit.edu. ECCV 2020 Paper "Detecting natural disasters

Ethan Weber 67 Dec 27, 2022
A new version of the CIDACS-RL linkage tool suitable to a cluster computing environment.

Fully Distributed CIDACS-RL The CIDACS-RL is a brazillian record linkage tool suitable to integrate large amount of data with high accuracy. However,

Robespierre Pita 5 Nov 04, 2022
学习 python3 以来写的一些垃圾玩具……

和东哥做兄弟 Author: chiupam 版权 未经本人同意,仓库内所有资源文件,禁止任何公众号、自媒体、开发者进行任何形式的转载、发布、搬运。 声明 这不是一个开源项目,只是把 GitHub 当作一个代码的存储空间,本项目不接受任何开源要求。 仅用于学习研究,禁止用于商业用途,不能保证其合法性

Chiupam 67 Mar 26, 2022
The Official PyTorch Implementation of DiscoBox.

DiscoBox: Weakly Supervised Instance Segmentation and Semantic Correspondence from Box Supervision Paper | Project page | Demo (Youtube) | Demo (Bilib

NVIDIA Research Projects 89 Jan 09, 2023
EssentialMC2 Video Understanding

EssentialMC2 Introduction EssentialMC2 is a complete system to solve video understanding tasks including MHRL(representation learning), MECR2( relatio

Alibaba 106 Dec 11, 2022
Spatial-Location-Constraint-Prototype-Loss-for-Open-Set-Recognition

Spatial Location Constraint Prototype Loss for Open Set Recognition Official PyTorch implementation of "Spatial Location Constraint Prototype Loss for

Xia Ziheng 12 Jun 24, 2022
Activity image-based video retrieval

Cross-modal-retrieval Our approach is focus on Activity Image-to-Video Retrieval (AIVR) task. The compared methods are state-of-the-art single modalit

BCMI 75 Oct 21, 2021
Simple image captioning model - CLIP prefix captioning.

Simple image captioning model - CLIP prefix captioning.

688 Jan 04, 2023
Is RobustBench/AutoAttack a suitable Benchmark for Adversarial Robustness?

Adversrial Machine Learning Benchmarks This code belongs to the papers: Is RobustBench/AutoAttack a suitable Benchmark for Adversarial Robustness? Det

Adversarial Machine Learning 9 Nov 27, 2022
Constructing interpretable quadratic accuracy predictors to serve as an objective function for an IQCQP problem that represents NAS under latency constraints and solve it with efficient algorithms.

IQNAS: Interpretable Integer Quadratic programming Neural Architecture Search Realistic use of neural networks often requires adhering to multiple con

0 Oct 24, 2021
This is the code repository for the paper A hierarchical semantic segmentation framework for computer-vision-based bridge column damage detection

Bridge-damage-segmentation This is the code repository for the paper A hierarchical semantic segmentation framework for computer-vision-based bridge c

Jingxiao Liu 5 Dec 07, 2022
My implementation of DeepMind's Perceiver

DeepMind Perceiver (in PyTorch) Disclaimer: This is not official and I'm not affiliated with DeepMind. My implementation of the Perceiver: General Per

Louis Arge 55 Dec 12, 2022
A minimalist tool to display a network graph.

A tool to get a minimalist view of any architecture This tool has only be tested with the models included in this repo. Therefore, I can't guarantee t

Thibault Castells 1 Feb 11, 2022
A graph adversarial learning toolbox based on PyTorch and DGL.

GraphWar: Arms Race in Graph Adversarial Learning NOTE: GraphWar is still in the early stages and the API will likely continue to change. 🚀 Installat

Jintang Li 54 Jan 05, 2023