Cached file system for online resources in Python

Overview

Minato

Actions Status Python version License pypi version

Cache & file system for online resources in Python

Features

Minato enables you to:

  • Download & cache online recsources
    • minato supports the following protocols: HTTP(S) / AWS S3 / Google Cloud Storage
    • You can manage cached files via command line interface
  • Automatically update cached files based on ETag
    • minato downloads new versions if available when you access cached files
  • Open online files super easily
    • By using minato.open, you can read/write online resources like the built-in open method

Installation

pip install minato[all]

Usage

Python

import minato

# Read / write files on online storage
with minato.open("s3://your_bucket/path/to/file", "w") as f:
    f.write("Create a new file on AWS S3!")

# Cache & manage online resources in local storage
local_filename = minato.cached_path("http://example.com/path/to/archive.zip!inner/path/to/file")

CLI

❯ poetry run minato --help
usage: minato

positional arguments:
  {cache,list,remove,update}
    cache               cache remote file and return cached local file path
    list                show list of cached files
    remove              remove cached files
    update              update cached files

optional arguments:
  -h, --help            show this help message and exit
  --version             show program's version number and exit
You might also like...
Exploit Discord's cache system to remote upload payloads on Discord users machines
Exploit Discord's cache system to remote upload payloads on Discord users machines

Exploit Discord's cache system to hide payloads PoC Remote upload embedded payload from image using EOF to Discord users machines through cache. Depen

A generic system for filtering Django QuerySets based on user selections

Django Filter Django-filter is a reusable Django application allowing users to declaratively add dynamic QuerySet filtering from URL parameters. Full

A Django based shop system

django-SHOP Django-SHOP aims to be a the easy, fun and fast e-commerce counterpart to django-CMS. Here you can find the full documentation for django-

Login System Django

Login-System-Django Login System Using Django Tech Used Django Python Html Run Locally Clone project git clone https://link-to-project Get project for

A Blog Management System Built with django
A Blog Management System Built with django

Blog Management System Backend use: Django Features Enhanced Ui

This repository contains django library management system project.

Library Management System Django ** INSTALLATION** First of all install python on your system. Then run pip install -r requirements.txt to required se

Djang Referral System
Djang Referral System

Djang Referral System About | Features | Technologies | Requirements | Starting | License | Author 🎯 About I created django referral system and I wan

Advanced school management system written in Django :)
Advanced school management system written in Django :)

Advanced school management system written in Django :) ⚙️ Config the project First you should make venv for this project. So in the main root of proje

A visual indicator of what environment/system you're using in django

A visual indicator of what environment/system you're using in django

Comments
  • Bump protobuf from 4.21.5 to 4.21.6

    Bump protobuf from 4.21.5 to 4.21.6

    Bumps protobuf from 4.21.5 to 4.21.6.

    Release notes

    Sourced from protobuf's releases.

    Protocol Buffers v3.0.0-beta-4

    Version 3.0.0-beta-4

    General

    • Added a deterministic serialization API for C++. The deterministic serialization guarantees that given a binary, equal messages will be serialized to the same bytes. This allows applications like MapReduce to group equal messages based on the serialized bytes. The deterministic serialization is, however, NOT canonical across languages; it is also unstable across different builds with schema changes due to unknown fields. Users who need canonical serialization, e.g. persistent storage in a canonical form, fingerprinting, etc, should define their own canonicalization specification and implement the serializer using reflection APIs rather than relying on this API.

    • Added OneofOptions. You can now define custom options for oneof groups.

      import "google/protobuf/descriptor.proto";
      extend google.protobuf.OneofOptions {
        optional int32 my_oneof_extension = 12345;
      }
      message Foo {
        oneof oneof_group {
          (my_oneof_extension) = 54321;
          ...
        }
      }
      

    C++ (beta)

    • Introduced a deterministic serialization API in CodedOutputStream::SetSerializationDeterministic(bool). See the notes about deterministic serialization in the General section.
    • Added google::protobuf::Map::swap() to swap two map fields.
    • Fixed a memory leak when calling Reflection::ReleaseMessage() on a message allocated on arena.
    • Improved error reporting when parsing text format protos.
    • JSON
      • Added a new parser option to ignore unknown fields when parsing JSON.
      • Added convenient methods for message to/from JSON conversion.
    • Various performance optimizations.

    Java (beta)

    • File option "java_generate_equals_and_hash" is now deprecated. equals() and hashCode() methods are generated by default.
    • Added a new JSON printer option "omittingInsignificantWhitespace" to produce a more compact JSON output. The printer will pretty-print by default.
    • Updated Java runtime to be compatible with 2.5.0/2.6.1 generated protos.

    Python (beta)

    ... (truncated)

    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • Bump ujson from 5.3.0 to 5.4.0

    Bump ujson from 5.3.0 to 5.4.0

    Bumps ujson from 5.3.0 to 5.4.0.

    Release notes

    Sourced from ujson's releases.

    5.4.0

    Added

    Fixed

    Commits
    • 9c20de0 Merge pull request from GHSA-fm67-cv37-96ff
    • b21da40 Fix double free on string decoding if realloc fails
    • 67ec071 Merge pull request #555 from JustAnotherArchivist/fix-decode-surrogates-2
    • bc7bdff Replace wchar_t string decoding implementation with a uint32_t-based one
    • cc70119 Merge pull request #548 from JustAnotherArchivist/arbitrary-ints
    • 4b5cccc Merge pull request #553 from bwoodsend/pypy-ci
    • abe26fc Merge pull request #551 from bwoodsend/bye-bye-travis
    • 3efb5cc Delete old TravisCI workflow and references.
    • 404de1a xfail test_decode_surrogate_characters() on Windows PyPy.
    • f7e66dc Switch to musl docker base images.
    • Additional commits viewable in compare view

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • CVE-2007-4559 Patch

    CVE-2007-4559 Patch

    Patching CVE-2007-4559

    Hi, we are security researchers from the Advanced Research Center at Trellix. We have began a campaign to patch a widespread bug named CVE-2007-4559. CVE-2007-4559 is a 15 year old bug in the Python tarfile package. By using extract() or extractall() on a tarfile object without sanitizing input, a maliciously crafted .tar file could perform a directory path traversal attack. We found at least one unsantized extractall() in your codebase and are providing a patch for you via pull request. The patch essentially checks to see if all tarfile members will be extracted safely and throws an exception otherwise. We encourage you to use this patch or your own solution to secure against CVE-2007-4559. Further technical information about the vulnerability can be found in this blog.

    If you have further questions you may contact us through this projects lead researcher Kasimir Schulz.

    opened by TrellixVulnTeam 0
Releases(v0.10.0)
  • v0.10.0(Aug 11, 2022)

    What's Changed

    • Bump ujson from 5.3.0 to 5.4.0 by @dependabot in https://github.com/altescy/minato/pull/36
    • Table supports full-width characters by @altescy in https://github.com/altescy/minato/pull/37
    • Modify auto-format settings by @altescy in https://github.com/altescy/minato/pull/38
    • Add progress & remove tqdm by @altescy in https://github.com/altescy/minato/pull/39
    • Add disable-progressbar option by @altescy in https://github.com/altescy/minato/pull/40
    • Add interactive selector by @altescy in https://github.com/altescy/minato/pull/41
    • Bump version number to v0.10.0 by @altescy in https://github.com/altescy/minato/pull/42

    New Contributors

    • @dependabot made their first contribution in https://github.com/altescy/minato/pull/36

    Full Changelog: https://github.com/altescy/minato/compare/v0.9.0...v0.10.0

    Source code(tar.gz)
    Source code(zip)
  • v0.9.0(Jul 2, 2022)

    What's Changed

    • Use options for open_file function by @altescy in https://github.com/altescy/minato/pull/33
    • Support Python 3.9 and 3.10 by @altescy in https://github.com/altescy/minato/pull/34
    • Bump version number to v0.9.0 by @altescy in https://github.com/altescy/minato/pull/35

    Full Changelog: https://github.com/altescy/minato/compare/v0.8.2...v0.9.0

    Source code(tar.gz)
    Source code(zip)
  • v0.8.2(Feb 24, 2022)

    What's Changed

    • Modify log level by @altescy in https://github.com/altescy/minato/pull/30
    • Upgrade dependent packages by @altescy in https://github.com/altescy/minato/pull/31
    • Bump version number to v0.8.2 by @altescy in https://github.com/altescy/minato/pull/32

    Full Changelog: https://github.com/altescy/minato/compare/v0.8.1...v0.8.2

    Source code(tar.gz)
    Source code(zip)
  • v0.8.1(Feb 11, 2022)

    What's Changed

    • Fix s3 progress by @altescy in https://github.com/altescy/minato/pull/27
    • Add all command to Makefile by @altescy in https://github.com/altescy/minato/pull/28
    • Bump version number to v0.8.1 by @altescy in https://github.com/altescy/minato/pull/29

    Full Changelog: https://github.com/altescy/minato/compare/v0.8.0...v0.8.1

    Source code(tar.gz)
    Source code(zip)
  • v0.8.0(Feb 8, 2022)

    What's Changed

    • Modify tqdm progress by @altescy in https://github.com/altescy/minato/pull/22
    • Upgrade dependent packages by @altescy in https://github.com/altescy/minato/pull/23
    • Split github workflows into ci and pypi-publish by @altescy in https://github.com/altescy/minato/pull/24
    • Modify type annotations by @altescy in https://github.com/altescy/minato/pull/25
    • Bump version number to v0.8.0 by @altescy in https://github.com/altescy/minato/pull/26

    Full Changelog: https://github.com/altescy/minato/compare/v0.7.0...v0.8.0

    Source code(tar.gz)
    Source code(zip)
  • v0.7.0(Nov 19, 2021)

    • Shrink table column width automatically (#16)
    • Modify S3 download (#17)
    • Move boto3-stubs to dev-dependencies (#18)
    • Add upload method to S3FileSystem (#19)
    • Add upload method to GCSFileSystem (#20)
    • Bump version number to v0.7.0 (#21)
    Source code(tar.gz)
    Source code(zip)
  • v0.6.2(Oct 23, 2021)

  • v0.6.1(Oct 1, 2021)

  • v0.6.0(Sep 30, 2021)

    • Remove quotations of S3 Etags (#2)
    • Modify CI settings (#3)
    • Update dependent packages (#4)
    • Make filesystems optional (#5)
    • Add opening options (#6)
    • Add helps to each command line options (#7)
    • Add feature description to readme (#8)
    • Bump version number to v0.6.0 (#9)
    Source code(tar.gz)
    Source code(zip)
  • v0.5.0(Jul 31, 2021)

  • v0.4.0(Jun 25, 2021)

    • Support file-level expiration
    • Enable automatic cache updating based on HTTP ETag
    • Show download progress of S3 and GCS
    • Deprecate upload/download commands
    Source code(tar.gz)
    Source code(zip)
  • v0.3.0(Jun 23, 2021)

Owner
Yasuhiro Yamaguchi
Research Engineer / NLP / ML
Yasuhiro Yamaguchi
Bootstrap 4 integration with Django.

django-bootstrap 4 Bootstrap 4 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 4. Requirements Pytho

Zostera B.V. 980 Dec 29, 2022
Probably the best abstract model / admin for your tree based stuff.

django-treenode Probably the best abstract model / admin for your tree based stuff. Features Fast - get ancestors, children, descendants, parent, root

Fabio Caccamo 360 Jan 05, 2023
A Django Online Library Management Project.

Why am I doing this? I started learning 📖 Django few months back, and this is a practice project from MDN Web Docs that touches the aspects of Django

1 Nov 13, 2021
Django Email Sender

Email-Sender Django Email Sender Installation 1.clone Repository & Install Packages git clone https://github.com/telman03/Email-Sender.git pip install

Telman Gadimov 0 Dec 26, 2021
Django-powered application about blockchain (bitcoin)

Django-powered application about blockchain (bitcoin)

Igor Izvekov 0 Jun 23, 2022
Money fields for Django forms and models.

django-money A little Django app that uses py-moneyed to add support for Money fields in your models and forms. Django versions supported: 1.11, 2.1,

1.4k Jan 06, 2023
Django-pwned - A collection of django password validators

Django Pwned A collection of django password validators. Compatibility Python: 3

Quera 22 Jun 27, 2022
Django/Jinja template indenter

DjHTML A pure-Python Django/Jinja template indenter without dependencies. DjHTML is a fully automatic template indenter that works with mixed HTML/CSS

Return to the Source 378 Jan 01, 2023
Easy thumbnails for Django

Easy Thumbnails A powerful, yet easy to implement thumbnailing application for Django 1.11+ Below is a quick summary of usage. For more comprehensive

Chris Beaven 1.3k Dec 30, 2022
A Django app to initialize Sentry client for your Django applications

Dj_sentry This Django application intialize Sentry SDK to your Django application. How to install You can install this packaging by using: pip install

Gandi 1 Dec 09, 2021
Django models and endpoints for working with large images -- tile serving

Django Large Image Models and endpoints for working with large images in Django -- specifically geared towards geospatial tile serving. DISCLAIMER: th

Resonant GeoData 42 Dec 17, 2022
Analytics services for Django projects

django-analytical The django-analytical application integrates analytics services into a Django project. Using an analytics service with a Django proj

Jazzband 1.1k Dec 31, 2022
Official Python agent for the Elastic APM

elastic-apm -- Elastic APM agent for Python This is the official Python module for Elastic APM. It provides full out-of-the-box support for many of th

elastic 369 Jan 05, 2023
Coltrane - A simple content site framework that harnesses the power of Django without the hassle.

coltrane A simple content site framework that harnesses the power of Django without the hassle. Features Can be a standalone static site or added to I

Adam Hill 58 Jan 02, 2023
The pytest framework makes it easy to write small tests, yet scales to support complex functional testing

The pytest framework makes it easy to write small tests, yet scales to support complex functional testing for applications and libraries. An example o

pytest-dev 9.6k Jan 06, 2023
Agenda feita usando o django para adicionar eventos

Agenda de Eventos Projeto Agenda com Django Inicio O projeto foi iniciado no Django, usando o models.py foi adicionado os dados dos eventos e feita as

Bruno Fernandes 1 Apr 14, 2022
Automatic caching and invalidation for Django models through the ORM.

Cache Machine Cache Machine provides automatic caching and invalidation for Django models through the ORM. For full docs, see https://cache-machine.re

846 Nov 26, 2022
Simpliest django(uvicorn)+postgresql+nginx docker-compose (ready for production and dev)

simpliest django(uvicorn)+postgresql+nginx docker-compose (ready for production and dev) To run in production: docker-compose up -d Site available on

Artyom Lisovskii 1 Dec 16, 2021
Django based webapp pulling in crypto news and price data via api

Deploy Django in Production FTA project implementing containerization of Django Web Framework into Docker to be placed into Azure Container Services a

0 Sep 21, 2022
Django Livre Bank

Django Livre Bank Projeto final da academia Construdelas. API de um banco fictício com clientes, contas e transações. Integrantes da equipe Bárbara Sa

Cecília Costa 3 Dec 22, 2021