importlib_resources is a backport of Python standard library importlib.resources module for older Pythons.

Overview
tests Code style: Black https://readthedocs.org/projects/importlib-resources/badge/?version=latest https://img.shields.io/badge/skeleton-2021-informational

importlib_resources is a backport of Python standard library importlib.resources module for older Pythons.

The key goal of this module is to replace parts of pkg_resources with a solution in Python's stdlib that relies on well-defined APIs. This makes reading resources included in packages easier, with more stable and consistent semantics.

Compatibility

New features are introduced in this third-party library and later merged into CPython. The following table indicates which versions of this library were contributed to different versions in the standard library:

importlib_resources stdlib
5.2 3.11
5.0 3.10
1.3 3.9
0.5 (?) 3.7
Comments
  • Allow resources to be in subdirectories

    Allow resources to be in subdirectories

    In GitLab by @rob.speer on May 16, 2018, 01:23

    Suppose I have a Web application, myapp, that needs to serve a static file, which by convention needs to be in the path myapp/static/ld/context.ld.json. Suppose I also want to be able to access that file from Python code, because its contents are used in a test.

    As importlib_resources is currently defined, I would need to rewrite the path as if it were a Python submodule, even though it does not contain actual Python code: path(myapp.static.ld, "context.ld.json"). I would also need to create empty files named myapp/static/__init__.py and myapp/static/ld/__init__.py, and hopefully exclude them from being served as static files.

    That would be enough for me to give up and use paths relative to __file__ instead. In general, I would heartily recommend importlib if I could reasonably promise that it was an improvement over using __file__ or over existing uses of pkg_resources, which it wouldn't be if it doesn't support subdirectories.

    The call I would like to be able to make in this situation is path(myapp, "static/ld/context.ld.json").

    enhancement 
    opened by jaraco 43
  • Please drop setuptools_scm

    Please drop setuptools_scm

    This kind of release process is a huge nuisance for those of us packaging software downstream. Not being able to use PyPI tarballs or even GitHub tarballs is requires us to go well out of our way to accomodate your package. Please just conform to normal Python packaging norms.

    opened by ddevault 34
  • Unable to retrieve resources from a namespace package

    Unable to retrieve resources from a namespace package

    In GitLab by @jaraco on Nov 2, 2018, 02:48

    Attempting to retrieve resources from a namespace package fails.

    draft $ mkdir foo
    draft $ touch foo/bar.txt
    draft $ rwt importlib_resources
    Collecting importlib_resources
      Using cached https://files.pythonhosted.org/packages/2f/f7/b4aa02cdd3ee7ebba375969d77c00826aa15c5db84247d23c89522dccbfa/importlib_resources-1.0.2-py2.py3-none-any.whl
    Installing collected packages: importlib-resources
    Successfully installed importlib-resources-1.0.2
    Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
    [Clang 6.0 (clang-600.0.57)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import importlib_resources
    >>> importlib_resources.read_text(__import__('foo'), 'bar.txt')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/resources.py", line 169, in read_text
        with open_text(package, resource, encoding, errors) as fp:
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/resources.py", line 126, in open_text
        _check_location(package)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/importlib/resources.py", line 82, in _check_location
        raise FileNotFoundError(f'Package has no location {package!r}')
    FileNotFoundError: Package has no location <module 'foo' (namespace)>
    

    I see an obvious problem here - that a namespace package can have more than one base path, so it has no single location. But it does have a location... and pkg_resources lets one load resources from namespace package:

    draft $ cat > foo/__init__.py
    import pkg_resources; pkg_resources.declare_namespace('foo')
    draft $ python
    Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
    [Clang 6.0 (clang-600.0.57)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pkg_resources
    >>> pkg_resources.resource_stream('foo', 'bar.txt')
    <_io.BufferedReader name='/Users/jaraco/draft/foo/bar.txt'>
    

    pkg_resources doesn't succeed with a PEP 420 namespace package:

    $ rm foo/__init__.py
    >>> pkg_resources.resource_stream('foo', 'bar.txt')
    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1145, in resource_stream
        return get_provider(package_or_requirement).get_resource_stream(
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 359, in get_provider
        return _find_adapter(_provider_factories, loader)(module)
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/pkg_resources/__init__.py", line 1387, in __init__
        self.module_path = os.path.dirname(getattr(module, '__file__', ''))
      File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/posixpath.py", line 156, in dirname
        p = os.fspath(p)
    TypeError: expected str, bytes or os.PathLike object, not NoneType
    

    But even in that situation, it does allow for loading resources for a module within a PEP 420 namespace package:

    draft $ touch foo/mod.py
    draft $ tree
    .
    └── foo
        ├── bar.txt
        └── mod.py
    draft $ python
    Python 3.7.1 (v3.7.1:260ec2c36a, Oct 20 2018, 03:13:28)
    [Clang 6.0 (clang-600.0.57)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> import pkg_resources
    >>> pkg_resources.resource_stream('foo.mod', 'bar.txt')
    <_io.BufferedReader name='/Users/jaraco/draft/foo/bar.txt'>
    

    This issue sort-of relates to #60, but is more serious because it seems there's no input that importlib_resources can accept to load resources from a namespace package.

    The issue emerged in pmxbot when I tried to convert the package from a (deprecated) pkg_resources-style namespace package to a PEP 420 namespace package. The code failed at this line when it tried to load the phrases.

    It seems to me (without looking at the code) it should be straightforward to support loading resources from namespace packages, following the same logic that importlib follows to import a module from that package.

    The only other option I see is to force packages to rewrite their packages to only put resources in non-namespace packages, which is a bit of an imposition and unintuitive constraint.

    enhancement api 
    opened by jaraco 32
  • Thoughts on the current API

    Thoughts on the current API

    In GitLab by @warsaw on Nov 13, 2017, 12:27

    Over the weekend, I wrote an experimental branch to switch Mailman 3 to importlib_resources. Sorry for the catch-all issue, but I want to at least capture my thoughts on the current API. We can turn these into separate issues if/when we want to change anything. And at least this should serve as a good conversion guide for #16 .

    • open() isn't a very good name, especially if you tend to use from importlib_resources import style imports. I do like the shorter name, but the problem is obvious: this shadows the built-in open() function. In my branch, I've used from importlib_resources import open as resources_open but that's less than ideal. As open() is also the obvious choice for the function name, I don't yet have a good suggested alternative.

    • path() returning a pathlib.Path is problematic. It's likely the right thing to do, but it means that conversion requires the liberal use of str() wrapping, since much existing code is expecting a concrete str object. Maybe this is better in 3.7 (I only tested with 3.6) for stdlib APIs, but won't help you much for internal project APIs that expect a string. Interestingly, this tends to be a bit easier if you use an ExitStack rather than a with-statement. Compare:

            with path('mailman.testing', 'config-with-instances.pck') as pckpath:
                pckfile = str(pckpath)
    

    and

            with ExitStack() as resources:
                pckfile = str(resources.enter_context(
                    path('mailman.testing', 'config.pck')))
    
    • Returning a context from path() is often inconvenient. Of course, I understand why we have to do that, but ensuring that the context is properly cleaned up can be tricky, and it's certainly much less convenient at times. I've had to change a few internal APIs to accept an ExitStack argument, and then add the path context to that, since the resources have to be managed several call frames higher. E.g.
    @public
    def use_test_organizational_data(resources):
        # Point the organizational URL to our test data.
        filename = str(resources.enter_context(
            path('mailman.rules.tests.data', 'org_domain.txt')))
        url = 'file:///{}'.format(filename)
        return resources.enter_context(
            configuration('dmarc', org_domain_data_url=url))
    
    • Missing API. I couldn't totally rid myself of pkg_resources because there's no current alternative to resource_isdir() and resource_listdir().

    • read() should support binary reads. I know we have open() for that, but it turns out to be convenient if read(encoding=None) returned bytes. I know we talked about that before, but it's become more obvious when using this in real-world code. Conversely, it's still a bit inconvenient that open() only returns a file opened in binary mode. Fortunately, there's email.message_from_binary_file() where I was previously using email.message_from_file(), so it wasn't difficult to fix, but it does mean that converting from pkg_resources to importlib_resources is less mechanical.

    opened by jaraco 26
  • WIP: importlib_resources>=1.1.0 Pulp 2.21 Sync/Upload Fail

    WIP: importlib_resources>=1.1.0 Pulp 2.21 Sync/Upload Fail

    In GitLab by @goshansp on Aug 21, 2020, 09:38

    Scope

    We are unable to upload or sync importlib_resources>=1.1.0 into our python repository (rhel7-pulp2.21-python2.7.5). Until version 1.0.2 this went fine.

    Workaround rhel7-pulp2.21-python2.7.5 (Manual Upload to Pulp)

    1. delete / create pytho repo and dont add importlib_resources via feed
    2. download importlib_resources-1.0.2 from pypi
    3. upload / publish
    4. profit

    other packages that dont cause said issue

    ansible,ansible-base,ansible-lint,ansible-tower-cli,anyconfig,arrow,asn1crypto,aspy-yaml,atomicwrites,attrs,babel,backports-ssl-match-hostname,bcrypt,binaryornot,cerberus,certifi,cffi,cfgv,chardet,click,click_help_colors,click-completion,commonmark,colorama,configparser,contextlib2,cookiecutter,cryptography,dataclasses,debtcollector,distro,dnspython,docker,docker-py,docker-pycreds,dparse,entrypoints,enum34,fasteners,flake8,funcsigs,functools32,future,git-url-parse,identify,idna,importlib-metadata,infoblox-client,ipaddress,iso8601,jinja2,jinja2-time,jmespath,lxml,markupsafe,mccabe,molecule,monotonic,more-itertools,msgpack,netaddr,netifaces,nodeenv,ntlm-auth,oslo-config,oslo-context,oslo-i18n,oslo-log,oslo-serialization,oslo-utils,packaging,paramiko,pathlib2,pathspec,pbr,pexpect,pip,pluggy,poyo,pre-commit,prometheus-client,psutil,ptyprocess,py,pyasn1,pycodestyle,pycparser,pydf,pygments,pyflakes,pyinotify,pykerberos,pynacl,pyopenssl,pyparsing,pytest,python-dateutil,python-gilt,python-gitlab,python-memcached,python-slugify,pytz,pyup-tools,pyvmomi,pywinrm,pyyaml,requests,requests-kerberos,requests-ntlm,rich,rfc3986,ruamel-ordereddict,ruamel-yaml,ruamel-yaml-clib,safety,selinux,scandir,setuptools,sh,shellingham,six,stevedore,tabulate,testinfra,text-unidecode,toml,tree-format,typing,typing-extensions,urllib3,virtualenv,wcwidth,websocket-client,whichcraft,wrapt,xmltodict,yamllint,yq,zipp

    suspected cause

    There seems to be incompatibilities with the versioning. Possibly the importer gets 0.0.0 from two packages resulting in an error during sync. This may be due to setuptools==0.9.8

    reproduction pulp sync fail

     $ pulp-admin python repo create $pulp_proxy --repo-id python-test --feed https://pypi.org/ --package-names importlib_resources
     $ pulp-admin python repo sync run --repo-id python-test
    +----------------------------------------------------------------------+
                     Synchronizing Repository [python-test]
    +----------------------------------------------------------------------+
    
    This command may be exited via ctrl+c without affecting the request.
    
    Downloading Python metadata.
    [-]
    ... completed
    
    Copying units already in pulp
    [\]
    ... completed
    
    Downloading and processing Python packages.
    [\]
    ... completed
    
    Task Succeeded
    
    Task Failed
    
    [Errno 17] File exists:
    '/var/cache/pulp/[email protected]/c895ee09-f18d-417
    8-85d6-95d26b0a5bc8/python-test/simple/importlib-resources'
    $ rpm -qa | grep pulp
    pulp-python-plugins-2.0.3-1.el7.noarch
    pulp-server-2.21.0-1.el7.noarch
    pulp-puppet-plugins-2.21.0-1.el7.noarch
    python-pulp-bindings-2.21.0-1.el7.noarch
    pulp-consumer-client-2.21.0-1.el7.noarch
    pulp-puppet-admin-extensions-2.21.0-1.el7.noarch
    pulp-selinux-2.21.0-1.el7.noarch
    python-pulp-python-common-2.0.3-1.el7.noarch
    python-pulp-rpm-common-2.21.0-1.el7.noarch
    python-pulp-agent-lib-2.21.0-1.el7.noarch
    python-pulp-oid_validation-2.21.0-1.el7.noarch
    pulp-rpm-plugins-2.21.0-1.el7.noarch
    pulp-rpm-admin-extensions-2.21.0-1.el7.noarch
    python-isodate-0.5.0-4.pulp.el7.noarch
    pulp-docker-admin-extensions-3.2.5-1.el7.noarch
    pulp-python-admin-extensions-2.0.3-1.el7.noarch
    python-pulp-repoauth-2.21.0-1.el7.noarch
    python-pulp-client-lib-2.21.0-1.el7.noarch
    pulp-agent-2.21.0-1.el7.noarch
    pulp-rpm-handlers-2.21.0-1.el7.noarch
    pulp-docker-plugins-3.2.5-1.el7.noarch
    python-pulp-common-2.21.0-1.el7.noarch
    pulp-admin-client-2.21.0-1.el7.noarch
    pulp-rpm-consumer-extensions-2.21.0-1.el7.noarch
    python-pulp-docker-common-3.2.5-1.el7.noarch
    python-pulp-puppet-common-2.21.0-1.el7.noarch
    pulp-rpm-yumplugins-2.21.0-1.el7.noarch
    

    journal -f / publish fail

    Aug 26 10:34:49 redacted pulp[1866]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[2e2d5a08-05e3-4673-9fcc-4bce3b8b86da]
    Aug 26 10:34:49 redacted pulp[2223]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.sync.sync[8266d2ef-391c-41a1-8b6d-c4ef8454d8a1]
    Aug 26 10:34:49 redacted pulp[3098]: celery.app.trace:INFO: [2e2d5a08] Task pulp.server.async.tasks._queue_reserved_task[2e2d5a08-05e3-4673-9fcc-4bce3b8b86da] succeeded in 0.0141177220503s: None
    Aug 26 10:34:49 redacted pulp[2223]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[28b04604-c48c-4d50-91d9-083c3429fb57]
    Aug 26 10:34:49 redacted pulp[3074]: requests.packages.urllib3.connectionpool:INFO: Starting new HTTPS connection (1): pypi.org
    Aug 26 10:34:49 redacted pulp[3074]: nectar.downloaders.threaded:INFO: Download succeeded: https://pypi.org/pypi/importlib_resources/json.
    Aug 26 10:34:49 redacted pulp[3074]: pulp_python.plugins.importers.sync:INFO: Processing metadata retrieved from https://pypi.org/pypi/importlib_resources/json.
    Aug 26 10:34:51 redacted pulp[1866]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._queue_reserved_task[f2abd34d-6883-498d-9d39-49a7b976f3d4]
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:INFO: [8266d2ef] Task pulp.server.managers.repo.sync.sync[8266d2ef-391c-41a1-8b6d-c4ef8454d8a1] succeeded in 2.19935470994s: <pulp.server.async.tasks.TaskResult object at 0x7f96e4aebc10>
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:INFO: [28b04604] Task pulp.server.async.tasks._release_resource[28b04604-c48c-4d50-91d9-083c3429fb57] succeeded in 0.00107404403389s: None
    Aug 26 10:34:51 redacted pulp[2223]: celery.worker.strategy:INFO: Received task: pulp.server.managers.repo.publish.publish[c895ee09-f18d-4178-85d6-95d26b0a5bc8]
    Aug 26 10:34:51 redacted pulp[3098]: celery.app.trace:INFO: [f2abd34d] Task pulp.server.async.tasks._queue_reserved_task[f2abd34d-6883-498d-9d39-49a7b976f3d4] succeeded in 0.015954433009s: None
    Aug 26 10:34:51 redacted pulp[2223]: celery.worker.strategy:INFO: Received task: pulp.server.async.tasks._release_resource[a8098104-c1e6-45a4-bd0b-5850aef21e40]
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488) Exception caught from plugin during publish for repo [python-test]
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488) Traceback (most recent call last):
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1259, in _do_publish
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     publish_report = publish_repo(transfer_repo, conduit, call_config)
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 894, in wrap_f
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     return f(*args, **kwargs)
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/web.py", line 94, in publish_repo
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     return self._publisher.process_lifecycle()
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     super(PluginStep, self).process_lifecycle()
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     step.process()
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     self._process_block()
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     self.process_main()
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 69, in process_main
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     self.write_simple_api(projects)
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 100, in write_simple_api
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     PublishMetadataStep._create_project_index(project_name, simple_path, packages)
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 118, in _create_project_index
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     os.makedirs(project_path)
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488)     mkdir(name, mode)
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.controllers.repository:ERROR: [c895ee09] (3074-35488) OSError: [Errno 17] File exists: '/var/cache/pulp/[email protected]/c895ee09-f18d-4178-85d6-95d26b0a5bc8/python-test/simple/importlib-resources'
    Aug 26 10:34:51 redacted pulp[3074]: pulp.server.async.tasks:INFO: [c895ee09] Task failed : [c895ee09-f18d-4178-85d6-95d26b0a5bc8]
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488) Task pulp.server.managers.repo.publish.publish[c895ee09-f18d-4178-85d6-95d26b0a5bc8] raised unexpected: OSError(17, 'File exists')
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488) Traceback (most recent call last):
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 367, in trace_task
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     R = retval = fun(*args, **kwargs)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 686, in __call__
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     return super(Task, self).__call__(*args, **kwargs)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 108, in __call__
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     return super(PulpTask, self).__call__(*args, **kwargs)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/celery/app/trace.py", line 622, in __protected_call__
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     return self.run(*args, **kwargs)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1110, in publish
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     result = check_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1207, in check_publish
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     result = _do_publish(repo_obj, dist_id, dist_inst, transfer_repo, conduit, call_config)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/controllers/repository.py", line 1259, in _do_publish
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     publish_report = publish_repo(transfer_repo, conduit, call_config)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/server/async/tasks.py", line 894, in wrap_f
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     return f(*args, **kwargs)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/web.py", line 94, in publish_repo
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     return self._publisher.process_lifecycle()
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 573, in process_lifecycle
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     super(PluginStep, self).process_lifecycle()
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 164, in process_lifecycle
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     step.process()
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 257, in process
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     self._process_block()
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp/plugins/util/publish_step.py", line 304, in _process_block
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     self.process_main()
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 69, in process_main
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     self.write_simple_api(projects)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 100, in write_simple_api
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     PublishMetadataStep._create_project_index(project_name, simple_path, packages)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib/python2.7/site-packages/pulp_python/plugins/distributors/steps.py", line 118, in _create_project_index
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     os.makedirs(project_path)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)   File "/usr/lib64/python2.7/os.py", line 157, in makedirs
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488)     mkdir(name, mode)
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:ERROR: [c895ee09] (3074-35488) OSError: [Errno 17] File exists: '/var/cache/pulp/[email protected]/c895ee09-f18d-4178-85d6-95d26b0a5bc8/python-test/simple/importlib-resources'
    Aug 26 10:34:51 redacted pulp[3074]: celery.app.trace:INFO: [a8098104] Task pulp.server.async.tasks._release_resource[a8098104-c1e6-45a4-bd0b-5850aef21e40] succeeded in 0.00126324000303s: None
    

    reproduction steps pulp upload fail

    The following will fail for any version >=1.1.0

    $ pulp-admin -vvvvv python repo upload --repo-id python-test --file importlib_resources-1.1.0.tar.gz
    ...
    Task Failed
    
    The importer python_importer indicated a failed response when uploading
    python_package unit to repository python-test. Summary: 'expected string or
    buffer'. Details: '{}'
    
    opened by jaraco 22
  • virtualenv can't be established on Python 3.5 since the last update

    virtualenv can't be established on Python 3.5 since the last update

    In GitLab by @aleskva on Mar 1, 2020, 10:23

    See the following log from our Windows tests: https://ci.appveyor.com/project/ladsgroup/pywikibot-g4xqx/build/job/w3ygc2f3winb4nnb

    $ pip install virtualenv
    Collecting virtualenv
      Downloading https://files.pythonhosted.org/packages/c1/61/7506ddd79ef6f09beeefb81c4c55bf395a8ad96b33ff1c6b06e40f8aa101/virtualenv-20.0.7-py2.py3-none-any.whl (8.0MB)
    Collecting filelock<4,>=3.0.0 (from virtualenv)
      Downloading https://files.pythonhosted.org/packages/93/83/71a2ee6158bb9f39a90c0dea1637f81d5eef866e188e1971a1b1ab01a35a/filelock-3.0.12-py3-none-any.whl
    Collecting distlib<1,>=0.3.0 (from virtualenv)
      Downloading https://files.pythonhosted.org/packages/7d/29/694a3a4d7c0e1aef76092e9167fbe372e0f7da055f5dcf4e1313ec21d96a/distlib-0.3.0.zip (571kB)
    Collecting six<2,>=1.9.0 (from virtualenv)
      Downloading https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
    Collecting importlib-metadata<2,>=0.12 (from virtualenv)
      Downloading https://files.pythonhosted.org/packages/8b/03/a00d504808808912751e64ccf414be53c29cad620e3de2421135fcae3025/importlib_metadata-1.5.0-py2.py3-none-any.whl
    Collecting appdirs<2,>=1.4.3 (from virtualenv)
      Downloading https://files.pythonhosted.org/packages/56/eb/810e700ed1349edde4cbdc1b2a21e28cdf115f9faf263f6bbf8447c1abf3/appdirs-1.4.3-py2.py3-none-any.whl
    Collecting importlib-resources<2,>=1.0 (from virtualenv)
      Downloading https://files.pythonhosted.org/packages/a4/23/ffca4a0003519a156b178b3edd98ac9c9de051af76a993d15ed19862ea4d/importlib_resources-1.1.0-py2.py3-none-any.whl
    Collecting zipp>=0.5 (from importlib-metadata<2,>=0.12->virtualenv)
      Downloading https://files.pythonhosted.org/packages/6f/6d/a55f6e81ac213942b9a19cbc05b560c726c3e16f8fb17555f059c17d65f2/zipp-3.0.0-py3-none-any.whl
    Installing collected packages: filelock, distlib, six, zipp, importlib-metadata, appdirs, importlib-resources, virtualenv
      Running setup.py install for distlib
    Successfully installed appdirs-1.4.3 distlib-0.3.0 filelock-3.0.12 importlib-metadata-1.5.0 importlib-resources-1.1.0 six-1.14.0 virtualenv-20.0.7 zipp-3.0.0
    You are using pip version 7.1.2, however version 20.0.2 is available.
    You should consider upgrading via the 'python -m pip install --upgrade pip' command.
    $ virtualenv env
    ImportError: cannot import name 'ContextManager'
    
    opened by jaraco 22
  • Implement a low-level interface for loaders to implement instead of TraversableResources

    Implement a low-level interface for loaders to implement instead of TraversableResources

    In GitLab by @jaraco on Mar 24, 2020, 11:36

    As discussed in #90, this MR attempts to implemented the proposed interface for loaders to supply more basic functionality for loading resources.

    opened by jaraco 21
  • Add support for mechanized deployments as implemented in importlib_metadata. - [merged]

    Add support for mechanized deployments as implemented in importlib_metadata. - [merged]

    In GitLab by @jaraco on Sep 11, 2018, 21:09

    Merges feature/mechanized-releases -> master

    Based on the work in python-devs/importlib_metadata#8, follow the same pattern here.

    I'm marking this a WIP while I work out issues and consider python-devs/importlib_metadata!9.

    housekeeping gitlab merge request 
    opened by jaraco 21
  • zipimporter breaks in python3.5/python3.6 within a zipapp when zipapp call path is not absolute

    zipimporter breaks in python3.5/python3.6 within a zipapp when zipapp call path is not absolute

    In GitLab by @jokerjokerer on Jan 14, 2020, 12:48

    Within a zipapp the loaders archive path will be exactly the path passed on the caller side. So for:

    py -3.5 tasks/virtualenv.pyz
    

    the zipimporter archive path will be tasks/virtualenv.py.

    https://gitlab.com/python-devs/importlib_resources/blob/1.0.2/importlib_resources/_py3.py#L83 sets the resource path always absolute. The logic for the zipimporter seems to care about this discrepancy (https://github.com/python/cpython/blob/v3.5.9/Modules/zipimport.c#L1131) as only paths that are starting with loader.archive will be returned.

    For example for resource tasks/virtualenv.py/a.data:

    • fails absolute format /w/tasks/virtualenv.py/a.data
    • succeeds tasks/virtualenv.py/a.data
    • succeeds a.data

    The current code returns the first format though. Shouldn't we normalize paths relative to the archive at https://gitlab.com/python-devs/importlib_resources/blob/1.0.2/importlib_resources/_py3.py#L93?

    bug help wanted wontfix 
    opened by jaraco 20
  • Fully support pyproject.toml as the single source of truth - [merged]

    Fully support pyproject.toml as the single source of truth - [merged]

    In GitLab by @warsaw on Dec 6, 2017, 11:40

    Merges pep518 -> master

    Make the pyproject.toml file the single source of truth about the project metadata. This should let us flit publish the package when the time comes.

    The trick is that tox still requires the setup.py, but I have added a new update-setup.py script that generates that from the pyproject.toml. Yes, it does evil things but it works. Based on a suggestion by upstream flit.

    housekeeping gitlab merge request 
    opened by jaraco 19
  • TraversalError in MultiplexedPath.joinpath when parent in compound path is missing

    TraversalError in MultiplexedPath.joinpath when parent in compound path is missing

    I have a regression between version 5.7.1 and 5.8.0. I think the cause is this commit.

    I have a reproduction scenario in the zip. To reproduce:

    • unzip repro.zip
    • cd repro
    • pip install -e .
    • python -m test.test --info

    If you change the setup.cfg to use version 5.7.1, no error is raised repro.zip

    opened by MattiasDC 18
  • How to import a file in the program's root ?

    How to import a file in the program's root ?

    Hi,

    I would like to import a file which is in the root of my program.

    I read the docs but didn't find an answer : https://importlib-resources.readthedocs.io/en/latest/using.html I would like to know how to import the "resource3.txt" file of the doc's example.

    Someone else on Stack Overflow has a similar problem. https://stackoverflow.com/q/73559659/9665509

    Could you help me ? I apologize as it's not really an issue.

    question 
    opened by gcailly 1
  • Enable MultiplexedPath.joinpath to return MultiplexedPath for common subdirectories

    Enable MultiplexedPath.joinpath to return MultiplexedPath for common subdirectories

    • When MultiplexedPath.joinpath is passed a subdirectory (tree) that is common to multiple of the parent's paths, joinpath will return a MultiplexedPath. Fixes #265
    opened by jooste 0
  • Possible bug in MultiplexedPath with partially overlapping subdirectory structure

    Possible bug in MultiplexedPath with partially overlapping subdirectory structure

    I ran into the following issue while trying to update my MultiplexedPath example in #264 to the current structure. Let's say you have the following path structure:

    data01
      |_x
    
    data02
      |_x
        |_y
          |_z.txt
    

    If you combine data01 and data02 in aMultiplexedPath, data01/x will make it impossible to find data02/x/y, see the following example:

    from importlib_resources.readers import MultiplexedPath
    
    p = MultiplexedPath('data01', 'data02')
    

    In this case if you do:

    print(p / 'x/y')
    

    this will give data01/x/y instead of data02/x/y, and

    for f in (p / 'x/y').iterdir():
        print(f)
    

    will raise a FileNotFoundError.

    The reason for this is that in Traversable.joinpath(), only the first path segment is used to search for a match:

    names = itertools.chain.from_iterable(
        path.parts for path in map(pathlib.PurePosixPath, descendants)
    )
    target = next(names)
    

    and then, only the first match is returned:

    matches = (
        traversable for traversable in self.iterdir() if traversable.name == target
    )
    try:
        match = next(matches)
    

    The solution to this problem could be related to the suggestion in #264, unless returning MultiplexedPaths isn't the desired behaviour of MultiplexedPath.joinpath().

    help wanted 
    opened by jooste 2
  • Expand usage guide to illustrate namespace package usage.

    Expand usage guide to illustrate namespace package usage.

    Do the importlib.resources documentation (or maybe packaging guides) discuss any of these use cases and best practices?

    I just reviewed the docs, and they are pretty sparse about the purpose of the API. It does link to the "using" guide, and that does seem like it might be an appropriate place to publish that information.

    Originally posted by @jaraco in https://github.com/python/importlib_resources/issues/68#issuecomment-1199476054

    documentation enhancement 
    opened by jaraco 1
  • files(...).joinpath doesn't accept variable number of arguments in Python 3.10+

    files(...).joinpath doesn't accept variable number of arguments in Python 3.10+

    This happens to me when I try to access resources in a native namespace package.

    Tested on:

    • Windows, Python 3.10.4
    • Docker in WSL2, image python:3.10-slim, Python 3.10.5

    The reason seems to be that importlib_resources.files() returns MultiplexedPath from stdlib (importlib.readers) rather than from importlib_resources.readers.

    To recreate:

    1. Install any namespace package, e.g. sphinxcontrib-htmlhelp;

      pip install sphinxcontrib-htmlhelp
      
    2. Use files with joinpath to get the path of a resource.

      from importlib_resources import files
      
      print(root:=files('sphinxcontrib'), 'from', root.__module__)
      print(root.joinpath('htmlhelp', 'templates'))
      

    Expected output (which I get with Pythons 3.7-3.9):

    MultiplexedPath('/usr/local/lib/python3.9/site-packages/sphinxcontrib') from importlib_resources.readers
    /usr/local/lib/python3.9/site-packages/sphinxcontrib/htmlhelp/templates
    

    Bug with Pythons 3.10:

    MultiplexedPath('/usr/local/lib/python3.10/site-packages/sphinxcontrib') from importlib.readers
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
    TypeError: MultiplexedPath.joinpath() takes 2 positional arguments but 3 were given
    

    I also tested with the code in #254 but to no aval.

    opened by Jakub-CZ 4
Releases(v5.10.2)
Owner
Python
Repositories related to the Python Programming language
Python
One destination for all the developer's learning resources.

DevResources One destination for all the developer's learning resources. Find all of your learning resources under one roof and add your own. Live ✨ Y

Gaurav Sharma 33 Oct 21, 2022
What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space

What Do Deep Nets Learn? Class-wise Patterns Revealed in the Input Space Introduction: Environment: Python3.6.5, PyTorch1.5.0 Dataset: CIFAR-10, Image

8 Mar 23, 2022
A type based dependency injection framework for Python 3.9+

Alluka A type based dependency injection framework for Python 3.9+. Installation You can install Alluka from PyPI using the following command in any P

Lucina 16 Dec 15, 2022
Standalone PyQGIS application for executing custom scripts without a QGIS GUI.

PyQGIS Standalone Script Executer Standalone PyQGIS application that is able to run a custom script, in this case Proximity.py without the need of a G

6 Sep 23, 2022
Demo of patching a python context manager

patch-demo-20211203 demo of patching a python context manager poetry install poetry run python -m my_great_app to run the code poetry run pytest to te

Brad Smith 1 Feb 09, 2022
Minimal, super readable string pattern matching for python.

simplematch Minimal, super readable string pattern matching for python. import simplematch simplematch.match("He* {planet}!", "Hello World!") {"p

Thomas Feldmann 147 Dec 01, 2022
Improving Representations via Similarities

embetter warning I like to build in public, but please don't expect anything yet. This is alpha stuff! notes Improving Representations via Similaritie

vincent d warmerdam 229 Jan 08, 2023
Verification of Monty Hall problem by experimental simulation.

Verification of Monty Hall problem by experimental simulation. |中文|English| In the process of learning causal inference, I learned about the Monty Hal

云端听茗 1 Nov 22, 2022
rebalance is a simple Python 3.9+ library for rebalancing investment portfolios

rebalance rebalance is a simple Python 3.9+ library for rebalancing investment portfolios. It supports cash flow rebalancing with contributions and wi

Darik Harter 5 Feb 26, 2022
run-js Goal: The Easiest Way to Run JavaScript in Python

run-js Goal: The Easiest Way to Run JavaScript in Python features Stateless Async JS Functions No Intermediary Files Functional Programming CommonJS a

Daniel J. Dufour 9 Aug 16, 2022
eyes is a Public Opinion Mining System focusing on taiwanese forums such as PTT, Dcard.

eyes is a Public Opinion Mining System focusing on taiwanese forums such as PTT, Dcard. Features 🔥 Article monitor: helps you capture the trend at a

Sean 116 Dec 29, 2022
Python dictionaries with advanced dot notation access

from box import Box movie_box = Box({ "Robin Hood: Men in Tights": { "imdb stars": 6.7, "length": 104 } }) movie_box.Robin_Hood_Men_in_Tights.imdb_s

Chris Griffith 2.1k Dec 28, 2022
Tools for dos (denial-of-service) website / web server

DoS Attack Tools Tools for dos (denial-of-service) website / web server di buat olah NurvySec How to install on debian / ubuntu $ apt update $ apt ins

nurvy 1 Feb 10, 2022
Este script añade la config de s4vitar a bspwm automaticamente!

Se ha testeado este script en ParrotOS, Kali y Ubuntu. Funciona para todos los sistemas operativos basados en Debian. Instalación git clone https://gi

yorkox 201 Dec 30, 2022
Python interface to IEX and IEX cloud APIs

Python interface to IEX Cloud Referral Please subscribe to IEX Cloud using this referral code. Getting Started Install Install from pip pip install py

IEX Cloud 41 Dec 21, 2022
Binary++ is an esoteric programming language based on* binary

Binary++ is an esoteric programming language based on* binary. * It's meant to be based on binary, but you can write Binary++ code using different mea

Supercolbat 3 Feb 18, 2022
HPomb Is Socail Engineering Tool , Used For Bombing , Spoofing and Anonymity Available For Linux And Android(Termux)

HPomb v2020.02 Coming Soon Created By Secanonm HPomb Is Socail Engineering Tool , Used For Bombing , Spoofing and Anonymity Available For Linux And An

Secanonm 10 Jul 25, 2022
Procscan is a quick and dirty python script used to look for potentially dangerous api call patterns in a Procmon PML file.

PROCSCAN Procscan is a quick and dirty python script used to look for potentially dangerous api call patterns in a Procmon PML file. Installation git

Daniel Santos 9 Sep 02, 2022
The official repository of iGEM Paris Bettencourt team's software tools.

iGEM_ParisBettencourt21 The official repository of iGEM Paris Bettencourt team's software tools. Cell counting There are two programs dedicated to the

Abhay Koushik 1 Oct 21, 2021
Ergonomic option parser on top of dataclasses, inspired by structopt.

oppapī Ergonomic option parser on top of dataclasses, inspired by structopt. Usage from typing import Optional from oppapi import from_args, oppapi @

yukinarit 4 Jul 19, 2022