The blog link provided in the readme is broken. Please fix it.

https://deploy-preview-211--quansight-labs.netlify.app/blog/2021/06/pytest-pytorch/

I would suggest to replace it with:

https://labs.quansight.org/blog/2021/06/pytest-pytorch/

:bulb: Note: I am pushing a PR to fix this.

Previously, we used PyTorch nightly to have a second device available in CI. As of torch==1.9 the "meta" device is included in the stable binaries. Thus, there is no need for less safe nightly testing anymore.

In some cases new_cmds == legacy_cmds. This made it more verbose to write and additionally resulted in duplicate tests.

After this PR, if no legacy_cmds is passed to Config, the value of new_cmds is used. Plus, duplicate configs are filtered out.

We don't need to test this for every os / python combination. It should be sufficient to test every os with the minimum python requirement as well as one (linux) with every supported python version.

This should speed up the CI runs and save some resources.

Before we tested the collection by using different outcomes for each test based on the respective parameter. Afterwards we checked the pytest result against that. This has two downsides:

1. It takes more mental effort to not only parse which test will run, but also how the outcome of all run tests is.
2. Since we could only test against the aggregated result for multiple tests, we can't be sure the test is actually right.

With this PR we are actually testing the collection, by parsing the pytest output. Additionally, you can now add this codeblock

# ======================================================================================
# This block is necessary to autogenerate the parametrization for
# tests/test_plugin.py::test_standard_collection.
# It needs to be placed **after** the import of 'instantiate_device_type_tests' and
# **before** its first usage.
# ======================================================================================
try:
    from _spy import Spy

    __spy__ = Spy()
    del Spy
    instantiate_device_type_tests = __spy__(instantiate_device_type_tests)
except ModuleNotFoundError:
    pass
# ======================================================================================

to a test file to automatically test the selection of

• everything in the file,
• every test case, and
• every test case function.

Anything beyond that still needs to be configured manually.

Fixes #16.

Currently we rely on the device identifier comes directly after the test case function name. That is no longer true when using OpInfo's. The name of the instantiated test follows the scheme (template_name)_(op_name)_(device)_(dtype).

This is a complete rewrite of the internal matching logic:

• test case: Test cases are only parametrized by the device. Since every TestCase has a device_type attribute, we can simply strip the device identifier from the instantiated name.
• test case function: Since both template_name and op_name in the pattern above might contain underscores and they are also separated by a single underscore, it is impossible to extract the two parts without further knowledge. To overcome this, we can inspect the source of the function and extract the template_name (which is the function name) directly.

Consider the following test setup:

from torch.testing._internal.common_device_type import (
    instantiate_device_type_tests,
    ops,
)
from torch.testing._internal.common_utils import TestCase
from torch.testing._internal.common_methods_invocations import OpInfo

BazOpInfo = OpInfo("add")


class TestFoo(TestCase):
    @ops([BazOpInfo])
    def test_bar(self, device, dtype, op):
        pass


instantiate_device_type_tests(TestFoo, globals(), only_for="cpu")

Running pytest test_foo.py --collect-only on that results in:

<PyTorchTestCase TestFooCPU>
  <PyTorchTestCaseFunction test_bar_add_cpu_float32>
  <PyTorchTestCaseFunction test_bar_add_cpu_float64>
<PyTorchTestCase TestFooMETA>
  <PyTorchTestCaseFunction test_bar_add_meta_float32>
  <PyTorchTestCaseFunction test_bar_add_meta_float64>

Naming schemes:

• test cases: (template_name)(device)
• test case functions: (template_name)_(op_name)_(device)_(dtype)

After #12 test case functions now require the device to follow right after the template name. Thus, it is no longer possible to select an individual test by name pytest test_foo.py::TestFoo::test_bar

Instead of instantiating the tests for all devices and use the @onlyCPU decorator everywhere, used the only_for keyword when instantiating. With that the meta tests are not generated in the first place and do not need to be considered for the number of skipped tests.

Right now we have torch as dependency:

https://github.com/Quansight/pytest-pytorch/blob/bd98f6b23214460605e4b8c0ee2bd4956e846291/setup.cfg#L32-L34

If you set up the development environment to work on PyTorch, you would probably not have it installed already. Thus installing pytest-pytorch would also install torch which is usually not desired.

Currently we match the test case (function) names based on this:

https://github.com/Quansight/pytest-pytorch/blob/ff8f2d86906486a2d437b2617ef9973394f5e216/pytest_pytorch/plugin.py#L9-L10

This works well until you have a setup similar to this:

class TestFoo(TestCase):
    pass

class TestFooBar(TestCase):
    pass

If you run pytest test_foo.py::TestFoo on this, both test cases are collected instead of just TestFoo.

The CI tests are failing, because we need torch>=1.9, but it is only available through the nightlies. Unfortunately, tox-ltt is not able to handle the nightly channel yet.