Issue

• Closes https://github.com/slidoapp/dbt-coverage/issues/24:

"Please add options in the CLI to include and exclude models to filter out the checks in some of the models or a path."

Summary

I added the ability to perform compute commands on only a subset of tables by adding a --model-path-filter option. This means that a subset of models can be selected based upon their original_file_path value (taken from the manifest.json artifact).

This functionality means that dbt-coverage can now be used in monolithic dbt projects which contain sub-projects owned by different teams. Before adding model selection functionality, using dbt-coverage would not have been useful/advisable in such a structure because another, unrelated team may decrease the overall coverage, which can then block PR merging (should dbt-coverage have been integrated as part of a CI/CD pipeline for example).

• See example of added functionality from updated README.md below:

$ cd jaffle_shop
$ dbt run  # Materialize models
$ dbt docs generate  # Generate catalog.json and manifest.json
$ dbt-coverage compute doc --cov-report coverage-doc.json --model-path-filter models/staging/  # Compute doc coverage for a subset of tables, print it and write it to coverage-doc.json file

Coverage report
======================================================
jaffle_shop.stg_customers              0/3       0.0%
jaffle_shop.stg_orders                 0/4       0.0%
jaffle_shop.stg_payments               0/4       0.0%
======================================================
Total                                  0/11      0.0%

$ dbt-coverage compute doc --cov-report coverage-doc.json --model-path-filter models/orders.sql --model-path-filter models/staging/  # Compute doc coverage for a subset of tables, print it and write it to coverage-doc.json file

Coverage report
======================================================
jaffle_shop.orders                     0/9       0.0%
jaffle_shop.stg_customers              0/3       0.0%
jaffle_shop.stg_orders                 0/4       0.0%
jaffle_shop.stg_payments               0/4       0.0%
======================================================
Total                                  0/20      0.0%

Note: this is a relatively 'rough' solution and there are likely many improvements that could be made to my code / far more elegant implementations that would achieve the same functionality. Please, feel free to suggest changes!

Testing

• I have tested these changes on dbt's jaffle_shop 'testing project' and have not encountered issues so far.

Goal

Give more flexibility to user who have their manifest.json & catalog.json files not stored in /target, which is mainly the case for user using dbt cloud (need to retrieve the file by api call), adding a new parameter to use a custom path allow user not to create an unnecessary targetdir in their cicd pipeline

Description

• Add run_artifacts_dir : Path parameter to load_manifest and load_catalog
• Add optional run_artifacts_dir parameter to cli

From my understanding we can only save .json files as a result of dbt-coverage compute.

It would be nice if we could pick other formats, for example .csv That way we can put those files as seeds and write to the db.

• Support dbt 1.0 and drop support for dbt<1
• Warn when using with unsupported manifest.json version

@mrshu, I acknowledged the changes in the new v4 manifest.json format introduced by dbt 1.0 and by that the support for v3 dropped. Since the difference between the formats is only tiny, do you think it would make sense to support also v3 manifest.json (used in dbt<1) by having 2 methods for parsing the manifest, one for v3 and one for v4?

Closes #26.

Resolves #46

What has changed?

Introduces a new argument called cov_format that accepts string (default) or markdown for the compute doc command.

Outputs the results to the terminal in either the default string format, or in a markdown friendly format

Example

dbt-coverage compute doc --cov-format markdown

results in:

Planning to submit a PR, just wanted to tee this up first to document the need

Steps to reproduce:

pip install dbt-coverage==0.2.2
git clone https://github.com/dbt-labs/jaffle_shop
git checkout b0b77aa
dbt-coverage compute doc

Line of code throwing the error: https://github.com/slidoapp/dbt-coverage/blob/9f97e6c8fc24e83babc51c2b0584afc88e45308c/dbt_coverage/init.py#L507

Error message:

INFO:root:Loading catalog and manifest files from project dir: .
Traceback (most recent call last):
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/bin/dbt-coverage", line 8, in <module>
    sys.exit(app())
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/typer/main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 660, in compute
    return do_compute(project_dir, cov_report, cov_type, cov_fail_under, cov_fail_compare)
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 616, in do_compute
    catalog = load_files(project_dir)
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 533, in load_files
    catalog = load_catalog(project_dir)
  File "/Users/brentbrewington/.pyenv/versions/3.9.0/lib/python3.9/site-packages/dbt_coverage/__init__.py", line 507, in load_catalog
    with open(project_dir / 'target/catalog.json') as f:
FileNotFoundError: [Errno 2] No such file or directory: 'target/catalog.json'

dbt has released version 1.2 and bumped the manifest version to v6 as part of this release.

They have allowed manifest v6 to defer to v4 and v5 of the manifest which means that changes are not breaking. They are related to adding information about configs and metrics. Neither of these are currently consumed from the manifest by dbt coverage

I believe this means dbt-coverage can add manifest v6 support without fundamental changes to the project

It seems that seeds are not considered while reading manifest.json. As a result, the following error pops while running dbt-coverage compute doc

(venv_dbt) ➜  myCodeFolder git:(master) dbt-coverage compute doc                                
INFO:root:Loading catalog and manifest files from project dir: .
INFO:root:Successfully loaded 200 tables from catalog
Traceback (most recent call last):
  File "/Users/victor/myCodeFolder/venv_dbt/bin/dbt-coverage", line 8, in <module>
    sys.exit(app())
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/typer/main.py", line 214, in __call__
    return get_command(self)(*args, **kwargs)
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1668, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/click/core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/typer/main.py", line 500, in wrapper
    return callback(**use_params)  # type: ignore
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 604, in compute
    do_compute(project_dir, cov_report, cov_type, cov_fail_under, cov_fail_compare)
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 562, in do_compute
    catalog = load_files(project_dir)
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 486, in load_files
    manifest = load_manifest(project_dir)
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 477, in load_manifest
    manifest = Manifest.from_nodes(manifest_nodes)
  File "/Users/victor/myCodeFolder/venv_dbt/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 121, in from_nodes
    table_name = id_to_table_name[table_id]
KeyError: 'seed.dbt_project.browsers'

Versions:

dbt-core==0.21.0
dbt-coverage==0.1.7

Python 3.8

# dbt build
...
Done. PASS=23 WARN=0 ERROR=1 SKIP=1 TOTAL=25
# dbt docs generate
Running with dbt=0.21.1-rc1
...
$ pip install dbt-coverage
Successfully installed dbt-coverage-0.1.7 typer-0.4.0
$ dbt-coverage compute doc 
INFO:root:Loading catalog and manifest files from project dir: .
INFO:root:Successfully loaded 17 tables from catalog

    catalog = load_files(project_dir)
  File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 486, in load_files
    manifest = load_manifest(project_dir)
  File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 477, in load_manifest
    manifest = Manifest.from_nodes(manifest_nodes)
  File "/root/.local/share/virtualenvs/data-pipelines-kwiw1yve/lib/python3.8/site-packages/dbt_coverage/__init__.py", line 122, in from_nodes
    column_name = node['column_name'] or node['test_metadata']['kwargs']['column_name'] \
KeyError: 'column_name'

This fixes two issues:

1. Some systems (e.g. Snowflake) uppercase all table and column names. In accordance with the dbt-docs official source code, we lowercase all table and column names before matching them.
2. There can be multiple tables with the same name but in different schemas. We therefore use schema_name.table_name to distinguish tables.

What was done

• Copy contents of dbt_docs_coverage.py verbatim to __init__.py.
• Add README with basic documentation.
• Create Python package using poetry.
• Register dbt_coverage.__init__:app as a CLI script with name dbt-coverage.

Review objectives

• Please review README.
• Please review the poetry files.
• Ignore __init__.py, it was copied verbatim.

Hi team,

I am wondering whether it is already possible for me to display the coverage doc/test result as a percentage on gitlab badges. Please see similar example for python unit test below

Thank you

Originally I had used is not None as you say, however, this doesn't work due to a known issue with typer: https://github.com/tiangolo/typer/issues/410 whereby [an] "Empty tuple [is] returned for Option instead of expected None when no values passed". As such, I have to use len() since until the issue is resolved model_path_filter will never be None.

~~Since you asked the question it's obvious that I could make the reason for my choice clearer, WDYT to the below suggestion?~~

Originally posted by @followingell in https://github.com/slidoapp/dbt-coverage/pull/45#discussion_r1012694530

The script does not currently recognize tests applied to a source that is set-up using an identifier rather than just the raw as being associated with the source.

I think, but am not certain, that this might be related to #30

To be able to retrieve dbt-coverage output in gitlab, a cobertura format is needed.

See: https://docs.gitlab.com/ee/ci/yaml/artifacts_reports.html#artifactsreportscoverage_report

Could you add a command line argument to choose output format ? (json or xml-cobertura)

Our project is relying on tests defined in dbt_utils - is there a way of adding support for these as well?

https://github.com/dbt-labs/dbt-utils#unique_combination_of_columns-source