Klara is a static analysis tools to automatic generate test case, based on SMT (z3) solver, with a powerful ast level inference system.

Overview

Klara

Klara is a static analysis tools to automatic generate test case, based on SMT (z3) solver, with a powerful ast level inference system. Klara will take python file as input and generate corresponding test file in pytest format, that attempt to cover all return values. For example, following function in file test.py

def triangle(x: int, y: int, z: int) -> str:
    if x == y == z:
        return "Equilateral triangle"
    elif x == y or y == z or x == z:
        return "Isosceles triangle"
    else:
        return "Scalene triangle"

will generate

import test
def test_triangle_0():
    assert test.triangle(0, 0, 0) == 'Equilateral triangle'
    assert test.triangle(0, 0, 1) == 'Isosceles triangle'
    assert test.triangle(2, 0, 1) == 'Scalene triangle'

See the Klara's documentation at https://klara-py.readthedocs.io

Installing

Klara can be installed via pip tool by using:

pip install klara

Usage

We can invoke klara on any python source file, and it will generate a corresponding pytest test file.

$ cat source.py
def foo(x: int, y: int, z: str):
    if x + y > 2:
        return x + y + 12
    elif x < y:
        return x + y
    elif (z + "me") == "some":
        return z + "thing"
    else:
        return x - y

$ klara source.py
$ cat test_source.py
import contract_test


def test_foo_0():
    assert contract_test.foo(0, 3, \'\') == 15
    assert contract_test.foo(0, 1, \'\') == 1
    assert contract_test.foo(0, 0, \'so\') == \'sothing\'
    assert contract_test.foo(0, 0, \'\') == 0

Consult the quick start manual for more examples and guidance. To use it as a static analysis library, go to Inference.

Why Klara?

Klara works on ast level and it doesn't execute user code in any way, which is a very important difference compared to similar tool like Crosshair and Pynguin that utilize concolic symbolic execution that required user code execution that might cause unwanted side effects. Klara work on ast level, combine with data flow analysis that utilize Control Flow Graph(CFG), Static Single Assignment(SSA), use-def chain, etc... to build a powerful python inference system that leverages Z3-solver for constraints solving and path feasibility check. Because of this, Klara is able to operate on both python2/3 source code with the help of typed_ast. To specify the source code is in python 2, pass in -py 2 argument. It's python 3 by default.

Klara can also be used as a static analysis tool, allow user to define custom rule to identify programming bugs, error or enforcing coding standard. With SMT solver support, analysis will be more accurate and greatly reduce false-positive case. For example

4: if v1 < 3: z = 1 else: z = 2 else: z = 3 s = z """) with klara.MANAGER.initialize_z3_var_from_func(tree.body[0]): print(list(tree.body[0].body[-1].value.infer())) ">
import klara
tree = klara.parse("""
    def foo(v1: int):
        if v1 > 4:
            if v1 < 3:
                z = 1
            else:
                z = 2
        else:
            z = 3
        s = z
""")
with klara.MANAGER.initialize_z3_var_from_func(tree.body[0]):
    print(list(tree.body[0].body[-1].value.infer()))

Will print out:

[2, 3]

Because z = 1 is not possible due to v1 > 4 and v1 < 3 is unsatisfiable

The inference system architecture and api is largely inspired by Astroid, a static inference library used by Pylint.

Klara utilize the inference system to generate test case, in other words, it generate test case for all possible return values of the function, instead of generate test case for all control path of the function.

To illustrate the point, consider the function below, with divide by zero vulnerabilities at line 3

def foo(v1: int, v2: float):
    if v1 > 10000:
        s = v1 / 0  # unused statement
    if v1 > v2:
        s = v1
    else:
        s = v2
    return s

Klara will generate test inputs below

import contract_test
def test_foo_0():
    assert contract_test.foo(0, -1.0) == 0
    assert contract_test.foo(0, 0.0) == 0.0

It doesn't generate input v1 > 10000, so the test case would not be able to find out the exceptions. This is because the s at line 3 is unused in the return value.

If we modify the second if statement to elif, which we'll be able to return the [s]{.title-ref} at line 3, klara will generate test inputs that cover v1 > 10000 case.

This is an important distinction with other automatic test case generation available now, because by only generate test case for return values, we can generate a minimal test case, and it's easier to customize how do Klara cover the function.

For example, say we are composing a complex system

    def main(number: int, cm: int, dc: int, wn: int):
        mc = 0
        if wn > 2:
            if number > 2 and number > 2 or number > 2:
                if number > 0:
                    if wn > 2 or wn > 2:
                        mc = 2
                    else:
                        mc = 5
                else:
                    mc = 100
        else:
            mc = 1
        nnn = number * cm
        if cm <= 4:
            num_incr = 4
        else:
            num_incr = cm
        n_num_incr = nnn / num_incr
        nnn_left = dc * num_incr * (n_num_incr / 2 + n_num_incr % 2)
        nnn_right = nnn - nnn_left
        is_flag = nnn_right
        if is_flag:
            cell = Component(nnn_right, options=[mc])
        else:
            cell = Component(nnn_right)
        return cell

It isn't immediately clear to us how many possible return values there are. But we can utilize Klara to generate inputs instantly, below is the generated test

import contract_test
def test_main_0():
    assert contract_test.main(2, 4, 1, 3) is not None
    assert contract_test.main(2, 4, -1, 6) is not None
    assert contract_test.main(2, 4, 1, 4) is not None
    assert contract_test.main(-2, 4, 3, 4) is not None
    assert contract_test.main(-1, -1, -1, 2) is not None
    assert contract_test.main(0, 0, 0, 3) is not None
    assert contract_test.main(0, 0, 0, 6) is not None
    assert contract_test.main(0, 0, 0, 4) is not None
    assert contract_test.main(-2, 0, 0, 4) is not None
    assert contract_test.main(0, 0, 0, 0) is not None

Above generated 10 total results, which is product of nnn_right which have 2 possibilities and mc which have 5 possibilities.

Suppose that 10 tests input is too much, and we have determine that the options argument to Component is redundant to test, we can use Klara's custom plugin to selectively determine which part to ignore in test generation. Go to customize coverage strategy for more information.

After we have setup the plugin, Klara will generate following test

import contract_test
def test_main_0():
    assert contract_test.main(1, 3, 0, 0) == 3.0
    assert contract_test.main(0, 0, 0, 0) == 0.0

Which is only 2 combinations of nnn_right

Because Klara can't dynamically execute the code, it will provide extension to specify how to infer specific ast node or user defined type to make Klara 'smarter'. It's described in extending, extending user type and customize coverage strategy.

Contributing

We use Poetry to manage dependencies. After poetry is installed, run:

$ poetry shell
$ poetry install

To run the test case, do:

$ poetry run pytest test

Acknowledgements

  • The architecture of the inference system is largely inspired by Astroid.
  • Special thanks to Dr. Poh for guiding the early stages of the project.

License

This project is licensed under the terms of the GNU Lesser General Public License.

Comments
  • Multiple errors and confusions in the docs

    Multiple errors and confusions in the docs

    From formal, computer science point of view, docs included in the project contain multiple confusing, or just incorrect, statements.

    Meta-issue trying to pinpoint the issues to help a novice reader, who may get misconceptions after reading these docs.

    opened by pfalcon 6
  • Many Errors

    Many Errors

    1. When I run klara on https://github.com/erezsh/runtype/blob/master/runtype/dataclass.py, I get AttributeError: 'JoinedStr' object has no attribute 'statement' :
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 354, in rename
        super(ParentScopeBlock, self).rename()
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 191, in rename
        blk.enumerate()
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 156, in enumerate
        AttributeEnumerator.enumerate(ast_stmt, False, False)
      File "c:\python38\lib\site-packages\klara\core\ssa.py", line 115, in enumerate
        var.convert_to_ssa()
      File "c:\python38\lib\site-packages\klara\core\node_classes.py", line 399, in convert_to_ssa
        stmt = field.statement()
    AttributeError: 'JoinedStr' object has no attribute 'statement'
    
    1. When I run it on https://github.com/lark-parser/lark/blob/master/lark/utils.py, I get KeyError: 'is not'
    ...
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1146, in infer_compare
        for result in calc_compare(comp, self.ops, context):
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1195, in calc_compare
        methods = _comp_op_methods(left, comp, op, context)
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1161, in _comp_op_methods
        method_name=COMP_OP_DUNDER_METHOD[op],
    KeyError: 'is not'
    
    1. When I run it on https://github.com/lark-parser/lark/blob/master/lark/tree.py I get AttributeError: type object 'Del' has no attribute 'targets'
    ...
        return [self._visit_generic(child) for child in node]
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 36, in _visit_generic
        return self._visit(node)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 24, in _visit
        returned = self._visit_generic(value)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 36, in _visit_generic
        return self._visit(node)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 23, in _visit
        value = getattr(node, name)
    AttributeError: type object 'Del' has no attribute 'targets'
    

    I'm sure I can keep going, but let's stop here for now :)

    opened by erezsh 5
  • AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'

    AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'

    Code:

    async def f():
        pass
    

    Error:

    ❯ klara tmp.py
    loaded extension: {'typeshed_stub.py', 'builtin_inference.py', '__init__.py', 'infer_z3.py', '99_math_z3.py'}
    
    using configuration value: 
    {   'config_file': None,
        'eq_neq': False,
        'input_test_file': 'tmp.py',
        'max_inference_value': None,
        'py_version': 3,
        'stubs': [],
        'type_inference': True,
        'typeshed_select': []}
    
    Traceback (most recent call last):
      File "/home/gram/.local/bin/klara", line 8, in <module>
        sys.exit(main())
      File "/home/gram/.local/lib/python3.9/site-packages/klara/contract/__main__.py", line 50, in main
        output_test = run(input_file.read_text(), input_file.stem)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/contract/__main__.py", line 36, in run
        cfg = MANAGER.build_cfg(tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/klara_z3/cov_manager.py", line 64, in build_cfg
        c = cfg.Cfg(as_tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 509, in __init__
        self.root, _, _ = self.parse(as_tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 629, in parse
        head = self.build(basic_block, head, all_tail_list, func_tail_list)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 535, in build
        tail_list, func_tail = meth(block)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 589, in build_module
        head_returned, tail_list, _ = self.parse(self.as_tree.body)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 627, in parse
        for basic_block in basic_block_parser.get_basic_block():
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 840, in get_basic_block
        basic_block_list = self.visit(node)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 821, in visit
        return visitor(ast_node)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 824, in generic_visit
        self._append_cache(ast_node.statement())
    AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'
    
    
    opened by orsinium 3
  • Improve this by defining the identity table as class variable

    Improve this by defining the identity table as class variable

    https://github.com/usagitoneko97/python-ast/blob/78330cbc3d4601160175f5073a6630a157fce6db/A3.LVN/lvn.py#L30

    Unlike the above, a class variable is instantiated only once when the class is created.

    The following shows how to define it as class variable.

    class Lvn:
        identity_expr = {(None, '*', 2):(None, '+', None), (None, '+', 0):('0', '+', None), .....}
    
        # Accessing the table is a bit tricky though since you need this class' handle to access it.
        # To get the current class, use type(self) as shown below.
        def get_alternate_id(self, id):
            return type(self).identity_expr.get(id)
    
    opened by chaosAD 2
  • Broken links to images, etc. in the docs

    Broken links to images, etc. in the docs

    It seems that https://github.com/usagitoneko97/python-static-code-analysis/commit/1595f838b49f364d456a10673d42914f1d859118 broke various links around. E.g. in https://github.com/usagitoneko97/python-static-code-analysis/blob/master/lvn_optimization/readme.md , links to SVG images are broken . E.g. go to https://github.com/usagitoneko97/python-static-code-analysis/tree/master/lvn_optimization#113-algorithm-in-details, "IvnThird" pseudo-link is rendered instead of an image.

    opened by pfalcon 1
  • Replaced with more aesthetic diagrams and reworded the Dominance:Introduction section

    Replaced with more aesthetic diagrams and reworded the Dominance:Introduction section

    I have replaced your old diagrams with aesthetically appealing diagrams and reworded your text. Please have a look.

    Please use my SVG templates to draw/redraw your diagrams.

    Please don't convert images to PNG files. Now GitHub support SVG directly. See my example.

    Please don't use full URL to your images like https://github.com/usagitoneko97/python-ast/blob/master/A4.Cfg/resources/cfg_ssa_intro.svg.png You should use just resources/cfg_ssa_intro.svg.png instead.

    opened by chaosAD 0
  • Edited readme for Python Implementation version 2

    Edited readme for Python Implementation version 2

    Please have a look at my edited readme.md file. I think you have to rewrite section 1.2.2 Converting Back To SSA. It is unclear and also I think the list given there is incorrect.

    opened by chaosAD 0
  • Api documentation

    Api documentation

    Hello, it seems that you api documentation page is empty. Maybe a misconfiguration of the autodoc Sphinx extension.

    As a temporary solution, the docs are available here: https://pydocbrowser.github.io/klara/latest/index.html

    opened by tristanlatr 0
  • Support for imports?

    Support for imports?

    Hello,

    First good job at writing this library. It looks like it has powerful capabilities.

    I was wondering if this library still maintained? Do you planned support for inferring imports anytime soon ?

    I’ve myself implemented a astroid-alike (intra procedural) inference engine: https://github.com/tristanlatr/astuce and it supports imports, except wildcard imports. But klara seems much more intelligent.

    Tell me what you think. Thanks

    opened by tristanlatr 0
  • BREAKING CHANGE: remove py2 support

    BREAKING CHANGE: remove py2 support

    I've removed typed-ast package at there are problems with this package in apple-silicon. Also, typed ast recommends using the builtin ast module from python 3.8 and up.

    I could not understand why the uts are failing, I would like to get help to fix those.

    opened by jochman 3
Releases(0.6.3)
  • 0.6.3(Sep 19, 2021)

    v0.6.3 Release Notes (09/19/21)

    • Fixed conditions in loop causing conflicting conditions propagation (#7)
    • fixed Del and Delete shared the same node, and caused fields error (#7)
    • implemented identity (is, is not) comparison for const and instance (#7)
    • added AsyncFunctionDef, Await, AsyncFor, AsyncWith ast support
    • implemented repr, ascii builtin call, and JoinedStr, FormattedValue inference (#7, #8)

    Fixed #7, #8

    Source code(tar.gz)
    Source code(zip)
Owner
Ho Guo Xian
I like automation.
Ho Guo Xian
Lightweight, scriptable browser as a service with an HTTP API

Splash - A javascript rendering service Splash is a javascript rendering service with an HTTP API. It's a lightweight browser with an HTTP API, implem

Scrapinghub 3.8k Jan 03, 2023
pytest plugin that let you automate actions and assertions with test metrics reporting executing plain YAML files

pytest-play pytest-play is a codeless, generic, pluggable and extensible automation tool, not necessarily test automation only, based on the fantastic

pytest-dev 67 Dec 01, 2022
Data-Driven Tests for Python Unittest

DDT (Data-Driven Tests) allows you to multiply one test case by running it with different test data, and make it appear as multiple test cases. Instal

424 Nov 28, 2022
1st Solution to QQ Browser 2021 AIAC Track 2

1st Solution to QQ Browser 2021 AIAC Track 2 This repository is the winning solution to QQ Browser 2021 AI Algorithm Competition Track 2 Automated Hyp

DAIR Lab 24 Sep 10, 2022
Spam the buzzer and upgrade automatically - Selenium

CookieClicker Usage: Let's check your chrome navigator version : Consequently, you have to : download the right chromedriver in the follow link : http

Iliam Amara 1 Nov 22, 2021
reCaptchaBypasser For Bypass Any reCaptcha For Selenium Python

reCaptchaBypasser ' Usage : from selenium import webdriver from reCaptchaBypasser import reCaptchaScraper import time driver = webdriver.chrome(execu

Dr.Linux 8 Dec 17, 2022
Python Projects - Few Python projects with Testing using Pytest

Python_Projects Few Python projects : Fast_API_Docker_PyTest- Just a simple auto

Tal Mogendorff 1 Jan 22, 2022
Mypy static type checker plugin for Pytest

pytest-mypy Mypy static type checker plugin for pytest Features Runs the mypy static type checker on your source files as part of your pytest test run

Dan Bader 218 Jan 03, 2023
Divide full port scan results and use it for targeted Nmap runs

Divide Et Impera And Scan (and also merge the scan results) DivideAndScan is used to efficiently automate port scanning routine by splitting it into 3

snovvcrash 226 Dec 30, 2022
Codeforces Test Parser for C/C++ & Python on Windows

Codeforces Test Parser for C/C++ & Python on Windows Installation Run pip instal

Minh Vu 2 Jan 05, 2022
Hypothesis is a powerful, flexible, and easy to use library for property-based testing.

Hypothesis Hypothesis is a family of testing libraries which let you write tests parametrized by a source of examples. A Hypothesis implementation the

Hypothesis 6.4k Jan 05, 2023
Hamcrest matchers for Python

PyHamcrest Introduction PyHamcrest is a framework for writing matcher objects, allowing you to declaratively define "match" rules. There are a number

Hamcrest 684 Dec 29, 2022
Subprocesses for Humans 2.0.

Delegator.py — Subprocesses for Humans 2.0 Delegator.py is a simple library for dealing with subprocesses, inspired by both envoy and pexpect (in fact

Amit Tripathi 1.6k Jan 04, 2023
ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries.

ApiPy was created for api testing with Python pytest framework which has also requests, assertpy and pytest-html-reporter libraries. With this f

Mustafa 1 Jul 11, 2022
Compiles python selenium script to be a Window's executable

Problem Statement Setting up a Python project can be frustrating for non-developers. From downloading the right version of python, setting up virtual

Jerry Ng 8 Jan 09, 2023
Descriptor Vector Exchange

Descriptor Vector Exchange This repo provides code for learning dense landmarks without supervision. Our approach is described in the ICCV 2019 paper

James Thewlis 74 Nov 29, 2022
pytest_pyramid provides basic fixtures for testing pyramid applications with pytest test suite

pytest_pyramid pytest_pyramid provides basic fixtures for testing pyramid applications with pytest test suite. By default, pytest_pyramid will create

Grzegorz Śliwiński 12 Dec 04, 2022
Pymox - open source mock object framework for Python

Pymox is an open source mock object framework for Python. First Steps Installation Tutorial Documentation http://pymox.readthedocs.io/en/latest/index.

Ivan Rocha 7 Feb 02, 2022
Show coverage stats online via coveralls.io

Coveralls for Python Test Status: Version Info: Compatibility: Misc: coveralls.io is a service for publishing your coverage stats online. This package

Kevin James 499 Dec 28, 2022
Make Selenium work on Github Actions

Make Selenium work on Github Actions Scraping with BeautifulSoup on GitHub Actions is easy-peasy. But what about Selenium?? After you jump through som

Jonathan Soma 33 Dec 27, 2022