Klara is a static analysis tools to automatic generate test case, based on SMT (z3) solver, with a powerful ast level inference system.

Overview

Klara

Klara is a static analysis tools to automatic generate test case, based on SMT (z3) solver, with a powerful ast level inference system. Klara will take python file as input and generate corresponding test file in pytest format, that attempt to cover all return values. For example, following function in file test.py

def triangle(x: int, y: int, z: int) -> str:
    if x == y == z:
        return "Equilateral triangle"
    elif x == y or y == z or x == z:
        return "Isosceles triangle"
    else:
        return "Scalene triangle"

will generate

import test
def test_triangle_0():
    assert test.triangle(0, 0, 0) == 'Equilateral triangle'
    assert test.triangle(0, 0, 1) == 'Isosceles triangle'
    assert test.triangle(2, 0, 1) == 'Scalene triangle'

See the Klara's documentation at https://klara-py.readthedocs.io

Installing

Klara can be installed via pip tool by using:

pip install klara

Usage

We can invoke klara on any python source file, and it will generate a corresponding pytest test file.

$ cat source.py
def foo(x: int, y: int, z: str):
    if x + y > 2:
        return x + y + 12
    elif x < y:
        return x + y
    elif (z + "me") == "some":
        return z + "thing"
    else:
        return x - y

$ klara source.py
$ cat test_source.py
import contract_test


def test_foo_0():
    assert contract_test.foo(0, 3, \'\') == 15
    assert contract_test.foo(0, 1, \'\') == 1
    assert contract_test.foo(0, 0, \'so\') == \'sothing\'
    assert contract_test.foo(0, 0, \'\') == 0

Consult the quick start manual for more examples and guidance. To use it as a static analysis library, go to Inference.

Why Klara?

Klara works on ast level and it doesn't execute user code in any way, which is a very important difference compared to similar tool like Crosshair and Pynguin that utilize concolic symbolic execution that required user code execution that might cause unwanted side effects. Klara work on ast level, combine with data flow analysis that utilize Control Flow Graph(CFG), Static Single Assignment(SSA), use-def chain, etc... to build a powerful python inference system that leverages Z3-solver for constraints solving and path feasibility check. Because of this, Klara is able to operate on both python2/3 source code with the help of typed_ast. To specify the source code is in python 2, pass in -py 2 argument. It's python 3 by default.

Klara can also be used as a static analysis tool, allow user to define custom rule to identify programming bugs, error or enforcing coding standard. With SMT solver support, analysis will be more accurate and greatly reduce false-positive case. For example

4: if v1 < 3: z = 1 else: z = 2 else: z = 3 s = z """) with klara.MANAGER.initialize_z3_var_from_func(tree.body[0]): print(list(tree.body[0].body[-1].value.infer())) ">
import klara
tree = klara.parse("""
    def foo(v1: int):
        if v1 > 4:
            if v1 < 3:
                z = 1
            else:
                z = 2
        else:
            z = 3
        s = z
""")
with klara.MANAGER.initialize_z3_var_from_func(tree.body[0]):
    print(list(tree.body[0].body[-1].value.infer()))

Will print out:

[2, 3]

Because z = 1 is not possible due to v1 > 4 and v1 < 3 is unsatisfiable

The inference system architecture and api is largely inspired by Astroid, a static inference library used by Pylint.

Klara utilize the inference system to generate test case, in other words, it generate test case for all possible return values of the function, instead of generate test case for all control path of the function.

To illustrate the point, consider the function below, with divide by zero vulnerabilities at line 3

def foo(v1: int, v2: float):
    if v1 > 10000:
        s = v1 / 0  # unused statement
    if v1 > v2:
        s = v1
    else:
        s = v2
    return s

Klara will generate test inputs below

import contract_test
def test_foo_0():
    assert contract_test.foo(0, -1.0) == 0
    assert contract_test.foo(0, 0.0) == 0.0

It doesn't generate input v1 > 10000, so the test case would not be able to find out the exceptions. This is because the s at line 3 is unused in the return value.

If we modify the second if statement to elif, which we'll be able to return the [s]{.title-ref} at line 3, klara will generate test inputs that cover v1 > 10000 case.

This is an important distinction with other automatic test case generation available now, because by only generate test case for return values, we can generate a minimal test case, and it's easier to customize how do Klara cover the function.

For example, say we are composing a complex system

    def main(number: int, cm: int, dc: int, wn: int):
        mc = 0
        if wn > 2:
            if number > 2 and number > 2 or number > 2:
                if number > 0:
                    if wn > 2 or wn > 2:
                        mc = 2
                    else:
                        mc = 5
                else:
                    mc = 100
        else:
            mc = 1
        nnn = number * cm
        if cm <= 4:
            num_incr = 4
        else:
            num_incr = cm
        n_num_incr = nnn / num_incr
        nnn_left = dc * num_incr * (n_num_incr / 2 + n_num_incr % 2)
        nnn_right = nnn - nnn_left
        is_flag = nnn_right
        if is_flag:
            cell = Component(nnn_right, options=[mc])
        else:
            cell = Component(nnn_right)
        return cell

It isn't immediately clear to us how many possible return values there are. But we can utilize Klara to generate inputs instantly, below is the generated test

import contract_test
def test_main_0():
    assert contract_test.main(2, 4, 1, 3) is not None
    assert contract_test.main(2, 4, -1, 6) is not None
    assert contract_test.main(2, 4, 1, 4) is not None
    assert contract_test.main(-2, 4, 3, 4) is not None
    assert contract_test.main(-1, -1, -1, 2) is not None
    assert contract_test.main(0, 0, 0, 3) is not None
    assert contract_test.main(0, 0, 0, 6) is not None
    assert contract_test.main(0, 0, 0, 4) is not None
    assert contract_test.main(-2, 0, 0, 4) is not None
    assert contract_test.main(0, 0, 0, 0) is not None

Above generated 10 total results, which is product of nnn_right which have 2 possibilities and mc which have 5 possibilities.

Suppose that 10 tests input is too much, and we have determine that the options argument to Component is redundant to test, we can use Klara's custom plugin to selectively determine which part to ignore in test generation. Go to customize coverage strategy for more information.

After we have setup the plugin, Klara will generate following test

import contract_test
def test_main_0():
    assert contract_test.main(1, 3, 0, 0) == 3.0
    assert contract_test.main(0, 0, 0, 0) == 0.0

Which is only 2 combinations of nnn_right

Because Klara can't dynamically execute the code, it will provide extension to specify how to infer specific ast node or user defined type to make Klara 'smarter'. It's described in extending, extending user type and customize coverage strategy.

Contributing

We use Poetry to manage dependencies. After poetry is installed, run:

$ poetry shell
$ poetry install

To run the test case, do:

$ poetry run pytest test

Acknowledgements

  • The architecture of the inference system is largely inspired by Astroid.
  • Special thanks to Dr. Poh for guiding the early stages of the project.

License

This project is licensed under the terms of the GNU Lesser General Public License.

Comments
  • Multiple errors and confusions in the docs

    Multiple errors and confusions in the docs

    From formal, computer science point of view, docs included in the project contain multiple confusing, or just incorrect, statements.

    Meta-issue trying to pinpoint the issues to help a novice reader, who may get misconceptions after reading these docs.

    opened by pfalcon 6
  • Many Errors

    Many Errors

    1. When I run klara on https://github.com/erezsh/runtype/blob/master/runtype/dataclass.py, I get AttributeError: 'JoinedStr' object has no attribute 'statement' :
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 354, in rename
        super(ParentScopeBlock, self).rename()
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 191, in rename
        blk.enumerate()
      File "c:\python38\lib\site-packages\klara\core\cfg.py", line 156, in enumerate
        AttributeEnumerator.enumerate(ast_stmt, False, False)
      File "c:\python38\lib\site-packages\klara\core\ssa.py", line 115, in enumerate
        var.convert_to_ssa()
      File "c:\python38\lib\site-packages\klara\core\node_classes.py", line 399, in convert_to_ssa
        stmt = field.statement()
    AttributeError: 'JoinedStr' object has no attribute 'statement'
    
    1. When I run it on https://github.com/lark-parser/lark/blob/master/lark/utils.py, I get KeyError: 'is not'
    ...
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1146, in infer_compare
        for result in calc_compare(comp, self.ops, context):
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1195, in calc_compare
        methods = _comp_op_methods(left, comp, op, context)
      File "c:\python38\lib\site-packages\klara\core\inference.py", line 1161, in _comp_op_methods
        method_name=COMP_OP_DUNDER_METHOD[op],
    KeyError: 'is not'
    
    1. When I run it on https://github.com/lark-parser/lark/blob/master/lark/tree.py I get AttributeError: type object 'Del' has no attribute 'targets'
    ...
        return [self._visit_generic(child) for child in node]
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 36, in _visit_generic
        return self._visit(node)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 24, in _visit
        returned = self._visit_generic(value)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 36, in _visit_generic
        return self._visit(node)
      File "c:\python38\lib\site-packages\klara\core\transform.py", line 23, in _visit
        value = getattr(node, name)
    AttributeError: type object 'Del' has no attribute 'targets'
    

    I'm sure I can keep going, but let's stop here for now :)

    opened by erezsh 5
  • AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'

    AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'

    Code:

    async def f():
        pass
    

    Error:

    ❯ klara tmp.py
    loaded extension: {'typeshed_stub.py', 'builtin_inference.py', '__init__.py', 'infer_z3.py', '99_math_z3.py'}
    
    using configuration value: 
    {   'config_file': None,
        'eq_neq': False,
        'input_test_file': 'tmp.py',
        'max_inference_value': None,
        'py_version': 3,
        'stubs': [],
        'type_inference': True,
        'typeshed_select': []}
    
    Traceback (most recent call last):
      File "/home/gram/.local/bin/klara", line 8, in <module>
        sys.exit(main())
      File "/home/gram/.local/lib/python3.9/site-packages/klara/contract/__main__.py", line 50, in main
        output_test = run(input_file.read_text(), input_file.stem)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/contract/__main__.py", line 36, in run
        cfg = MANAGER.build_cfg(tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/klara_z3/cov_manager.py", line 64, in build_cfg
        c = cfg.Cfg(as_tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 509, in __init__
        self.root, _, _ = self.parse(as_tree)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 629, in parse
        head = self.build(basic_block, head, all_tail_list, func_tail_list)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 535, in build
        tail_list, func_tail = meth(block)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 589, in build_module
        head_returned, tail_list, _ = self.parse(self.as_tree.body)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 627, in parse
        for basic_block in basic_block_parser.get_basic_block():
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 840, in get_basic_block
        basic_block_list = self.visit(node)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 821, in visit
        return visitor(ast_node)
      File "/home/gram/.local/lib/python3.9/site-packages/klara/core/cfg.py", line 824, in generic_visit
        self._append_cache(ast_node.statement())
    AttributeError: 'AsyncFunctionDef' object has no attribute 'statement'
    
    
    opened by orsinium 3
  • Improve this by defining the identity table as class variable

    Improve this by defining the identity table as class variable

    https://github.com/usagitoneko97/python-ast/blob/78330cbc3d4601160175f5073a6630a157fce6db/A3.LVN/lvn.py#L30

    Unlike the above, a class variable is instantiated only once when the class is created.

    The following shows how to define it as class variable.

    class Lvn:
        identity_expr = {(None, '*', 2):(None, '+', None), (None, '+', 0):('0', '+', None), .....}
    
        # Accessing the table is a bit tricky though since you need this class' handle to access it.
        # To get the current class, use type(self) as shown below.
        def get_alternate_id(self, id):
            return type(self).identity_expr.get(id)
    
    opened by chaosAD 2
  • Broken links to images, etc. in the docs

    Broken links to images, etc. in the docs

    It seems that https://github.com/usagitoneko97/python-static-code-analysis/commit/1595f838b49f364d456a10673d42914f1d859118 broke various links around. E.g. in https://github.com/usagitoneko97/python-static-code-analysis/blob/master/lvn_optimization/readme.md , links to SVG images are broken . E.g. go to https://github.com/usagitoneko97/python-static-code-analysis/tree/master/lvn_optimization#113-algorithm-in-details, "IvnThird" pseudo-link is rendered instead of an image.

    opened by pfalcon 1
  • Replaced with more aesthetic diagrams and reworded the Dominance:Introduction section

    Replaced with more aesthetic diagrams and reworded the Dominance:Introduction section

    I have replaced your old diagrams with aesthetically appealing diagrams and reworded your text. Please have a look.

    Please use my SVG templates to draw/redraw your diagrams.

    Please don't convert images to PNG files. Now GitHub support SVG directly. See my example.

    Please don't use full URL to your images like https://github.com/usagitoneko97/python-ast/blob/master/A4.Cfg/resources/cfg_ssa_intro.svg.png You should use just resources/cfg_ssa_intro.svg.png instead.

    opened by chaosAD 0
  • Edited readme for Python Implementation version 2

    Edited readme for Python Implementation version 2

    Please have a look at my edited readme.md file. I think you have to rewrite section 1.2.2 Converting Back To SSA. It is unclear and also I think the list given there is incorrect.

    opened by chaosAD 0
  • Api documentation

    Api documentation

    Hello, it seems that you api documentation page is empty. Maybe a misconfiguration of the autodoc Sphinx extension.

    As a temporary solution, the docs are available here: https://pydocbrowser.github.io/klara/latest/index.html

    opened by tristanlatr 0
  • Support for imports?

    Support for imports?

    Hello,

    First good job at writing this library. It looks like it has powerful capabilities.

    I was wondering if this library still maintained? Do you planned support for inferring imports anytime soon ?

    I’ve myself implemented a astroid-alike (intra procedural) inference engine: https://github.com/tristanlatr/astuce and it supports imports, except wildcard imports. But klara seems much more intelligent.

    Tell me what you think. Thanks

    opened by tristanlatr 0
  • BREAKING CHANGE: remove py2 support

    BREAKING CHANGE: remove py2 support

    I've removed typed-ast package at there are problems with this package in apple-silicon. Also, typed ast recommends using the builtin ast module from python 3.8 and up.

    I could not understand why the uts are failing, I would like to get help to fix those.

    opened by jochman 3
Releases(0.6.3)
  • 0.6.3(Sep 19, 2021)

    v0.6.3 Release Notes (09/19/21)

    • Fixed conditions in loop causing conflicting conditions propagation (#7)
    • fixed Del and Delete shared the same node, and caused fields error (#7)
    • implemented identity (is, is not) comparison for const and instance (#7)
    • added AsyncFunctionDef, Await, AsyncFor, AsyncWith ast support
    • implemented repr, ascii builtin call, and JoinedStr, FormattedValue inference (#7, #8)

    Fixed #7, #8

    Source code(tar.gz)
    Source code(zip)
Owner
Ho Guo Xian
I like automation.
Ho Guo Xian
User-interest mock backend server implemnted using flask restful, and SQLAlchemy ORM confiugred with sqlite

Flask_Restful_SQLAlchemy_server User-interest mock backend server implemnted using flask restful, and SQLAlchemy ORM confiugred with sqlite. Backend b

Austin Weigel 1 Nov 17, 2022
Repository for JIDA SNP Browser Web Application: Local Deployment

JIDA JIDA is a web application that retrieves SNP information for a genomic region of interest in Homo sapiens and calculates specific summary statist

3 Mar 03, 2022
py.test fixture for benchmarking code

Overview docs tests package A pytest fixture for benchmarking code. It will group the tests into rounds that are calibrated to the chosen timer. See c

Ionel Cristian Mărieș 1k Jan 03, 2023
A pytest plugin, that enables you to test your code that relies on a running PostgreSQL Database

This is a pytest plugin, that enables you to test your code that relies on a running PostgreSQL Database. It allows you to specify fixtures for PostgreSQL process and client.

Clearcode 252 Dec 21, 2022
nose is nicer testing for python

On some platforms, brp-compress zips man pages without distutils knowing about it. This results in an error when building an rpm for nose. The rpm bui

1.4k Dec 12, 2022
A simple Python script I wrote that scrapes NASA's James Webb Space Telescope tracker website using Selenium and returns its current status and location.

A simple Python script I wrote that scrapes NASA's James Webb Space Telescope tracker website using Selenium and returns its current status and location.

9 Feb 10, 2022
Selenium Manager

SeleniumManager I'm fed up with always having to struggle unnecessarily when I have to use Selenium on a new machine, so I made this little python mod

Victor Vague 1 Dec 24, 2021
Spam the buzzer and upgrade automatically - Selenium

CookieClicker Usage: Let's check your chrome navigator version : Consequently, you have to : download the right chromedriver in the follow link : http

Iliam Amara 1 Nov 22, 2021
A web scraping using Selenium Webdriver

Savee - Images Downloader Project using Selenium Webdriver to download images from someone's profile on https:www.savee.it website. Usage The project

Caio Eduardo Lobo 1 Dec 17, 2021
Simple assertion library for unit testing in python with a fluent API

assertpy Simple assertions library for unit testing in Python with a nice fluent API. Supports both Python 2 and 3. Usage Just import the assert_that

19 Sep 10, 2022
Declarative HTTP Testing for Python and anything else

Gabbi Release Notes Gabbi is a tool for running HTTP tests where requests and responses are represented in a declarative YAML-based form. The simplest

Chris Dent 139 Sep 21, 2022
Automated Penetration Testing Framework

Automated Penetration Testing Framework

OWASP 2.1k Jan 01, 2023
WomboAI Art Generator

WomboAI Art Generator Automate AI art generation using wombot.art. Also integrated into SnailBot for you to try out. Setup Install Python Go to the py

nbee 7 Dec 03, 2022
Youtube Tool using selenium Python

YT-AutoLikeComment-AutoReportComment-AutoComment Youtube Tool using selenium Python Auto Comment Auto Like Comment Auto Report Comment Usage: 1. Insta

Rahul Joshua Damanik 1 Dec 13, 2021
MultiPy lets you conveniently keep track of your python scripts for personal use or showcase by loading and grouping them into categories. It allows you to either run each script individually or together with just one click.

MultiPy About MultiPy is a graphical user interface built using Dear PyGui Python GUI Framework that lets you conveniently keep track of your python s

56 Oct 29, 2022
Pytest modified env

Pytest plugin to fail a test if it leaves modified os.environ afterwards.

wemake.services 7 Sep 11, 2022
GitHub action for AppSweep Mobile Application Security Testing

GitHub action for AppSweep can be used to continuously integrate app scanning using AppSweep into your Android app build process

Guardsquare 14 Oct 06, 2022
A modern API testing tool for web applications built with Open API and GraphQL specifications.

Schemathesis Schemathesis is a modern API testing tool for web applications built with Open API and GraphQL specifications. It reads the application s

Schemathesis.io 1.6k Dec 30, 2022
Code for "SUGAR: Subgraph Neural Network with Reinforcement Pooling and Self-Supervised Mutual Information Mechanism"

SUGAR Code for "SUGAR: Subgraph Neural Network with Reinforcement Pooling and Self-Supervised Mutual Information Mechanism" Overview train.py: the cor

41 Nov 08, 2022
🎓 Stepik Academy Автоматизация тестирования на Python

🎓 Stepik Academy Автоматизация тестирования на Python Запуск тестов выполняется в командной строке: pytest -v --tb=line --language=en --alluredir=all

Sergey 1 Dec 03, 2021