PyCG: Practical Python Call Graphs

Overview

PyCG - Practical Python Call Graphs

PyCG generates call graphs for Python code using static analysis. It efficiently supports

  • Higher order functions
  • Twisted class inherritance schemes
  • Automatic discovery of imported modules for further analysis
  • Nested definitions

You can read the full methodology as well as a complete evaluation on the ICSE 2021 paper.

Abstract:

Call graphs play an important role in different contexts, such as profiling and vulnerability propagation analysis. Generating call graphs in an efficient manner can be a challenging task when it comes to high-level languages that are modular and incorporate dynamic features and higher-order functions. Despite the language's popularity, there have been very few tools aiming to generate call graphs for Python programs. Worse, these tools suffer from several effectiveness issues that limit their practicality in realistic programs. We propose a pragmatic, static approach for call graph generation in Python. We compute all assignment relations between program identifiers of functions, variables, classes, and modules through an inter-procedural analysis. Based on these assignment relations, we produce the resulting call graph by resolving all calls to potentially invoked functions. Notably, the underlying analysis is designed to be efficient and scalable, handling several Python features, such as modules, generators, function closures, and multiple inheritance. We have evaluated our prototype implementation, which we call PyCG, using two benchmarks: a micro-benchmark suite containing small Python programs and a set of macro-benchmarks with several popular real-world Python packages. Our results indicate that PyCG can efficiently handle thousands of lines of code in less than a second (0.34 seconds for 1k LoC on average). Further, it outperforms the state-of-the-art for Python in both precision and recall: PyCG achieves high rates of precision ~99, and adequate recall ~69.3. Finally, we demonstrate how PyCG can aid dependency impact analysis by showcasing a potential enhancement to GitHub's security advisory notification service using a real-world example.

Installation

PyCG is implemented in Python3 and has no dependencies. Simply:

pip install pycg

Usage

~ >>> pycg -h
usage: pycg [-h] [--package PACKAGE] [--fasten] [--product PRODUCT]
            [--forge FORGE] [--version VERSION] [--timestamp TIMESTAMP]
            [-o OUTPUT]
            [entry_point [entry_point ...]]

positional arguments:
  entry_point           Entry points to be processed

optional arguments:
  -h, --help            show this help message and exit
  --package PACKAGE     Package containing the code to be analyzed
  --fasten              Produce call graph using the FASTEN format
  --product PRODUCT     Package name
  --forge FORGE         Source the product was downloaded from
  --version VERSION     Version of the product
  --timestamp TIMESTAMP
                        Timestamp of the package's version
  -o OUTPUT, --output OUTPUT
                        Output path

where the command line arguments are:

  • entry_point: A list of paths to Python modules that PyCG will analyze. It is suggested that this list of paths contains only entry points since PyCG automatically discovers all other (local) imported modules.
  • --package: The unix path to the module's namespace (i.e. the path from which the module would be executed). This parameter is really important for the correct resolving of imports.
  • --fasten: Output the callgraph in FASTEN format.
  • -output: The unix path where the output call graph will be stored in JSON format.

The following command line arguments should used only when --fasten is provied:

  • --product: The name of the package.
  • --forge: Source the package was downloaded from.
  • --version: The version of the package.
  • --timestamp : The timestamp of the package's version.

Output

Simple JSON format

The call edges are in the form of an adjacency list where an edge (src, dst) is represented as an entry of dst in the list assigned to key src:

{
    "node1": ["node2", "node3"],
    "node2": ["node3"],
    "node3": []
}

FASTEN Format

For an up-to-date description of the FASTEN format refer to the FASTEN wiki.

Examples

All the entry points are known and we want the simple JSON format

~ >>> pycg --package pkg_root pkg_root/module1.py pkg_root/subpackage/module2.py -o cg.json

All entry points are not known and we want the simple JSON format

~ >>> pycg --package django $(find django -type f -name "*.py") -o django.json

We want the FASTEN format:

~ >>> pycg --package pypi_pkg --fasten --product "pypipkg" --forge "PyPI" \
        --version "0.1" --timestamp 42 \
        pypi_pkg/module1.py pkg_root/subpackage/module2.py -o cg.json

Running Tests

From the root directory:

make test
Owner
Vitalis Salis
ECE undergraduate @NTUA Researcher @AUEB-BALab
Vitalis Salis
AutoCalypsoBTS - A simple GUI CalypsoBTS for DragonOS by CrTh

AutoCalypsoBTS 📱 📞 A simple GUI CalypsoBTS for DragonOS by CrTh Download Drago

CrTx0 18 Dec 15, 2022
A simple browser for familiarity to PyQt :)

Erown_Browser a simple browser for familiarity to PyQt :) for start work with this install requirements by pip install -r requirements or you can use

Ehsan Amirahmadi 9 May 18, 2022
A Python Tkinter based Inventory managment System

Inventory Management System Using Python Tkinter Introduction Inventory managemrnt system is an open source platform for manage business. It has a com

Amit Kumar Datta 2 Oct 14, 2021
Learn to build a Python Desktop GUI app using pywebview, Python, JavaScript, HTML, & CSS.

Python Desktop App Learn how to make a desktop GUI application using Python, JavaScript, HTML, & CSS all thanks to pywebview. pywebview is essentially

Coding For Entrepreneurs 55 Jan 05, 2023
Python3.9/Kivy2.0 project to manage Valkyrie Scenarios

.valkyrie Manager This is a Python 3.9/Kivy 2.0 application that helps to manage quest (.valkyrie) files (from Valkyrie app) It identifies new quests

Kempes J. 1 Jan 28, 2022
Create highly interactive web pages purely in Python

A package for building highly interactive user interfaces in pure Python inspired by ReactJS.

701 Jan 03, 2023
A simple desktop news application written using python created using PyQt5

News-Application---Python This is a news application created using PyQt5. News is fetched through API from newsapi.org. Available top headlines from c

Sritiman Adak 1 Nov 14, 2021
Cross-platform BrowserViews for the desktop.

Webview We use wxPython Phoenix to provide webviews. It's cross platform between Windows and macOS primarily, Linux versions require extra setup. Appl

1 Feb 12, 2022
All you need to learn Tkinter!

Tkinter This repository contains the codes and resources which I used to learn the standard GUI library of Python, Tkinter! Best Tkinter Resources Vid

Samyak Jain 3 May 02, 2022
TkArt - A repository created to explore geometry and art creation using TkInter

tkArt A repository created to explore geometry and art creation using TkInter, a

Jayant Sogikar 18 Oct 01, 2022
A simple todo GUI applicaiton

simple_todo_gui A simple todo GUI applicaiton To create an .exe file, run 'Python setup.py build' after installing PyQt5 and cx_Freeze with pip. Then

Dhammike Piyumal 2 Nov 11, 2021
PyCG: Practical Python Call Graphs

PyCG - Practical Python Call Graphs PyCG generates call graphs for Python code using static analysis. It efficiently supports Higher order functions T

Vitalis Salis 185 Dec 29, 2022
Python Screen Recorder

Python Screen Recorder a simple customizable screen recorder made in python 🐍 Requirements Operation system: Windows Python Version: 3.9.x Required M

Arsh 3 May 25, 2022
A python Script For Taking Screenshot Of Windows

PyShot A Python Script For Taking Screenshot Of Windows Disclaimer This tool is for educational purposes only ! Don't use this to take revenge I will

Nazim Cp 2 Jun 22, 2022
A GUI based CRUD database management system built using mysql and python

A GUI based CRUD database management system built using mysql and python

Aquila 2 Feb 13, 2022
A GUI frontend developed in Python3 for simple admin tasks on MySQL-Server running on the localhost

A GUI frontend developed in Python3 for simple admin tasks on MySQL-Server running on the localhost. Developed for my CS School Project.

1 May 29, 2022
Tukaan is the new framework that aims to replace Tkinter

Tukaan is the new, pythonic and colorful (like a keel-billed toucan) framework that aims to replace Tkinter. It has everything (on my computer, not at GitHub) that you need to develop cross-platform

Tukaan 101 Jan 08, 2023
The GUI application by Python3.8. Using QT Design draw UI and generator UI XML file provides to PySide2 build GUI components

The GUI application by Python3.8. Using QT Design draw UI and generator UI XML file provides to PySide2 build GUI components. Total adopt OOD design class, service, and abstract class. OOP implemente

Jiage 1 Jan 11, 2022
Dear PyGui Extensions is a collection of useful tools, abstractions, and simplification layers built with/for Dear PyGui users.

Dear PyGui Extensions: A collection of useful tools, abstractions, and simplification layers built with/for Dear PyGui users.

Jonathan Hoffstadt 34 Jan 01, 2023
A Minimalistic Backup GUI for your Windows, Mac or Linux

BlobBackup is a minimalistic backup utility for your Windows, Mac or Linux computer. With an excellent engine, extensive storage support, and an easy

Bimba Shrestha 283 Nov 30, 2022