Python3 to Crystal Translation using Python AST Walker

Related tags

Text Data & NLPpy2cr
Overview

py2cr.py

A code translator using AST from Python to Crystal. This is basically a NodeVisitor with Crystal output. See AST documentation (https://docs.python.org/3/library/ast.html) for more information.

Status

Currently more than 80% of the relevant tests are passing. See more information below.

Installation

Execute the following:

pip install py2cr

or

git clone git://github.com/nanobowers/py2cr.git

Versions

  • Python 3.6 .. 3.9
  • Crystal 1.1+

Dependencies

Python

pip install pyyaml

# Probably not needed for much longer since py2 support is going to be removed.
pip install six 

# Probably not really needed since there is no crystal equivalent
pip install numpy

Crystal

currently there are no external dependencies

Methodology

In addition to walking and writing the AST tree and writing a Crystal syntax output, this tool either:

  • Monkey-patches some common Crystal stdlib Structs/Classes in order to emulate the Python equivalent functionality.
  • Calls equivalent Crystal methods to the Python equivalent
  • Calls wrapped Crystal methods that provide Python equivalent functionality

Usage

Generally, py2cr.py somefile.py > somefile.cr

There is a Crystal shim/wrapper library in src/py2cr (and linked into lib/py2cr) that is also referenced in the generated script. You may need to copy that as needed, though eventually it may be appropriate to convert it to a shard if that is more appropriate.

Example

TODO

Tests

$ ./run_tests.py

Will run all tests that are supposed to work. If any test fails, its a bug. (Currently there are a lot of failing tests!!)

$ ./run_tests.py -a

Will run all tests including those that are known to fail (currently). It should be understandable from the output.

$ ./run_tests.py basic

Will run all tests matching basic. Useful because running the entire test-suite can take a while.

$ ./run_tests.py -x or $ ./run_tests.py --no-error

Will run tests but ignore if an error is raised by the test. This is not affecting the error generated by the test files in the tests directory.

For additional information on flags, run:

./run_tests.py -h

Writing new tests

Adding tests for most new or existing functionality involves adding additional python files at tests/ .py .

The test-runner scripts will automatically run py2cr to produce a Crystal script, then run both the Python and Crystal scripts, then compare stdout/stderr and check return codes.

For special test-cases, it is possible to provide a configuration YAML file on a per test basis named tests/ / .config.yaml which overrides defaults for testing. The following keys/values are supported:

min_python_version: [int, int] # minimum major/minor version
max_python_version: [int, int] # maximum major/minor version
expected_exit_status: int      # exit status for py/cr test script
argument_list: [str, ... str]  # list of strings as extra args for argv

Typing

Some amount of typing support in Python is translated to Crystal. Completely untyped Python code in many cases will not be translatable to compilable Crystal. Rudimentary for python Optional and Union should convert appropriately to Crystal typing.

Some inference of bare list/dict types can now convert to [] of X and {} of X, however set and tuple may not work properly.

Status

This is incomplete and many of the tests brought forward from py2rb do not pass. Some of them may never pass as-is due to significant language / compilation differences (even moreso than Python vs. Ruby)

To some extent, it will always be incomplete. The goal is to cover common cases and reduce the additional work to minimum-viable-program.

Limitations

  • Many Python run-time exceptions are not translatable into Crystal as these issues manifest in Crystal as compile-time errors.
  • A significant portion of python code is untyped and may not translate properly in places where Crystal demands type information.
    • e.g. Crystal Lambda function parameters require typing and this is very uncommon in Python, though may be possible with Callable[] on the python side.
  • Python importing is significantly different than Crystal and thus may not ever map well.
  • Numpy and Unittest which are common in Python don't have equivalents in Crystal. With some significant additional work, converting tests into Spec format may be possible via https://github.com/jaredbeck/minitest_to_rspec as a guide

To-do

  • Remove python2/six dependencies to reduce clutter. Py2 has been end-of-lifed for a while now.
  • Remove numpy dependencies unless/until a suitable target for Crystal can be identified
  • Add additional Crystal shim methods to translate common python3 stdlib methods. Consider a mode that just maps to a close Crystal method rather than using a shim-method to reduce the python-ness.
  • Refactor the code-base. Most of it is in the __init__.py
  • Add additional unit-tests
  • Multi-thread the test-suite so it can run faster.

Contribute

Free to submit an issue. This is very much a work in progress, contributions or constructive feedback is welcome.

If you'd like to hack on py2cr, start by forking the repo on GitHub:

https://github.com/nanobowers/py2cr

Contributing

The best way to get your changes merged back into core is as follows:

  1. Fork it (https://github.com/nanobowers/py2cr/fork)
  2. Create a thoughtfully named topic branch to contain your change (git checkout -b my-new-feature)
  3. Hack away
  4. Add tests and make sure everything still passes by running crystal spec
  5. If you are adding new functionality, document it in the README
  6. If necessary, rebase your commits into logical chunks, without errors
  7. Commit your changes (git commit -am 'Add some feature')
  8. Push to the branch (git push origin my-new-feature)
  9. Create a new Pull Request

License

MIT, see the LICENSE file for exact details.

A Telegram bot to add notes to Flomo.

flomo bot 使用 Telegram 机器人发送笔记到你的 Flomo. 你需要有一台可访问 Telegram 的服务器。 Steps @BotFather 新建机器人,获取 token Flomo 官网获取 API,链接 https://flomoapp.com/mine?source=in

Zhen 44 Dec 30, 2022
Anomaly Detection 이상치 탐지 전처리 모듈

Anomaly Detection 시계열 데이터에 대한 이상치 탐지 1. Kernel Density Estimation을 활용한 이상치 탐지 train_data_path와 test_data_path에 존재하는 시점 정보를 포함하고 있는 csv 형태의 train data와

CLUST-consortium 43 Nov 28, 2022
Guide to using pre-trained large language models of source code

Large Models of Source Code I occasionally train and publicly release large neural language models on programs, including PolyCoder. Here, I describe

Vincent Hellendoorn 947 Dec 28, 2022
Pipeline for training LSA models using Scikit-Learn.

Latent Semantic Analysis Pipeline for training LSA models using Scikit-Learn. Usage Instead of writing custom code for latent semantic analysis, you j

Dani El-Ayyass 23 Sep 05, 2022
Korean extractive summarization. 2021 AI 텍스트 요약 온라인 해커톤 화성갈끄니까팀 코드

korean extractive summarization 2021 AI 텍스트 요약 온라인 해커톤 화성갈끄니까팀 코드 Leaderboard Notice Text Summarization with Pretrained Encoders에 나오는 bertsumext모델(ext

3 Aug 10, 2022
Code for text augmentation method leveraging large-scale language models

HyperMix Code for our paper GPT3Mix and conducting classification experiments using GPT-3 prompt-based data augmentation. Getting Started Installing P

NAVER AI 47 Dec 20, 2022
A simple tool to update bib entries with their official information (e.g., DBLP or the ACL anthology).

Rebiber: A tool for normalizing bibtex with official info. We often cite papers using their arXiv versions without noting that they are already PUBLIS

(Bill) Yuchen Lin 2k Jan 01, 2023
DAGAN - Dual Attention GANs for Semantic Image Synthesis

Contents Semantic Image Synthesis with DAGAN Installation Dataset Preparation Generating Images Using Pretrained Model Train and Test New Models Evalu

Hao Tang 104 Oct 08, 2022
Common Voice Dataset explorer

Common Voice Dataset Explorer Common Voice Dataset is by Mozilla Made during huggingface finetuning week Usage pip install -r requirements.txt streaml

Ceyda Cinarel 22 Nov 16, 2022
Code for the paper "BERT Loses Patience: Fast and Robust Inference with Early Exit".

Patience-based Early Exit Code for the paper "BERT Loses Patience: Fast and Robust Inference with Early Exit". NEWS: We now have a better and tidier i

Kevin Canwen Xu 54 Jan 04, 2023
RecipeReduce: Simplified Recipe Processing for Lazy Programmers

RecipeReduce This repo will help you figure out the amount of ingredients to buy for a certain number of meals with selected recipes. RecipeReduce Get

Qibin Chen 9 Apr 22, 2022
Transformers Wav2Vec2 + Parlance's CTCDecodeTransformers Wav2Vec2 + Parlance's CTCDecode

🤗 Transformers Wav2Vec2 + Parlance's CTCDecode Introduction This repo shows how 🤗 Transformers can be used in combination with Parlance's ctcdecode

Patrick von Platen 9 Jul 21, 2022
A simple version of DeTR

DeTR-Lite A simple version of DeTR Before you enjoy this DeTR-Lite The purpose of this project is to allow you to learn the basic knowledge of DeTR. P

Jianhua Yang 11 Jun 13, 2022
OpenAI CLIP text encoders for multiple languages!

Multilingual-CLIP OpenAI CLIP text encoders for any language Colab Notebook · Pre-trained Models · Report Bug Overview OpenAI recently released the pa

Fredrik Carlsson 481 Dec 30, 2022
Implemented shortest-circuit disambiguation, maximum probability disambiguation, HMM-based lexical annotation and BiLSTM+CRF-based named entity recognition

Implemented shortest-circuit disambiguation, maximum probability disambiguation, HMM-based lexical annotation and BiLSTM+CRF-based named entity recognition

0 Feb 13, 2022
A Facebook Messenger Chatbot using NLP

A Facebook Messenger Chatbot using NLP This project is about creating a messenger chatbot using basic NLP techniques and models like Logistic Regressi

6 Nov 20, 2022
DeepSpeech - Easy-to-use Speech Toolkit including SOTA ASR pipeline, influential TTS with text frontend and End-to-End Speech Simultaneous Translation.

(简体中文|English) Quick Start | Documents | Models List PaddleSpeech is an open-source toolkit on PaddlePaddle platform for a variety of critical tasks i

5.6k Jan 03, 2023
Unofficial Python library for using the Polish Wordnet (plWordNet / Słowosieć)

Polish Wordnet Python library Simple, easy-to-use and reasonably fast library for using the Słowosieć (also known as PlWordNet) - a lexico-semantic da

Max Adamski 12 Dec 23, 2022
🤗 Transformers: State-of-the-art Machine Learning for Pytorch, TensorFlow, and JAX.

English | 简体中文 | 繁體中文 | 한국어 State-of-the-art Machine Learning for JAX, PyTorch and TensorFlow 🤗 Transformers provides thousands of pretrained models

Hugging Face 77.1k Dec 31, 2022