An application that maps an image of a LaTeX math equation to LaTeX code.

Overview

Image to LaTeX

Code style: black pre-commit License

An application that maps an image of a LaTeX math equation to LaTeX code.

Image to Latex streamlit app

Introduction

The problem of image-to-markup generation has been attempted by Deng et al. (2016). They provide the raw and preprocessed versions of im2latex-100K, a dataset consisting of about 100K LaTeX math equation images. Using their dataset, I trained a model that uses ResNet-18 as encoder (up to layer3) and a Transformer as decoder with cross-entropy loss.

Initially, I used the preprocessed dataset to train my model, but the preprocessing turned out to be a huge limitation. Although the model can achieve a reasonable performance on the test set, it performs poorly if the image quality, padding, or font size is different from the images in the dataset. This phenomenon has also been observed by others who have attempted the same problem using the same dataset (e.g., this project, this issue and this issue). This is most likely due to the rigid preprocessing for the dataset (e.g. heavy downsampling).

To this end, I used the raw dataset and included image augmentation (e.g. random scaling, small rotation) in my data processing pipeline to increase the diversity of the samples. Moreover, unlike Deng et al. (2016), I did not group images by size. Rather, I sampled them uniformly and padded them to the size of the largest image in the batch, to increase the generalizability of the model.

Additional problems that I found in the dataset:

  • Some latex code produces visually identical outputs (e.g. \left( and \right) look the same as ( and )), so I normalized them.
  • Some latex code is used to add space (e.g. \vspace{2px} and \hspace{0.3mm}). However, the length of the space is diffcult to judge. Also, I don't want the model generates code on blank images, so I removed them.

The best run has a character error rate (CER) of 0.17 in test set. Most errors seem to come from unnecessary horizontal spacing, e.g., \;, \, and \qquad. (I only removed \vspace and \hspace during preprocessing. I did not know that LaTeX has so many horizontal spacing commands.)

Possible improvements include:

  • Do a better job cleaning the data (e.g., removing spacing commands)
  • Train the model for more epochs (for the sake of time, I only trained the model for 15 epochs, but the validation loss is still going down)
  • Use beam search (I only implemented greedy search)
  • Use a larger model (e.g., use ResNet-34 instead of ResNet-18)
  • Do some hyperparameter tuning

I didn't do any of these, because I had limited computational resources (I was using Google Colab).

How To Use

Setup

Clone the repository to your computer and position your command line inside the repository folder:

git clone https://github.com/kingyiusuen/image-to-latex.git
cd image-to-latex

Then, create a virtual environment named venv and install required packages:

make venv
make install-dev

Data Preprocessing

Run the following command to download the im2latex-100k dataset and do all the preprocessing. (The image cropping step may take over an hour.)

python scripts/prepare_data.py

Model Training and Experiment Tracking

Model Training

An example command to start a training session:

python scripts/run_experiment.py trainer.gpus=1 data.batch_size=32

Configurations can be modified in conf/config.yaml or in command line. See Hydra's documentation to learn more.

Experiment Tracking using Weights & Biases

The best model checkpoint will be uploaded to Weights & Biases (W&B) automatically (you will be asked to register or login to W&B before the training starts). Here is an example command to download a trained model checkpoint from W&B:

python scripts/download_checkpoint.py RUN_PATH

Replace RUN_PATH with the path of your run. The run path should be in the format of //. To find the run path for a particular experiment run, go to the Overview tab in the dashboard.

For example, you can use the following command to download my best run

python scripts/download_checkpoint.py kingyiusuen/image-to-latex/1w1abmg1

The checkpoint will be downloaded to a folder named artifacts under the project directory.

Testing and Continuous Integration

The following tools are used to lint the codebase:

isort: Sorts and formats import statements in Python scripts.

black: A code formatter that adheres to PEP8.

flake8: A code linter that reports stylistic problems in Python scripts.

mypy: Performs static type checking in Python scripts.

Use the following command to run all the checkers and formatters:

make lint

See pyproject.toml and setup.cfg at the root directory for their configurations.

Similar checks are done automatically by the pre-commit framework when a commit is made. Check out .pre-commit-config.yaml for the configurations.

Deployment

An API is created to make predictions using the trained model. Use the following command to get the server up and running:

make api

You can explore the API via the generated documentation at http://0.0.0.0:8000/docs.

To run the Streamlit app, create a new terminal window and use the following command:

make streamlit

The app should be opened in your browser automatically. You can also open it by visiting http://localhost:8501. For the app to work, you need to download the artifacts of an experiment run (see above) and have the API up and running.

To create a Docker image for the API:

make docker

Acknowledgement

Simple utility to tinker with OPlus images

OPlus image utilities Prerequisites Linux running kernel 5.4 or up (check with uname -r) Image rebuilding Used to rebuild read-only erofs images into

Wiley Lau 15 Dec 28, 2022
Utilities for SteamVR on Linux

This project contains scripts to improve the functionally of SteamVR on Linux:

86 Dec 29, 2022
An ascii art generator that's actually good. Does edge detection and selects the most appropriate characters.

Ascii Artist An ascii art generator that's actually good. Does edge detection and selects the most appropriate characters. Installing Installing with

18 Jan 03, 2023
PIX is an image processing library in JAX, for JAX.

PIX PIX is an image processing library in JAX, for JAX. Overview JAX is a library resulting from the union of Autograd and XLA for high-performance ma

DeepMind 294 Jan 08, 2023
This projects aim is to simulate flowers(Gerbera Daisy) phyllotaxis.

phyllotaxis This projects aim is to simulate flowers(Gerbera Daisy) phyllotaxis. Take a look at the arrangement of this flower's seeds, this project's

amirsalar 3 Dec 10, 2021
Nudity detection with Python

nude.py About Nudity detection with Python. Port of nude.js to Python. Installation from pip: $ pip install --upgrade nudepy from easy_install: $ eas

Hideo Hattori 881 Jan 06, 2023
Tool made for the FWA Yearbook Team to resize multiple images quickly.

ImageResize Tool Tool made for the FWA Yearbook Team to resize multiple images quickly. Make sure to check this repo for future updates How to Use The

LGobin 1 Jan 07, 2022
This is a python project which detects color of an image when you double click on it.

This is a python project which detects color of an image when you double click on it. You have to press ESC button to close the pop-up Image window. There are mainly two library CV2 and Pandas that a

Yashwant Kumar Singh 0 Aug 16, 2022
粉專/IG圖文加工器

粉專/IG圖文加工器 介紹 給PS智障(ex:我)使用,用於產生圖文 腳本省去每次重複步驟 可載入圖片(方形,請先處理過,歡迎PR) 圖片簡易套用濾鏡 可將圖片切片 要求 Python 版本 3.9 安裝 安裝最新 python pip3 install -r requirement.txt 效果

Louis Tang 7 Aug 10, 2022
AutoGiphyMovie lets you search giphy for gifs, converts them to videos, attach a soundtrack and stitches it all together into a movie!

AutoGiphyMovie lets you search giphy for gifs, converts them to videos, attach a soundtrack and stitches it all together into a movie!

Satya Mohapatra 18 Nov 13, 2022
Napari plugin for iteratively improving 3D instance segmentation of cells (u-net x watershed)

iterseg napari plugin for iteratively improving unet-watershed segmentation This napari plugin was generated with Cookiecutter using @napari's cookiec

Abigail McGovern 3 May 16, 2022
Script for the creation of metadatas and the randomization of images of MekaVerse

MekaVerse-random Script for the creation of metadata and the randomization of images of MekaVerse Step to replay the random : Create a folder : output

Miinded 8 Sep 07, 2022
Sample data for the napari image viewer.

napari-demo-data Sample data for the napari image viewer. This napari plugin was generated with Cookiecutter using @napari's cookiecutter-napari-plugi

Genevieve Buckley 1 Nov 08, 2021
Maze generator with most popular shapes - hexagon, triangle, square

Maze-Generator Maze generator with most popular shapes - hexagon, triangle, square (sqaure not implemented yet): Theory: Planar Graph https://en.wikip

Kacper Plesiak 2 Dec 28, 2021
Python wrappers for external BART computational imaging tools and internal libraries

bartpy Python bindings for BART. Overview This repo contains code to generate an updated Python wrapper for the Berkeley Advance Reconstruction Toolbo

Max Litster 7 May 09, 2022
MaryJane is a simple MJPEG server written in Python.

MaryJane is a simple MJPEG server written in Python.

bootrino 152 Dec 13, 2022
A scalable implementation of WobblyStitcher for 3D microscopy images

WobblyStitcher Introduction A scalable implementation of WobblyStitcher Dependencies $ python -m pip install numpy scikit-image Visualization ImageJ

CSE Lab, ETH Zurich 7 Jul 25, 2022
QR-code Generator with a basic GUI.

Qr_generator_python Qr code generator with a basic GUI. ❔ About the QR-Code-Generator This project Generates QR codes to sites, e-mails and plain text

Tecixck 2 Oct 11, 2021
Samila is a generative art generator written in Python

Samila is a generative art generator written in Python, Samila let's you create arts based on many thousand points. The position of every single point is calculated by a formula, which has random par

Sepand Haghighi 947 Dec 30, 2022
【萝莉图片算法】高损图像压缩算法!?

【萝莉图片算法】高损图像压缩算法!? 我又发明出新算法了! 这次我发明的是新型高损图像压缩算法——萝莉图片算法!为什么是萝莉图片,这是因为它是使动用法,让图片变小所以是萝莉图片,大家一定要学好语文哦! 压缩效果 太神奇了!压缩率竟然高达99.97%! 与常见压缩算法对比 在图片最终大小为1KB的情况

黄巍 49 Oct 17, 2022