TedEval: A Fair Evaluation Metric for Scene Text Detectors

Overview

TedEval: A Fair Evaluation Metric for Scene Text Detectors

Official Python 3 implementation of TedEval | paper | slides

Chae Young Lee, Youngmin Baek, and Hwalsuk Lee.

Clova AI Research, NAVER Corp.

Overview

We propose a new evaluation metric for scene text detectors called TedEval. Through separate instance-level matching policy and character-level scoring policy, TedEval solves the drawbacks of previous metrics such as IoU and DetEval. This code is based on ICDAR15 official evaluation code.

Methodology

1. Mathcing Policy

  • Non-exclusively gathers all possible matches of not only one-to-one but also one-to-many and many-to-one.
  • The threshold of both area recall and area precision are set to 0.4.
  • Multiline is identified and rejected when |min(theta, 180 - theta)| > 45 from Fig. 2.

2. Scoring Policy

We compute Pseudo Character Center (PCC) from word-level bounding boxes and penalize matches when PCCs are missing or overlapping.

Sample Evaluation

Experiments

We evaluated state-of-the-art scene text detectors with TedEval on two benchmark datasets: ICDAR 2013 Focused Scene Text (IC13) and ICDAR 2015 Incidental Scene Text (IC15). Detectors are listed in the order of published dates.

ICDAR 2013

Detector Date (YY/MM/DD) Recall (%) Precision (%) H-mean (%)
CTPN 16/09/12 82.1 92.7 87.6
RRPN 17/03/03 89.0 94.2 91.6
SegLink 17/03/19 65.6 74.9 70.0
EAST 17/04/11 77.7 87.1 82.5
WordSup 17/08/22 87.5 92.2 90.2
PixelLink 18/01/04 84.0 87.2 86.1
FOTS 18/01/05 91.5 93.0 92.6
TextBoxes++ 18/01/09 87.4 92.3 90.0
MaskTextSpotter 18/07/06 90.2 95.4 92.9
PMTD 19/03/28 94.0 95.2 94.7
CRAFT 19/04/03 93.6 96.5 95.1

ICDAR 2015

Detector Date (YY/MM/DD) Recall (%) Precision (%) H-mean (%)
CTPN 16/09/12 85.0 81.1 67.8
RRPN 17/03/03 79.5 85.9 82.6
SegLink 17/03/19 77.1 83.9 80.6
EAST 17/04/11 82.5 90.0 86.3
WordSup 17/08/22 83.2 87.1 85.2
PixelLink 18/01/04 85.7 86.1 86.0
FOTS 18/01/05 89.0 93.4 91.2
TextBoxes++ 18/01/09 82.4 90.8 86.5
MaskTextSpotter 18/07/06 82.5 91.8 86.9
PMTD 19/03/28 89.2 92.8 91.0
CRAFT 19/04/03 88.5 93.1 90.9

Frequency

Getting Started

Clone repository

git clone https://github.com/clovaai/TedEval.git

Requirements

  • python 3
  • python 3.x Polygon, Bottle, Pillow
# install
pip3 install Polygon3 bottle Pillow

Supported Annotation Type

  • LTRB(xmin, ymin, xmax, ymax)
  • QUAD(x1, y1, x2, y2, x3, y3, x4, y4)

Evaluation

Prepare data

The ground truth and the result data should be text files, one for each sample. Note that the naming rule of each text file is that there must be img_{number} in the filename and that the number indicate the image sample.

# gt/gt_img_38.txt
644,101,932,113,932,168,643,156,[email protected]
477,138,487,139,488,149,477,148,###
344,131,398,130,398,149,344,149,###
1195,148,1277,138,1277,177,1194,187,###
23,270,128,267,128,282,23,284,###

# result/res_img_38.txt
644,101,932,113,932,168,643,156,{Transcription},{Confidence}
477,138,487,139,488,149,477,148
344,131,398,130,398,149,344,149
1195,148,1277,138,1277,177,1194,187
23,270,128,267,128,282,23,284

Compress these text files.

zip gt.zip gt/*
zip result.zip result/*

Refer to gt/result.zip and gt/gt_*.zip for examples.

Run stand-alone evaluation

python script.pyg=gt/gt.zips=result/result.zip
  • Locate the path of GT and submission file using the flag -g and -s, respectively.
  • QUAD annotation type is used as default. To switch between {QUAD, LTRB}, add -p='{"LTRB" = False}' in the command or directly modify the default_evaluation_params() function in script.py.
  • If there are transcription or confidence values in your submission file, add -p='{"CONFIDENCES" = True} or -p='{"TRANSCRIPTION" = True}'.

Run Visualizer

python web.py
  • Place the zip file of images and GTs of the dataset named images.zip and gt.zip, respectively, in the gt directory.
  • Create an empty directory name output. This is where the DB, submission files, and result files will be created.
  • You can change the host and port number in the final line of web.py.

The file structure should then be:

.
├── gt
│   ├── gt.zip
│   └── images.zip
├── output   # empty dir
├── script.py
├── web.py
├── README.md
└── ...

Citation

@article{lee2019tedeval,
  title={TedEval: A Fair Evaluation Metric for Scene Text Detectors},
  author={Lee, Chae Young and Baek, Youngmin and Lee, Hwalsuk},
  journal={arXiv preprint arXiv:1907.01227},
  year={2019}
}

Contact us

We welcome any feedbacks to our metric. Please contact the authors via {cylee7133, youngmin.baek, hwalsuk.lee}@gmail.com. In case of code errors, open an issue and we will get to you.

License

Copyright (c) 2019-present NAVER Corp.

 Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

 The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

 THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.  IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
THE SOFTWARE.
Owner
Clova AI Research
Open source repository of Clova AI Research, NAVER & LINE
Clova AI Research
Virtualdragdrop - Virtual Drag and Drop Using OpenCV and Arduino

Virtualdragdrop - Virtual Drag and Drop Using OpenCV and Arduino

Rizky Dermawan 4 Mar 10, 2022
OCR system for Arabic language that converts images of typed text to machine-encoded text.

Arabic OCR OCR system for Arabic language that converts images of typed text to machine-encoded text. The system currently supports only letters (29 l

Hussein Youssef 144 Jan 05, 2023
A curated list of awesome synthetic data for text location and recognition

awesome-SynthText A curated list of awesome synthetic data for text location and recognition and OCR datasets. Text location SynthText SynthText_Chine

Tianzhong 283 Jan 05, 2023
This repo contains several opencv projects done while learning opencv in python.

opencv-projects-python This repo contains both several opencv projects done while learning opencv by python and opencv learning resources [Basic conce

Fatin Shadab 2 Nov 03, 2022
OCR, Object Detection, Number Plate, Real Time

README.md PrePareded anaconda env requirements.txt clova AI → deep text recognition → trained weights (ex, .pth) wpod-net weights (ex, .h5 , .json) ht

Kaven Lee 7 Dec 06, 2022
Generating .npy dataset and labels out of given image, containing numbers from 0 to 9, using opencv

basic-dataset-generator-from-image-of-numbers generating .npy dataset and labels out of given image, containing numbers from 0 to 9, using opencv inpu

1 Jan 01, 2022
The open source extract transaction infomation by using OCR.

Transaction OCR Mã nguồn trích xuất thông tin transaction từ file scaned pdf, ở đây tôi lựa chọn tài liệu sao kê công khai của Thuy Tien. Mã nguồn có

Nguyen Xuan Hung 18 Jun 02, 2022
A bot that plays TFT using OCR. Keeps track of bench, board, items, and plays the user defined team comp.

NOTES: To ensure best results, make sure you are running this on a computer that has decent specs. 1920x1080 fullscreen is required in League, game mu

francis 125 Dec 30, 2022
Generate text images for training deep learning ocr model

New version release:https://github.com/oh-my-ocr/text_renderer Text Renderer Generate text images for training deep learning OCR model (e.g. CRNN). Su

Qing 1.2k Jan 04, 2023
https://arxiv.org/abs/1904.01941

Character-Region-Awareness-for-Text-Detection- https://arxiv.org/abs/1904.01941 Train You can train SynthText data use python source/train_SynthText.p

DayDayUp 120 Dec 28, 2022
Shape Detection - It's a shape detection project with OpenCV and Python.

Shape Detection It's a shape detection project with OpenCV and Python. Setup pip install opencv-python for doing AI things. pip install simpleaudio fo

1 Nov 26, 2022
Document Layout Analysis Projects

Layout_Analysis Introduction This is an implementation of RLSA and X-Y Cut with OpenCV Dependencies OpenCV 3.0+ How to use Compile with g++ : g++ -std

22 Dec 08, 2022
Hand Detection and Finger Detection on Live Feed

Hand-Detection-On-Live-Feed Hand Detection and Finger Detection on Live Feed Getting Started Install the dependencies $ git clone https://github.com/c

Chauhan Mahaveer 2 Jan 02, 2022
基于Paddle框架的PSENet复现

PSENet-Paddle 基于Paddle框架的PSENet复现 本项目基于paddlepaddle框架复现PSENet,并参加百度第三届论文复现赛,将在2021年5月15日比赛完后提供AIStudio链接~敬请期待 AIStudio链接 参考项目: whai362-PSENet 环境配置 本项目

QuanHao Guo 4 Apr 24, 2022
fishington.io bot with OpenCV and NumPy

fishington.io-bot fishington.io bot with using OpenCV and NumPy bot can continue to fishing fully automatically how to use Open cmd in fishington.io-b

Bahadır Araz 77 Jan 02, 2023
Learning Camera Localization via Dense Scene Matching, CVPR2021

This repository contains code of our CVPR 2021 paper - "Learning Camera Localization via Dense Scene Matching" by Shitao Tang, Chengzhou Tang, Rui Hua

tangshitao 65 Dec 01, 2022
Code for CVPR'2022 paper ✨ "Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model"

PPE ✨ Repository for our CVPR'2022 paper: Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-

Zipeng Xu 34 Nov 28, 2022
A facial recognition program that plays a alarm (mp3 file) when a person i seen in the room. A basic theif using Python and OpenCV

Home-Security-Demo A facial recognition program that plays a alarm (mp3 file) when a person is seen in the room. A basic theif using Python and OpenCV

SysKey 4 Nov 02, 2021
Brief idea about our project is mentioned in project presentation file.

Brief idea about our project is mentioned in project presentation file. You just have to run attendance.py file in your suitable IDE but we prefer jupyter lab.

Dhruv ;-) 3 Mar 20, 2022
CellProfiler is a open-source application for biological image analysis

CellProfiler is a free open-source software designed to enable biologists without training in computer vision or programming to quantitatively measure phenotypes from thousands of images automaticall

CellProfiler 732 Dec 23, 2022