Crowd-Kit: Computational Quality Control for Crowdsourcing
Crowd-Kit is a powerful Python library that implements commonly-used aggregation methods for crowdsourced annotation and offers the relevant metrics and datasets. We strive to implement functionality that simplifies working with crowdsourced data.
Currently, Crowd-Kit contains:
- implementations of commonly-used aggregation methods for categorical, pairwise, textual, and segmentation responses
- metrics of uncertainty, consistency, and agreement with aggregate
- loaders for popular crowdsourced datasets
The library is currently in a heavy development state, and interfaces are subject to change.
Installing
Installing Crowd-Kit is as easy as pip install crowd-kit
Getting Started
This example shows how to use Crowd-Kit for categorical aggregation using the classical Dawid-Skene algorithm.
First, let us do all the necessary imports.
from crowdkit.aggregation import DawidSkene
from crowdkit.datasets import load_dataset
import pandas as pd
Then, you need to read your annotations into Pandas DataFrame with columns task, performer, label. Alternatively, you can download an example dataset.
df = pd.read_csv('results.csv') # should contain columns: task, performer, label
# df, ground_truth = load_dataset('relevance-2') # or download an example dataset
Then you can aggregate the performer responses as easily as in scikit-learn:
aggregated_labels = DawidSkene(n_iter=100).fit_predict(df)
Implemented Aggregation Methods
Below is the list of currently implemented methods, including the already available (
Categorical Responses
| Method | Status |
|---|---|
| Majority Vote | |
| Dawid-Skene | |
| Gold Majority Vote | |
| M-MSR | |
| Wawa | |
| Zero-Based Skill | |
| GLAD | |
| BCC | |
Textual Responses
| Method | Status |
|---|---|
| RASA | |
| HRRASA | |
| ROVER | |
Image Segmentation
| Method | Status |
|---|---|
| Segmentation MV | |
| Segmentation RASA | |
| Segmentation EM | |
Pairwise Comparisons
| Method | Status |
|---|---|
| Bradley-Terry | |
| Noisy Bradley-Terry | |
Citation
- Ustalov D., Pavlichenko N., Losev V., Giliazev I., and Tulin E. A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python. The Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track. HCOMP 2021. 2021. arXiv: 2109.08584 [cs.HC].
@inproceedings{HCOMP2021/CrowdKit,
author = {Ustalov, Dmitry and Pavlichenko, Nikita and Losev, Vladimir and Giliazev, Iulian and Tulin, Evgeny},
title = {{A General-Purpose Crowdsourcing Computational Quality Control Toolkit for Python}},
year = {2021},
booktitle = {The Ninth AAAI Conference on Human Computation and Crowdsourcing: Works-in-Progress and Demonstration Track},
series = {HCOMP~2021},
eprint = {2109.08584},
eprinttype = {arxiv},
eprintclass = {cs.HC},
url = {https://www.humancomputation.com/assets/wips_demos/HCOMP_2021_paper_85.pdf},
language = {english},
}
Questions and Bug Reports
- For reporting bugs please use the Toloka/bugreport page.
- Join our English-speaking slack community for both tech and abstract questions.
License
© YANDEX LLC, 2020-2021. Licensed under the Apache License, Version 2.0. See LICENSE file for more details.
23 Dec 21, 2022
25 Sep 06, 2022
9 Jul 05, 2022
444 Dec 30, 2022
19 Dec 22, 2022
53 Jul 17, 2022
40 Nov 30, 2022
268 Dec 24, 2022
76 Jan 03, 2023
694 Nov 23, 2022
6 Oct 25, 2022
3 Jul 14, 2021
3 Feb 06, 2022
13 Dec 08, 2022
8 Oct 27, 2022
14 Oct 21, 2022
391 Dec 27, 2022
74 Dec 26, 2022
2 Sep 15, 2022
15 Dec 16, 2022