SparkDataset 
Provides instant access to many datasets right from Pyspark (in Spark DataFrame structure).
Drop a star if you like the project.
What?
The idea is simple. There are various datasets available out there, but they are scattered in different places over the web. Is there a quick way (in Pyspark) to access them instantly without going through the hassle of searching, downloading, and reading ... etc? SparkDataset tries to address that question :)
Usage:
Start with importing data():
from sparkdataset import data
- To load a dataset:
titanic = data('titanic')
- To display the documentation of a dataset:
data('titanic', show_doc=True)
- To see the available datasets:
data()
- To search for datasets with terms
data('ab')
Did you mean:
crabs, abbey, Vocab
That's it.
Go to this notebook for a demonstration of the functionality
Why?
In R, there is a very easy and immediate way to access multiple statistical datasets, in almost no effort. All it takes is one line > data(dataset_name). This makes the life easier for quick prototyping and testing. Well, I am jealous that Pyspark does not have a similar functionality. Thus, the aim of sparkdataset is to fill that gap.
Currently, sparkdataset has about 757 (mostly numerical-based) datasets, that are based on RDatasets. In the future, I plan to scale it to include a larger set of datasets. For example,
- include textual data for NLP-related tasks, and
- allow adding a new dataset to the in-module repository.
Installation:
$ pip install sparkdataset
Uninstall:
$ pip uninstall sparkdataset$ rm -rf $HOME/.sparkdataset
Changelog
1.0.0
- Added search dataset by name similarity.
- Example:
>>> data('heat')
Did you mean:
Wheat, heart, Heating, Yeast, eidat, badhealth, deaths, agefat, hla, heptathlon, azt
- Added support to Windows.
Dependency:
- pandas
- pyspark :: 3.1.2
Miscellaneous:
- Tested on OSX and Linux (debian).
- Supports both Python 3 (3.8.8 and above).
TODO:
- add textual datasets (e.g. NLTK stuff).
- add samples generators.
Thanks to:
- RDatasets: R's datasets collection.
1 Nov 17, 2021
3 Oct 23, 2022
1 Oct 26, 2021
102 Nov 10, 2022
180 Dec 18, 2022
3 Jul 5, 2022
20 Jan 5, 2023
2 Feb 14, 2022
411 Dec 27, 2022
38 Dec 03, 2022
2 Feb 16, 2022
1 Nov 21, 2021
1 Jun 28, 2022
625 Jan 02, 2023
96 Jan 01, 2023
27 Dec 22, 2022
214 Jan 02, 2023
43 Nov 29, 2022
23 Dec 14, 2022
2 Jul 01, 2022
6 Oct 25, 2022
12 Jan 05, 2023
5 Dec 01, 2022
25 Dec 14, 2022
0 Jan 20, 2022
5 Sep 27, 2021
4 Jul 10, 2022
1 Feb 27, 2022