I come across this repo and found it very similar to our API, though much more mature. https://github.com/Glacier-Ice/data-sci-api

we have problems in creating a standard of dataset collection and API documentation for end-users

is there a way we can collaborate?

The current search result shows too much information. It would be good to format the result into a way that is easier to read and get the id if needed.

FuzzyWuzzy is GPLv2 licensed which would force you to licence the whole project under GPLv2. I had the same problem on one of my projects and so I wrote rapidfuzz which is implementing the same algorithm but is based on a version of fuzzywuzzy that was MIT Licensed and is therefor MIT Licensed aswell, so it can be used in here without forcing a License change. As a nice bonus it is fully implemented in C++ and comes with a few Algorithmic improvements making it faster than FuzzyWuzzy.

Is it better to have one file for each herb in flora?

Situition

Currently, the flora is defined in a single json file.

• It becomes hard to read. This is not fitting into the human-readable principle.
• It becomes hard to manage. We are currently sorting everything in the big file. When we have a problem, the whole flora will be unusable.

Solution

Use separate files for herbs.

Simply Copy dataherb.json

• Copy dataherb.json to workdir/{id}/dataherb.json or {id}.json will work.

  • Using folders allows us to put in more files. For example, we can take datapackage content out to make it more managable.

• Build the flora from all these files.

• [x] Implement this new structure.

Ready for a Demo repo of flora

In this way, we can put up a repo for open datasets easily and allow users to add more easily.

Possible creating process

• Create package directly on GitHub by uploading the dataherb.json file.

  • But there should be a validation process to avoid duplicate id.

• [ ] Setup a demo repo as demo flora.

This is a completely new era of Dataherb.

New Stuff

• Supporting S3 as source
• Serve whole flora as webpages with search
• User config for flora
• Multiple flora on one machine

We also redesigned the core.

We don't only upload datasets, we might also want to load datasets from remote.

Here we propose to add the option to add datasets using the URL.

• Build a Herb from remote data
• Option to add metadata only or download everything.
  • Adding metadata only will only add data to the flora
  • Thus we can not find the dataset folder with the corresponding id.
  • This can be used to decide if a dataset is metadata only or fully downloaded.

Managing flora using command line

Version control of the flora is not really hard. We just get into the folder and use git.

But it would be much easier if we can simply run dataherb sync flora

Approaches:

• Invoking command line: ref cookiecutter