Added tqdm progress bar to RemoteStorage.push/pull methods

• First determines the total number of bytes to push/pull
• Updates the number of transferred bytes after every file

resolve #4 #5 #9

This PR fixes the test failure related to failed name resolution on CI. It makes the tests run inside a container because this is apparently needed to enable host name resolution for the minio service from within the tests.

You can refer to these links for more information:

• https://docs.github.com/en/actions/using-containerized-services/about-service-containers#mapping-docker-host-and-service-container-ports
• https://docs.github.com/en/actions/using-containerized-services/creating-redis-service-containers#running-jobs-in-containers

Add a new flag to the RemoteStorage push/pull operation. If True the function should determine and return the operations that need to be conducted without actually performing them.

Currently, pushing and pulling of directories does not check whether the entire operation can be performed successfully (e.g. if modified files already exist and overwrite_existing=False). This leads to a partial execution before an error is thrown and thus to an unpredictable state.

We should check if the entire operation can be performed before pushing/pulling anything.

Also, to be more familiar to git users, overwrite_existing should be renamed to force. This is a breaking change, the minor version should be bumped

@MischaPanch can you release the current dev branch? I found a bug in the old version which seems to fixed now. Would be great to get the fix installed. Not urgent though, I can work with the dev branch for now :)

We have essentially no documentation on how to use accsr. The interplay of storage and config modules should be demonstrated in notebooks. See tests/conftest.py for an example how a storage service is instantiated during local testing an in CI.

We should make it easier to push/pull a bunch of paths based on patterns. For that we should add

• [ ] Permit passing glob-patterns to push/pull
• [ ] Add the possibility to pass a regex as except_matches kwarg to permit simple exclusion of files. The current regex kwarg should be renamed to if_matches.

This would permit things like

storage.push("data/**/*.jpg", except_matches=r".*test.*") 

We could additionally allow passing a except_condition: Callable[[str]], bool] = None (or do you think if_condition is more natural?), in which case the above can be rewritten

storage.push("data/**/*.jpg", except_condition=lambda n: "test" in n) 

The condition could be even made more general, mapping the metadata-object to a bool (thereby e.g. allowing filtering by size), at the cost of a more complicated interface for callables. @fariedabuzaid @AnesBenmerzoug what do you think?

@MischaPanch This fixes the tests but it currently break caching.

WARNING: The directory '/github/home/.cache/pip' or its parent directory is not owned or is not writable by the current user. The cache has been disabled. Check the permissions and owner of that directory. If executing pip with sudo, you should use sudo's -H flag.

I think running jobs inside containers is the way to go and we should invest some time to make caching work with it.

Originally posted by @AnesBenmerzoug in https://github.com/appliedAI-Initiative/accsr/issues/1#issuecomment-976536009