It's important that Zoobot has proper benchmarks so that we can be confident new releases work properly for users. This PR adds those benchmarks.

In the course of setting up the benchmarks, I have made some major changes/improvements:

• pytorch-galaxy-datasets refactored to work for tensorflow, imports adapted
• both tensorflow and pytorch zoobot versions use albumentations for augmentations. Old TF code removed.
• tensorflow version bumped to 2.10 (current latest) while I'm at it
• pytorch version now has logging for per-question loss. Loss func aggregation has new option to support this.
• TensorFlow version has per-question logging also, but awaiting issue with Keras team to enable
• Created minimal_example.py for TensorFlow (thanks, @katgre )
• Support CPU-only PyTorch training
• Refactor TF TrainingConfig to Trainer object, Lightning style, for consistency

Unclear what's causing this slowness. Presumably a callback I added - but none look like they should be heavy? Perhaps something wandb is doing?

• Remove all callbacks and rerun
• Remove wandb and rerun For each, check if slow warning continues (or if training speed changes at all)

Related to https://github.com/mwalmsley/zoobot/issues/18#issuecomment-1278635788

This PR adds an auto CI release mechanism for publishing zoobot to pypi. It uses the GH action to release to pypi https://github.com/pypa/gh-action-pypi-publish

A question rather than a request. Are there any plans to publish the refactored work ?

PyPi shows v0.0.1 is published https://pypi.org/project/zoobot/#history on 15th March 2021 but the latest code is ~v0.0.3 (tags) and the refactor seems to be working well.

Ideally I can pull in these packages to my own env / container and then train with the latest code vs pulling in from github etc.

https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/managing-a-branch-protection-rule

It may be too restrictive for your use case / dev flows but we use this for contributor PRs etc. Basically we ensure that a PR meets certain criteria in terms of our CI runs, can only merge a PR once one of the CI runs v3.7 or v3.9 tests pass.

Feel free to close if you don't think this is useful.

TFRecords are cumbersome and take up a lot of disk space. It's much simpler to learn directly from images on disk, at the cost of some I/O performance.

This PR removes support for TFRecords in favour of images-on-disk. This will ultimately enable new TensorFlow weights trained on all of DESI (impractical with TFRecords).

Breaking change for anyone using TFRecords (i.e. everyone using TensorFlow to train from scratch). Finetuning should not be affected.

TODO - will require new greyscale/colour pretrained models, just for safety.

This PR proposes to add a simple GH Action script that establishes a python environment, downloads the requirements and runs pytest.

Some other things to consider might be to use conda for virtual environments and creating CI scripts for Docker as well.

This PR changes the docker / compose setup, specifically it

• consolidates the docker files to cuda and tensorflow base images (no need for a python base image)
• adds a .dockerignore entry for all data files when building the container to keep the size down
• and provides an easy way to inject them at run time via local directory mounts in the compose file
• finally this removes specific to my machine local directory setup for injecting unrelated data files

Doing some polishing on finetuning

• Add wandb logging to the full_tree example. @camallen use this for dashboard. You will need to add import wandb, wandb.init(authkey, etc) just before when running on Azure.
• Freeze batch norm layers by default when finetuning, with new recursive function
• Pass additional params via config (thanks Cam)
• Minor cleanup

Key change is adding pytorch.training.finetune() method. Works on either classification (e.g. 0, 1) data or count (e.g. 12 said yes, 4 said no) data.

Includes three working examples:

• Binary classification, with tiny rings subset
• Counts for single question, with full internal rings data
• Counts for all questions, with GZ Cosmic Dawn schema

Also updates various imports for the galaxy-datasets refactor, fixes prediction method to work on unlabelled data, minor QoL improvements.

Finally, changes PyTorch dense layer initialisation to custom high-uncertainty initialisation - see efficientnet_custom.py

cc @camallen

Adds support (minimal working examples, a guide) for calculating new representations with a trained model.

Also adds significant new features:

• Distributed training with several GPUs
• Metric logging with Weights&Biases (add your own login credentials)
• Train on color (3-band) images, not just greyscale

Also adds a critical bugfix (when loading images for direct predictions i.e. not via TFRecords, correctly normalise to the 0-1 interval expected (without documentation) by the tf.keras.experimental.preprocessing layers).

Also adds misc. minor fixes and documentation tweaks.

This code was used for the morphology tools paper (to be submitted shortly).

It should be possible to search for non-standard package repositories using just setup.py, without having the user also set --extra-index-url.

https://setuptools.pypa.io/en/latest/deprecated/dependency_links.html

But I couldn't get this to work on a quick try.

I tried to go through finetune_binary_classification.py, but got the error:

ImportError: cannot import name 'finetune' from 'zoobot.pytorch.training' (/usr/local/lib/python3.8/dist-packages/zoobot/pytorch/training/init.py)

I tried it both with kasia and dev branch, went through "git clone" and "pip install" (I remembered there were some issues during Hackaton regarding that), also tried to import other features from the folder (i.e. losses) and it worked fine.

Instead of using on of the complicated decision trees from decals dr5, let's create a simple decision tree with one dependency already written in the minimal_example.py.