Solution to the Weather4cast 2021 challenge

Overview

This code was used for the entry by the team "antfugue" for the Weather4cast 2021 Challenge. Below, you can find the instructions for generating predictions, evaluating pre-trained models and training new models.

Installation

To use the code, you need to:

  1. Clone the repository.
  2. Setup a conda environment. You can find an environment verified to work in the environment.yml file. However, you might have to adapt it to your own CUDA installation.
  3. Fetch the data you want from the competition website. Follow the instructions here. The data should should be in the data directory following the structure specified here.
  4. (Optional) If you want to use the pre-trained models, load them from https://doi.org/10.5281/zenodo.5101213. Place the .h5 files in the models/best directory.

Running the code

Go to the weather4cast directory. There you can either launch the main.py script with instructions provided below, or launch an interactive prompt (e.g. ipython) and then import modules and call functions from them.

Reproducing predictions

Run:

python main.py submit --comp_dir=w4c-core-stage-1 --submission_dir="../submissions/test"

where you can change --comp_dir to indicate which competition you want to create predictions for (these correspond to the directory names in the data directory) and --submission_dir to indicate where you want to save the predictions.

This script automatically loads the best model weights corresponding to the "V4pc" submission that produced the best scores on the leaderboards. To experiment with other weights, see the function combined_model_with_weights in models.py and the call to that in main.py. You can change the combination of models and weights with the argument var_weights in combined_model_with_weights.

Generating the predictions should be possible in a reasonable time also on a CPU.

Evaluate pre-trained model

python main.py train --comp_dir=w4c-core-stage-1 --model=resgru --weights="../models/best/resrnn-temperature.h5" --dataset=CTTH --variable=temperature

This example trains the ResGRU model for the temperature variable, loading the pre-trained weights from the --weights file. You can change the model and the variable using the --model, --weights, --dataset and --variable arguments.

A GPU is recommended for this although in principle it can be done on a CPU.

Train a model

python main.py train --comp_dir="w4c-core-stage-1" --model="resgru" --weights=model.h5 --dataset=CTTH --variable=temperature

The arguments are the same as for evaluate except the --weights parameter indicates instead the weights file that the training process keeps saving in the models directory.

A GPU is basically mandatory. The default batch size is set to 32 used in the study but you may have to reduce it if you don't have a lot of GPU memory.

Hint: It is not recommended to train like this except for demonstration purposes. Instead I recommend you look at how the train function in main.py works and follow that in an interactive prompt. The batch generators batch_gen_train and batch_gen_valid are very slow at first but get faster as they cache data. Once the cache is fully populated they will be much faster. You can avoid this overhead by pickling a fully loaded generator. For example:

import pickle

for i in range(len(batch_gen_train)):
    batch_gen_train[i] # fetch all batches

with open("batch_gen_train.pkl", 'wb') as f:
    pickle.dump(batch_gen_train, f)
Owner
Jussi Leinonen
Data scientist working on Atmospheric Science problems
Jussi Leinonen
Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) in PyTorch

alias-free-gan-pytorch Unofficial implementation of Alias-Free Generative Adversarial Networks. (https://arxiv.org/abs/2106.12423) This implementation

Kim Seonghyeon 502 Jan 03, 2023
Collection of machine learning related notebooks to share.

ML_Notebooks Collection of machine learning related notebooks to share. Notebooks GAN_distributed_training.ipynb In this Notebook, TensorFlow's tutori

Sascha Kirch 14 Dec 22, 2022
Tensorflow Repo for "DeepGCNs: Can GCNs Go as Deep as CNNs?"

DeepGCNs: Can GCNs Go as Deep as CNNs? In this work, we present new ways to successfully train very deep GCNs. We borrow concepts from CNNs, mainly re

Guohao Li 612 Nov 15, 2022
U-Net: Convolutional Networks for Biomedical Image Segmentation

Deep Learning Tutorial for Kaggle Ultrasound Nerve Segmentation competition, using Keras This tutorial shows how to use Keras library to build deep ne

Yihui He 401 Nov 21, 2022
Revisiting Temporal Alignment for Video Restoration

Revisiting Temporal Alignment for Video Restoration [arXiv] Kun Zhou, Wenbo Li, Liying Lu, Xiaoguang Han, Jiangbo Lu We provide our results at Google

52 Dec 25, 2022
DeepDiffusion: Unsupervised Learning of Retrieval-adapted Representations via Diffusion-based Ranking on Latent Feature Manifold

DeepDiffusion Introduction This repository provides the code of the DeepDiffusion algorithm for unsupervised learning of retrieval-adapted representat

4 Nov 15, 2022
Advantage Actor Critic (A2C): jax + flax implementation

Advantage Actor Critic (A2C): jax + flax implementation Current version supports only environments with continious action spaces and was tested on muj

Andrey 3 Jan 23, 2022
Implementations of LSTM: A Search Space Odyssey variants and their training results on the PTB dataset.

An LSTM Odyssey Code for training variants of "LSTM: A Search Space Odyssey" on Fomoro. Check out the blog post. Training Install TensorFlow. Clone th

Fomoro AI 95 Apr 13, 2022
Classify music genre from a 10 second sound stream using a Neural Network.

MusicGenreClassification Academic research in the field of Deep Learning (Deep Neural Networks) and Sound Processing, Tel Aviv University. Featured in

Matan Lachmish 453 Dec 27, 2022
ColossalAI-Benchmark - Performance benchmarking with ColossalAI

Benchmark for Tuning Accuracy and Efficiency Overview The benchmark includes our

HPC-AI Tech 31 Oct 07, 2022
[ICCV 2021] Our work presents a novel neural rendering approach that can efficiently reconstruct geometric and neural radiance fields for view synthesis.

MVSNeRF Project page | Paper This repository contains a pytorch lightning implementation for the ICCV 2021 paper: MVSNeRF: Fast Generalizable Radiance

Anpei Chen 529 Dec 30, 2022
Studying Python release adoptions by looking at PyPI downloads

Analysis of version adoptions on PyPI We get PyPI download statistics via Google's BigQuery using the pypinfo tool. Usage First you need to get an acc

Julien Palard 9 Nov 04, 2022
DNA sequence classification by Deep Neural Network

DNA sequence classification by Deep Neural Network: Project Overview worked on the DNA sequence classification problem where the input is the DNA sequ

Mohammed Jawwadul Islam Fida 0 Aug 02, 2022
Official code for the paper "Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks".

Why Do Self-Supervised Models Transfer? Investigating the Impact of Invariance on Downstream Tasks This repository contains the official code for the

Linus Ericsson 11 Dec 16, 2022
Additional functionality for use with fastai’s medical imaging module

fmi Adding additional functionality to fastai's medical imaging module To learn more about medical imaging using Fastai you can view my blog Install g

14 Oct 31, 2022
Code for the paper "JANUS: Parallel Tempered Genetic Algorithm Guided by Deep Neural Networks for Inverse Molecular Design"

JANUS: Parallel Tempered Genetic Algorithm Guided by Deep Neural Networks for Inverse Molecular Design This repository contains code for the paper: JA

Aspuru-Guzik group repo 55 Nov 29, 2022
Trajectory Variational Autoencder baseline for Multi-Agent Behavior challenge 2022

MABe_2022_TVAE: a Trajectory Variational Autoencoder baseline for the 2022 Multi-Agent Behavior challenge This repository contains jupyter notebooks t

Andrew Ulmer 15 Nov 08, 2022
torchsummaryDynamic: support real FLOPs calculation of dynamic network or user-custom PyTorch ops

torchsummaryDynamic Improved tool of torchsummaryX. torchsummaryDynamic support real FLOPs calculation of dynamic network or user-custom PyTorch ops.

Bohong Chen 1 Jan 07, 2022
Relative Uncertainty Learning for Facial Expression Recognition

Relative Uncertainty Learning for Facial Expression Recognition The official implementation of the following paper at NeurIPS2021: Title: Relative Unc

35 Dec 28, 2022
Easy and Efficient Object Detector

EOD Easy and Efficient Object Detector EOD (Easy and Efficient Object Detection) is a general object detection model production framework. It aim on p

381 Jan 01, 2023