A reference implementation for processing the content.log files found at opendata.dwd.de/weather

Related tags

Miscellaneousopendata-content.log-tool
Overview

content.log Tool

A reference implementation for processing the content.log files found at opendata.dwd.de/weather.

Example usage:

wildcards.txt ./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt"> 
wget https://opendata.dwd.de/weather/nwp/content.log.bz2
bunzip2 content.log.bz2
echo "./icon-d2/grib/*/t_2m/*" > wildcards.txt
./get_updated_files.py -w wildcards.txt 2022-01-26T03:00 > updated_files.txt

The produced file updated_files.txt will hold all pathnames as given by content.log that are updated since 2022-01-26 03:00:00 UTC according to the file's modification date. Remember that those paths are relative to the directory the content.log.bz2 was in, in this case https://opendata.dwd.de/weather/nwp/.

Also mind that there are multiple servers behind https://opendata.dwd.de which might not be exactly in sync with each other regarding file modification timestamps. Look into the code of get_updated_files.py for a suggestion on how to deal with that.

While this program relies on the file modification timestamp dumped into content.log.bz2, it might be more feasible to process the data reference time that is contained in the filenames instead.

$ ./get_updated_files.py --help
usage: get_updated_files.py [-h] [--content-log CONTENT_LOG_FILE] [--wildcards WILDCARDS_FILE] [--min-delta MIN_DELTA] [--version] UPDATED_SINCE

Filters paths of a DWD Open Data content.log file for entries that have been updated.

positional arguments:
  UPDATED_SINCE         Last time files were checked for updates

optional arguments:
  -h, --help            show this help message and exit
  --content-log CONTENT_LOG_FILE, -f CONTENT_LOG_FILE
                        The decompressed content.log file (default: content.log)
  --wildcards WILDCARDS_FILE, -w WILDCARDS_FILE
                        Filter results by a set of wildcards
  --min-delta MIN_DELTA, -d MIN_DELTA
                        Minimum number of seconds a file needs to be younger than UPDATED_SINCE (default: 60)
  --version             show program's version number and exit
You might also like...
Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

listenerPython Script de monitoramento das teclas do teclado, salvando todos os dados digitados em um arquivo de log juntamente com os dados de rede.

Vinícius Azevedo 4 Nov 27, 2022
Python Common things by Problem Fighter Library, (Exception, Debug Log, etc.)

In the name of God, the Most Gracious, the Most Merciful. PF-PY-Common Documentation Install and update using pip: pip install -U xxxx Please find the

Problem Fighter 3 Jan 15, 2022
A log likelihood fit for extracting neutrino oscillation parameters

A-log-likelihood-fit-for-extracting-neutrino-oscillation-parameters Minimised the negative log-likelihood fit to extract neutrino oscillation paramete

Vid Homsak 1 Jan 23, 2022
A Python Based Utility for Processing GST-Return JSON Files to Multiple Formats
A Python Based Utility for Processing GST-Return JSON Files to Multiple Formats

GSTR 1/2A Utility by Shan.tk Open Source GSTR 1/GSTR 2A JSON to Excel utility based on Python. Useful for Auditors in Verifying GSTR 1 Return Invoices

Sudharshan TK 1 Oct 8, 2022
A python script that changes your desktop background based on current weather and time of the day.

Desktop background wallpaper, based on current weather and time A python script that changes your computer's desktop background based on current weath

Maj Gaberšček 1 Nov 16, 2021
A simple weather app.

keather A simple weather app. This is currently not finished. Dependencies: yay -S python-beautifulsoup4 tk

null 1 Jan 9, 2022
Use a real time weather API to apply wind to your mouse cursor.

wind-cursor Use a real time weather API to apply wind to your mouse cursor. Requirements PyAutoGUI pyowm Usage This program uses the OpenWeatherMap AP

Andreas Schmid 1 Feb 7, 2022
Python package for reference counting native pointers
Python package for reference counting native pointers

refcount master: testing: This package is primarily for managing resources in native libraries, written for instance in C++, from Python. While it boi

CSIRO Hydroinformatics 2 Nov 3, 2022
Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters.
Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters.

Project Faros Project Faros is a reference implimentation of Red Hat OpenShift 4 on small footprint, bare-metal clusters. The project includes referen

project: Faros 9 Jul 18, 2022
Comments
  • changing urljoin to os.path.join

    changing urljoin to os.path.join

    The behavior of urljoin is on my opionion not constructive, because of its description "Construct a full (“absolute”) URL by combining a “base URL” (base) with another URL (url). Informally, this uses components of the base URL, in particular the addressing scheme, the network location and (part of) the path, to provide missing components in the relative URL." (https://docs.python.org/3/library/urllib.parse.html#urllib.parse.urljoin)

    opened by nklever 0
Releases(v2.0.0)

  • v2.0.0(Jan 27, 2022)

    This new major release introduces support for timezones. Also, the deprecated feature --wildcards was removed and instead the use of external tools like bzgrep on the compressed file content.log.bz2 are recommended. The content.log file was promoted to the

    Changes

    • CONTENT_LOG_FILE is now a positional argument which defaults to STDIN when omitted
    • --updated-since is now a required option

    New features

    • supply multiple CONTENT_LOG_FILEs as positional arguments to parse their combined content
    • --updated-since now supports timezones when specified as +HH:MM while assuming the system's time zone if not specified; date-times in content.log are always treated as UTC
    Source code(tar.gz)
    Source code(zip)

  • v1.1.0(Jan 27, 2022)

  • v1.0.0(Jan 27, 2022)

    This is the first fully functional version. It implements minimal features with focus on readability to also act as a form of documentation around the quirks around DWD's Open Data emerging from having two seperate servers sitting behind opendata.dwd.de.

    Features

    • List the paths contained in a content.log file for entries that have likely been updated since a given date-time
    • Filter those results by one or more --wildcards
    Source code(tar.gz)
    Source code(zip)
Owner
Deutscher Wetterdienst (DWD)
Deutscher Wetterdienst (DWD)
GitHub Repository
A person does not exist image bot

A person does not exist image bot

Fayas Noushad 3 Dec 12, 2021
This Open-Source project is great for sensor capture and storage solutions.

Phase 1 This project helps developers in the creation of extended realities that communicate with Arduino and require the security of blockchain stora

Wolfberry, LLC 10 Dec 28, 2022
Sudoku solver using backtracking

Sudoku solver Sudoku solver using backtracking Basically in sudoku, we want to be able to solve a sudoku puzzle given an input like this, which repres

Kylie 99 Jan 07, 2023
This is a far more in-depth and advanced version of "Write user interface to a file API Sample"

Fusion360-Write-UserInterface This is a far more in-depth and advanced version of "Write user interface to a file API Sample" from https://help.autode

4 Mar 18, 2022
Functional collections extension functions for Python

pyfuncol pyfuncol Installation Usage API Documentation Compatibility Contributing License A Python functional collections library. It extends collecti

Andrea Veneziano 32 Nov 16, 2022
Dot Browser is a privacy-conscious web browser with smarts built-in for protection against trackers and advertisments online.

🌍 Take back your privacy with Dot Browser, the privacy-conscious web browser that protects you from being tracked and monitored online.

Dot HQ 1k Jan 07, 2023
Whole-day timezone comparison

Timezone Converter Compare a full day of your local timezone with foreign ones $ timezone-converter tijuana --zone $ timezone-converter tijuana new_yo

Iago Alonso 12 Nov 24, 2022
A fast python implementation of DTU MVS 2014 evaluation

DTUeval-python A python implementation of DTU MVS 2014 evaluation. It only takes 1min for each mesh evaluation. And the gap between the two implementa

82 Dec 27, 2022
A Python program that generates a maze that solves itself using DFS

Maze Generator And Solver Program Purpose: Generates a maze that then solves itself Language: Python and Pygame Algorithm: Randomized DFS / Floodfill

Joshua Liu 1 Jul 25, 2022
A inspector to be able to view and edit Qt style sheet while an application is running

Qt Style Sheet Inspector An inspector widget to view and modify the style sheet of a Qt app at runtime. Usage In order to use the inspector widget on

ESSS 46 Dec 10, 2022
token vesting escrow with cliff and clawback

Yearn Vesting Escrow A modified version of Curve Vesting Escrow contracts with added functionality: An escrow can have a start_date in the past.

62 Dec 08, 2022
A program that analyzes data from inertia measurement units installeed in aircraft and generates g-exceedance curves

A program that analyzes data from inertia measurement units installeed in aircraft and generates g-exceedance curves

Pooya 1 Nov 23, 2021
Python Create Your Own Tool Series

Python Create Your Own Tool Series Hey there! This is an additional Github repository that contains the final product files for each video in my Youtu

Joe Helle 21 Dec 02, 2022
Gmvault: Backup and restore your gmail account

Gmvault: Backup and restore your gmail account Gmvault is a tool for backing up your gmail account and never lose email correspondence. Gmvault is ope

Guillaume Aubert 3.5k Jan 01, 2023
Mnemosyne: efficient learning with powerful digital flash-cards.

Mnemosyne: Optimized Flashcards and Research Project Mnemosyne is: a free, open-source, spaced-repetition flashcard program that helps you learn as ef

359 Dec 24, 2022
Multifunctional Analysis of Regions through Input-Output

MARIO Multifunctional Analysis of Regions through Input-Output. (Documents) What is it MARIO is a python package for handling input-output tables and

14 Dec 25, 2022
The repository for AnyMacro: a Fusion360 Add-In

AnyMacro AnyMacro is an Autodesk® Fusion 360™ add-in for chaining multiple commands in a row to form Macros. Macros are created from a set of commands

1 Jan 07, 2022
CRC Reverse Engineering Tool in Python

CRC Beagle CRC Beagle is a tool for reverse engineering CRCs. It is designed for commnication protocols where you often have several messages of the s

Colin O'Flynn 51 Jan 05, 2023
Usando Multi Player Perceptron e Regressão Logistica para classificação de SPAM

Relatório dos procedimentos executados e resultados obtidos. Objetivos Treinar um modelo para classificação de SPAM usando o dataset train_data. Class

André Mediote 1 Feb 02, 2022
Repo with data from local elections in Portugal from 2009 to 2013

autarquicas - local elections in Portugal Repo with data from local elections in Portugal from 2009 to 2013 Objective To provide, to all, raw data fro

Jorge Gomes 6 Apr 06, 2022