Centralized whale instance using github actions, sourcing metadata from bigquery-public-data.

Overview

Whale Demo Instance: Bigquery Public Data

This is a fully-functioning demo instance of the whale data catalog, actively scraping data from Bigquery's public project bigquery-public-data using github actions.

To test out this repo with your own local installation of whale (i.e. to emulate what it'd be like to set up whale on github for your own team), clone the repo to your ~/.whale directory (if you already have a ~/.whale directory, move it or delete it with rm -rf ~/.whale or the clone won't work):

git clone https://github.com/dataframehq/whale-bigquery-public-data ~/.whale

Then install whale and run the following commands, following the prompts:

wh git-enable
wh schedule

At this point, if you run wh pull, it should run a git pull --autostash --rebase against this repo (meaning any locally scheduled cron jobs will simply pull down fresh metadata from this repo, rather than scraping directly from Bigquery).

For more information on how to set this up for your own warehouse, see the docs.

FAQ

Why doesn't wh.pull() work locally?

While wh pull (the CLI hook) will check for a flag in wh config and act appropriately (sourcing from github if is_git_etl_enabled is True, and from the connections in wh connections if not), wh.pull() (the python hook) performs no such check. This is by design, to ensure the remote repository's associated CI/CD pipelines to pull down data directly from the metadata source, by default (we suspect most people do not want to refresh metadata using the python client, but feel free to open an issue if you disagree).

Locally, this will fail, unless you modify the key_path value specified in wh connections to some credentials you have stored locally. If you choose to do this, ensure you have the following permissions enabled in the associated service account: BigQuery Data Viewer, BigQuery Job User, BigQuery Metadata Viewer.

Owner
Hyperquery
All-in-one workspace for data analytics
Hyperquery
Quack-SMS-BOMBER - Quack Toolkit By IkigaiHack

Quack Toolkit By IkigaiHack About Quack Toolkit Quack Toolkit is a set of tools

Marcel 2 Aug 19, 2022
Qbittorrent / Aria2 Mirror & Leech Telegram Bot

This is a Telegram Bot written in Python for mirroring files on the Internet to your Google Drive or Telegram. Based on python-aria-mirror-bot Feature

Hüzünlü Artemis [HuzunluArtemis] 81 Jul 15, 2022
A simple Python script using Telethon to log all (or some) messages a user or bot account can see on Telegram.

telegram-logger A simple Python script using Telethon to log all (or some) messages a user or bot account can see on Telegram. Requirements Python 3.6

Richard 13 Oct 06, 2022
Protect Discord server invite link

DiscordOauth2Join Protect discord server invite links! Setup I will not help setting up the discord application, but just python. First, install the r

ZEEE 4 Aug 12, 2021
Бот для скачивания треков с Deezer используя ISRC и UPC коды

deez_robot Запуск Установите необходимые библиотеки pip install -r requirements.txt Создайте файл config.py и поместите туда токен бота и ARL-токен De

Max 4 Jul 31, 2022
Gnosis-py includes a set of libraries to work with Ethereum and Gnosis projects

Gnosis-py Gnosis-py includes a set of libraries to work with Ethereum and Gnosis projects: EthereumClient, a wrapper over Web3.py Web3 client includin

Gnosis 93 Dec 23, 2022
Telegram Group Calls Streaming bot with some useful features, written in Python with Pyrogram and Py-Tgcalls. Supporting platforms like Youtube, Spotify, Resso, AppleMusic, Soundcloud and M3u8 Links.

Yukki Music Bot Yukki Music Bot is a Powerful Telegram Music+Video Bot written in Python using Pyrogram and Py-Tgcalls by which you can stream songs,

Team Yukki 996 Dec 28, 2022
A tool written in Python used to instalock agents in VALORANT using the local API.

Valorant Instalock Tool v2.1.0 by Mr. SOSA A tool written in Python used to instalock agents in VALORANT using the local API. This is NOT a hotkey pro

Mr. SOSA 3 Nov 18, 2021
A Discord bot to play bluffing games like Dobbins or Bobbins

Usage: pip install -r requirements.txt python3 bot.py DISCORD_BOT_TOKEN Gameplay: All commands are case-insensitive, with trailing punctuation and spa

4 May 27, 2022
An async-ready Python wrapper around FerrisChat's API.

FerrisWheel An async-ready Python wrapper around FerrisChat's API. Installation Instructions Linux: $ python3.9 -m pip install -U ferriswheel Python 3

FerrisChat 8 Feb 08, 2022
Discord-shell - Reverse shell accessible via discord.py bot

Discord-shell reverse shell on discord bot. (STILL IN DEVELOPMENT) Installation

Skultz 6 Oct 21, 2022
A Python library for the Docker Engine API

Docker SDK for Python A Python library for the Docker Engine API. It lets you do anything the docker command does, but from within Python apps – run c

Docker 6.1k Jan 03, 2023
A simple python bot that serves to send some notifications about GitHub events to Slack.

github alerts slack bot 🤖 What is it? 🔍 This is a simple bot that serves to send some notifications about GitHub events to Slack channels. These are

Jackson Alves 10 Dec 10, 2022
Python API for British Geological Survey magnetic field calculator

Magnetic field calculator Python API for British Geological Survey magnetic field calculator. Description This project magnetic field calculator. It u

Filip Š 3 Mar 11, 2022
NiceHash Python Library and Command Line Rest API

NiceHash Python Library and Command Line Rest API Requirements / Modules pip install requests Required data and where to get it Following data is nee

Ashlin Darius Govindasamy 2 Jan 02, 2022
Automatic generation of crypto-arts based on image layers

NFT Generator Автоматическая генерация крипто-артов на основе слоев изображения. Установка pip3 install -r requirements.txt rm -rf result/* Как это ра

Zproger 31 Dec 29, 2022
Subtitle Translater

Subtitle Translater

OshadhaVimukthi 2 Nov 29, 2021
Maintained Fork of Jishaku For nextcord

Onami a debugging and utility extension for nextcord bots Read the documentation online. Fork Onami is a actively maintained fork of Jishaku for nextc

RPS 11 Dec 14, 2022
Telegram bot for logistic - Telegram bot for logistic

Демонстрационный телеграм-бот для нужд транспортной компании Цель проекта Реализ

M1chigun 1 Feb 05, 2022
Pure Python 3 MTProto API Telegram client library, for bots too!

Telethon ⭐️ Thanks everyone who has starred the project, it means a lot! Telethon is an asyncio Python 3 MTProto library to interact with Telegram's A

LonamiWebs 7.3k Jan 01, 2023