Centralized whale instance using github actions, sourcing metadata from bigquery-public-data.

Overview

Whale Demo Instance: Bigquery Public Data

This is a fully-functioning demo instance of the whale data catalog, actively scraping data from Bigquery's public project bigquery-public-data using github actions.

To test out this repo with your own local installation of whale (i.e. to emulate what it'd be like to set up whale on github for your own team), clone the repo to your ~/.whale directory (if you already have a ~/.whale directory, move it or delete it with rm -rf ~/.whale or the clone won't work):

git clone https://github.com/dataframehq/whale-bigquery-public-data ~/.whale

Then install whale and run the following commands, following the prompts:

wh git-enable
wh schedule

At this point, if you run wh pull, it should run a git pull --autostash --rebase against this repo (meaning any locally scheduled cron jobs will simply pull down fresh metadata from this repo, rather than scraping directly from Bigquery).

For more information on how to set this up for your own warehouse, see the docs.

FAQ

Why doesn't wh.pull() work locally?

While wh pull (the CLI hook) will check for a flag in wh config and act appropriately (sourcing from github if is_git_etl_enabled is True, and from the connections in wh connections if not), wh.pull() (the python hook) performs no such check. This is by design, to ensure the remote repository's associated CI/CD pipelines to pull down data directly from the metadata source, by default (we suspect most people do not want to refresh metadata using the python client, but feel free to open an issue if you disagree).

Locally, this will fail, unless you modify the key_path value specified in wh connections to some credentials you have stored locally. If you choose to do this, ensure you have the following permissions enabled in the associated service account: BigQuery Data Viewer, BigQuery Job User, BigQuery Metadata Viewer.

Owner
Hyperquery
All-in-one workspace for data analytics
Hyperquery
a translator bot for discord

TranslatorBOT it is a simple and powerful discord bot, it been used for translating includes more than 100 language, it has a lot of integrated comman

Mear. 2 Feb 03, 2022
Discord Remote Administration Tool fully written in Python3.

DiscordRAT Discord Remote Administration Tool fully written in Python3. This is a RAT controlled over Discord with over 50 post exploitation modules.

hozzywozzy 2 Feb 06, 2022
Nautobot-custom-jobs - Custom jobs for Nautobot

nautobot-custom-jobs This repo contains custom jobs for Nautobot. Installation P

Dan Peachey 9 Oct 27, 2022
A discord token creator that uses the service capmonster for captcha solving!

Discord Token Creator A discord token creator that uses the service capmonster for captcha solving! Report Bug · Request Feature Features Autojoin dis

dropout 41 Oct 25, 2021
Telegram Url Upload Bot With Same more Features ✨

Telegram Url Upload Bot With Same more Features ✨

Group Dc Bots 4 Feb 12, 2022
💖 Telegram - Telethon - UserBot 💖

『᭙ꪖ᥅ƺẞø†』 🇮🇳 ⚡ ᭙ꪖ᥅ƺBot Is One Of The Fastest & Smoothest Bot On Telegram Based on Telethon ⚡ Status Of Bot Telegram 🏪 YouTube 📺 Dєρℓογ το нєяοκυ D

Team WarZ 1 Mar 28, 2022
Simple tool to gather domains from crt.sh using the organization name

Domain Collector: _ _ ___ _ _ _ __| | ___ _ __ ___ __ _(_)_ __ / __\___ | |

Cyber Guy 63 Dec 24, 2022
Get-Phone-Number-Details-using-Python - To get the details of any number, we can use an amazing Python module known as phonenumbers.

Get-Phone-Number-Details-using-Python To get the details of any number, we can use an amazing Python module known as phonenumbers. We can use the amaz

Coding Taggers 1 Jan 01, 2022
A smooth and powerful Telegram Userbot made to make Telegram easier.

| Xᴇɴᴏ Bᴏᴛ Is One Of The Fastest & Smoothest Bot On Telegram Based on Telethon|

SimpleBoy 1 Dec 01, 2021
Códigos pela Força Bruta e Algoritmo Genético para o Problema da Mochila

O problema da mochila é um problema de optimização combinatória. O nome dá-se devido ao modelo de uma situação em que é necessário preencher uma mochi

Hemili Beatriz 1 Jan 08, 2022
A suite of utilities for AWS Lambda Functions that makes tracing with AWS X-Ray, structured logging and creating custom metrics asynchronously easier

A suite of utilities for AWS Lambda Functions that makes tracing with AWS X-Ray, structured logging and creating custom metrics asynchronously easier

Amazon Web Services - Labs 1.9k Jan 07, 2023
Nowadays we don't have time to listen to each and every song that we come across in a playlist.

Nowadays we don't have time to listen to each and every song that we come across in a playlist. so, this project helps you. we used Spotify API for collecting the dataset information and able to do E

3 Apr 26, 2022
A Python Library to interface with Flickr REST API, OAuth & JSON Responses

Python-Flickr Python-Flickr is A Python library to interface with Flickr REST API & OAuth Features Photo Uploading Retrieve user information Common Fl

Mike Helmick 40 Sep 25, 2021
Provide discord buttons feature for discord.py

dpy_buttons wrapper library for discord.py, providing discord buttons feature. Future of the library Will be merged into discord interaction api libra

Minjun Kim (Lapis0875) 17 Feb 02, 2022
Telegram bot for stream music or video on telegram

Anonymous VC Bot + Stream Bot Telegram bot for stream music or video on telegram, powered by PyTgCalls and Pyrogram Features Playlist features Multi L

Anonymous Boy 111 Oct 04, 2022
Written in Python, freezed into stand-alone executable with PyInstaller. This app will make sure you stay in New World without getting kicked for inactivity.

New World - AFK Written in Python, freezed into stand-alone executable with PyInstaller. This app will make sure you stay in New World without getting

Rodney 5 Oct 31, 2021
A Discord bot written in Python to help with guild administration

forgotten-hydra A Discord bot written in Python to help with guild administration. External libraries Pycord-Development/pycord 1.7.3 djc/couchdb-pyth

1 May 14, 2022
Telegram to TamTam stickers

Telegram to TamTam stickers @tg_stickers TamTam бот, который конвертирует Telegram стикеры в формат TamTam и помогает загрузить их в TamTam. Все делае

Ivan Buymov 22 Nov 01, 2022
SC4.0 - BEST EXPERIENCE · HEX EDITOR · Discord Nuker · Plugin Adder · Cheat Engine

smilecreator4 This site is for people who want to hack or want to learn it! Furthermore, this program does not work without turning off Antivirus or W

1 Jan 04, 2022
Scheduled Block Checker for Cardano Stakepool Operators

ScheduledBlocks Scheduled Block Checker for Cardano Stakepool Operators Lightweight and Portable Scheduled Blocks Checker for Current Epoch. No cardan

SNAKE (Cardano Stakepool) 4 Oct 18, 2022