Optimus's transformation plugins are implementations of Task and Hook interfaces that allows execution of arbitrary jobs in optimus.
brew tap odpf/taps
brew install optimus-plugins-odpf
curl -sL ${PLUGIN_RELEASE_URL} | tar xvz
chmod +x optimus-*
mv optimus-* /usr/bin/
Hello, Currently, for any query, we try to find the dependancy and ignoredependancy with FindDependenciesWithRegex and then we again pull the Refereced table with big query dry run.
If query contains view which is marked with /* @ignoreupstream */ helper, then ignoredependancy will contain the view name but not the table referenced by view.
The change here is to revise ignoredependancy list with table referenced by view.
I kept the loop execution in sequential manner, please let me know if should add concurrency here
enhancementWe have a query referencing to table as well as view.
select * from proj.dataset.table t1 left join proj.dataset.view v1 on t1.date = v1.date and t1.id = v1.id
proj.dataset.table then it correctly ignores to create upstream dependancy for this table.proj.dataset.view ( note the view query refers to 2 more tables ) then it does not ignore view or table referenced by view.As per Optimus PR, the executor boot process is standardised and maintained at optimus. Plugin devs need no longer have to wrap the executor image. closes odpf/optimus#405
Hi team,
I have a bq2bq job with window configuration
window:
size: 720h
offset: -48h
truncate_to: M
I expect to have transformation for date 01 to last day of the month, e.g on April, I expect got transformation from date 01 - 30. but currently only got transformation from date 01 - 29
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-26 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-27 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-28 00:00:00+00:00
[2022-06-13 15:08:12,323] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: create transformation for partition: 2022-04-29 00:00:00+00:00
[2022-06-13 15:08:12,324] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: start transformation job
[2022-06-13 15:08:12,324] {pod_launcher.py:149} INFO - [2022-06-13 15:08:12] INFO:bumblebee.transformation: sql transformation query:
after checking, I suspect the logic may related to this line, where the last day generated by windows class not included as the transformation partition.
https://github.com/odpf/transformers/blob/ea1de4f0de3d17d9be7ccefb1e2f3beab1a685f1/task/bq2bq/executor/bumblebee/transformation.py#L393
please kindly check it, and release the fix. thank you
With this we are adding support for using secrets in macros, we do not want to print the env vars in the logs, so exporting them as a separate file from optimus.
Plugins can export this extra file to get env vars.
As per https://github.com/odpf/optimus/pull/425, the executor boot process is standardised and maintained at optimus. Plugin devs need no longer have to wrap the executor image. closes https://github.com/odpf/optimus/issues/405
The most intuitive way is to parse the query and hit the metadata apis instead of going through the dry run which should be definitly costly then the metadata fetch apis.
enhancement performanceOptions
aa14f19 feat: load method for partition replace using aggregation queries (#15)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.9cb26e8f feat: add destination type when generating destination of bq2bq task to support backup feature (#7)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.882d75fb build(deps): bump sqlparse from 0.4.1 to 0.4.2 in /task/bq2bq/executor (#6) 7b6ea2d feat: add execution_project configuration (#9)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.7047c3ba fix(bq2bq): handle BQ dry run error caused by self table SQL reference (#5)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.6631527a refactor: implement plugin mods 621dc7d refactor: update help for bigquery prompts to ask for destination (#3)
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.52122157 build(deps): bump rsa from 4.6 to 4.7 in /task/bq2bq/executor (#1) 5f4d0d2 build(deps): bump urllib3 from 1.25.9 to 1.26.5 in /task/bq2bq/executor (#2) db8307c feat(bq2bq): support default values inferred from job name
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.4365796e fix: dataset/table name passed during config gen
docker pull docker.io/odpf/optimus-task-bq2bq:0.0.3
Anuvada: Interpretable Models for NLP using PyTorch So, you want to know why your classifier arrived at a particular decision or why your flashy new d
Transformers Optimus's transformation plugins are implementations of Task and Hook interfaces that allows execution of arbitrary jobs in optimus. To i
___ ____ ____ _ __ ___ / _ \ | _ \ | _ \ ___ _ __ | '__| / _ \ | |_| || | | || | | | / _ \| '__| | | | __/ | _ || |_| || |_| || __
MicBot MicBot uses Google Translate to speak everyone's chat messages. It can al
epub2audiobook Creating an Audiobook (mp3 file) using a Ebook (epub) using BeautifulSoup and Google Text to Speech Input examples qual a pasta do seu
ckanext-sitesearch Index different CKAN entities in Solr, not just datasets Requirements This extension requires CKAN 2.9 or higher and Python 3 Featu
The CPC-big model and k-means checkpoints used in Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing.
XL-Sum This repository contains the code, data, and models of the paper titled "XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Lang
A Non-Autoregressive Text-to-Speech (NAR-TTS) framework, including official PyTorch implementation of PortaSpeech (NeurIPS 2021) and DiffSpeech (AAAI 2022)
vec2word Semi-automated vocabulary generation from semantic vector models This script generates a list of potential conlang word forms along with asso
This repository contains finetuning code and checkpoints for ElasticBERT. Towards Efficient NLP: A Standard Evaluation and A Strong Baseli
Graph Coloring - Weighted Vertex Coloring Problem This project proposes several local searches and an MCTS algorithm for the weighted vertex coloring
Word Search Puzzle A simple word search made in python Usage $ python3 main.py -h usage: main.py [-h] [-c] [-f FILE] Generates a word s
Music-Generation Creating an LSTM model to generate music music-generator Used to create basic sin wave sounds music-ai Contains the functions to conv
KinkyFurs-V0 Kinky furry assistant based on GPT2 How to run python3 V0.py then, open web browser and go to localhost:8080 Requirements: Flask trans
GuwenModels: 古文自然语言处理模型合集, 收录互联网上的古文相关模型及资源. A collection of Classical Chinese natural language processing models, including Classical Chinese related models and resources on the Internet.
cc_net Tools to download and clean Common Crawl as introduced in our paper CCNet. If you found these resources useful, please consider citing: @inproc
Learn how to process, classify, cluster, summarize, understand syntax, semantics and sentiment of text data with the power of Python! This repository contains code and datasets used in my book, Text
Poincaré Embeddings for Learning Hierarchical Representations PyTorch implementation of Poincaré Embeddings for Learning Hierarchical Representations
Speech Separation The simple project to separate mixed voice (2 clean voices) to 2 separate voices. Result Example (Clisk to hear the voices): mix ||
102 Oct 01, 2022
0 Aug 06, 2021
2 Mar 09, 2022
7 Aug 25, 2022
3 Dec 02, 2022
5 Nov 23, 2022
189 Jan 02, 2023
760 Jan 03, 2023
9 Nov 25, 2022
48 Dec 14, 2022
1 Jul 08, 2022
16 Mar 10, 2022
2 Dec 02, 2021
1 Jun 11, 2022
66 Dec 26, 2022
483 Jan 02, 2023
1.5k Jan 03, 2023
31 Oct 30, 2022