Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
A Discord bot that may save your day by predicting it.

Sage A Discord bot that may save your day by predicting it.

1 Nov 17, 2022
Best DDoS Attack Script Python3, Cyber Attack With 40 Methods

MXDDoS - DDoS Attack Script With 40 Methods (Code Lang - Python 3) Please Don't Attack '.gov' and '.ir' Websites :) Features And Methods 💣 Layer7 GET

7 Mar 07, 2022
Changes the Telegram bio, profile picture, first and last name to the song that the user is currently listening to.

TGBIOFY - Telegram & Spotify integration Changes the Telegram bio, profile picture, first and last name to the song that the user is currently listeni

elpideus 26 Dec 07, 2022
Unofficial Meteor Client wiki

Welcome to the Unofficial Meteor Client wiki! Meteor FAQs | A rewritten and better FAQ page. Installation Guide | A guide on how to install Meteor Cli

Anti Cope 0 Feb 21, 2022
A discord bot that will help you browse/download nhentai sources.

Risa Introduction Risa is an nHentai discord bot that will help you browse and download your favorite doujin inside your own discord server. Hosting M

markee7 14 Oct 25, 2021
Track player's stats, find out when they're online and grinding!

Hypixel Stats Tracker Track player's stats, find out when they're online and playing games! INFO Showcase Server: https://discord.gg/yY5qQHPar6 Suppor

4 Dec 18, 2022
:evergreen_tree: Python module for communicating with the Taiga API

python-taiga A python wrapper for the Taiga REST API. Documentation: https://python-taiga.readthedocs.io/ Usage: : https://python-taiga.readthedocs.io

Nephila 87 Oct 12, 2022
👨‍💼Linkedin API for Python

linkedin_api 👨‍💼 Linkedin API for Python No "official" API access required - just use a valid Linkedin account! Programmatically send messages, get

Tom Quirk 918 Dec 29, 2022
Python-random-quote - A file-based quote bot written in Python

Let's Write a Python Quote Bot! This repository will get you started with building a quote bot in Python. It's meant to be used along with the Learnin

amir mohammad fateh 1 Jan 02, 2022
A Tool to scrape URLs for a given domain from wayback machine, Commoncrawl and OTX Alienvault

Mr_URL Mr.URL fetches known URLs for a given domain from Wayback Machine, Commoncrawl and OTX Alienvault. It also finds old versions of any given URL

Stinger 9 Sep 05, 2022
A Recommendation System For Diabetes Detection And Treatment

Diabetes-detection-tg-bot A Recommendation System For Diabetes Detection And Treatment Данная система помогает определить наличие или отсутствие сахар

Alexander Kanonirov 1 Nov 22, 2021
A media upload to telegraph module

A media upload to telegraph module

Fayas Noushad 5 Dec 01, 2021
A Discord webhook spammer made in Python

A Python made Discord webhook spammer usually used for token loggers to spam them/delete them original by cattyn changes listed below.

2 Jan 12, 2022
(unofficial) Googletrans: Free and Unlimited Google translate API for Python. Translates totally free of charge.

Googletrans Googletrans is a free and unlimited python library that implemented Google Translate API. This uses the Google Translate Ajax API to make

Suhun Han 3.2k Jan 04, 2023
Boilerplate template for the discord-py-interactions library

discord-py-interactions_boilerplate Boilerplate template for the discord-py-interactions library Currently, this boilerplate supports discord-py-inter

Ventus 7 Dec 03, 2022
An open source raffle bot made to increase the chance of winning limited sneaker raffles by automating entries.

🚀 SyneziaRaffles An open source raffle bot made to increase the chance of winning limited sneaker raffles by automating entries. 🏄‍♂️ Quick Start Pr

Alexis M. 29 Dec 22, 2022
WordPress models and views for Django.

django-wordpress Models and views for reading a WordPress database. Compatible with WordPress version 3.5+. django-wordpress is a project of ISL and t

Jeremy Carbaugh 332 Dec 24, 2022
A powerful, cool and well-made userbot for your Telegram profile with promising extension capabilities.

Telecharm userbot A powerful, fast and simple Telegram userbot written in Python 3 and based on Pyrogram 1.X. Currently in active WIP state, so feel f

Daniil Kovalenko 16 Dec 01, 2022
Get an SNS alert for High Severity GuardDuty findings

Automation AWS-GuardDuty findings Get an SNS alert for High Severity GuardDuty findings Problem: Getting notified when there is Red finding in AWS Gua

Giten Mitra 4 Nov 18, 2022
Opasium AI was specifically designed for the Opasium Games discord only. It is a bot that covers the basic functions of any other bot.

OpasiumAI Opasium AI was specifically designed for the Opasium Games discord only. It is a bot that covers the basic functions of any other bot. Insta

Dan 3 Oct 15, 2021