Python function to construct an ODS spreadsheet on the fly - without having to store the entire file in memory or disk

Overview

stream-write-ods CircleCI Test Coverage

Python function to construct an ODS (OpenDocument Spreadsheet) on the fly - without having to store the entire file in memory or disk.

Can be used to convert CSV, SQLite, or JSON to ODS format.

Installation

pip install stream-write-ods

Usage

In general, pass a nested iterable to stream_write_ods and it will return an interable of bytes of the ODS file, as follows.

from stream_write_ods import stream_write_ods

def get_sheets():
    def get_rows_of_sheet_1():
        yield 'Value A', 'Value B'
        yield 'Value C', 'Value D'

    yield 'Sheet 1 name', ('col_1_name', 'col_2_name'), get_rows_of_sheet_1()

    def get_rows_of_sheet_2():
        yield 'col_1_value',

    yield 'Sheet 2 name', ('col_1_name',), get_rows_of_sheet_2()

ods_chunks = stream_write_ods(get_sheets())

Usage: Convert CSV to ODS

The following recipe converts a CSV to ODS.

import codecs
import csv
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a CSV file
# Hard coded for the purposes of this example
csv_bytes_iter = (
    b'col_1,col_2\n',
    b'1,"value"\n',
)

def get_sheets(sheet_name, csv_reader):
    yield sheet_name, next(csv_reader), csv_reader

csv_str_iter = codecs.iterdecode(csv_bytes_iter, 'utf-8')
csv_reader = csv.reader(csv_str_iter, csv.QUOTE_NONNUMERIC)
ods_chunks = stream_write_ods(get_sheets('Sheet 1', csv_reader))

Usage: Convert JSON to ODS

Using ijson to stream-parse a JSON file, it's possible to convert JSON data to ODS on the fly:

import ijson
import itertools
from stream_write_ods import stream_write_ods

# Any iterable that yields the bytes of a JSON file
# Hard coded for the purposes of this example
json_bytes_iter = (b'''{
  "data": [
      {"id": 1, "name": "Foo"},
      {"id": 2, "name": "Bar"}
  ]
}''',)

# ijson requires a file-like object
def to_file_like_obj(bytes_iter):
    chunk = b''
    offset = 0
    it = iter(bytes_iter)

    def up_to_iter(num):
        nonlocal chunk, offset

        while num:
            if offset == len(chunk):
                try:
                    chunk = next(it)
                except StopIteration:
                    break
                else:
                    offset = 0
            to_yield = min(num, len(chunk) - offset)
            offset = offset + to_yield
            num -= to_yield
            yield chunk[offset - to_yield:offset]

    class FileLikeObj:
        def read(self, n):
            return b''.join(up_to_iter(n))

    return FileLikeObj()

def get_sheets(json_file):
    columns = None

    def rows():
        nonlocal columns
        for item in ijson.items(json_file, 'data.item'):
            if columns is None:
                columns = list(item.keys())
            yield tuple(item[column] for column in columns)

    # Ensure columns populated
    rows_it = rows()
    first_row = next(rows_it)

    yield 'Sheet 1', columns, itertools.chain((first_row,), rows_it)

json_file = to_file_like_obj(json_bytes_iter)
ods_chunks = stream_write_ods(get_sheets(json_file))

Usage: Convert SQLite to ODS

SQLite isn't particularly streaming-friendly since typically you need random access to the file. But it's still possible to use stream-write-ods to convert SQLite to ODS.

import contextlib
import sqlite3
import tempfile
from stream_write_ods import stream_write_ods

@contextlib.contextmanager
def get_db():
    # Hard coded in memory database for the purposes of this example
    with sqlite3.connect(':memory:') as con:
        cur = con.cursor()
        cur.execute("CREATE TABLE my_table_a (my_col text);")
        cur.execute("CREATE TABLE my_table_b (my_col text);")
        cur.execute("INSERT INTO my_table_a VALUES ('Value A')")
        cur.execute("INSERT INTO my_table_b VALUES ('Value B')")
        yield con

def quote_identifier(value):
    return '"' + value.replace('"', '""') + '"'

def get_sheets(db):
    cur_table = db.cursor()
    cur_table.execute('''
        SELECT name FROM sqlite_master
        WHERE type = "table" AND name NOT LIKE 'sqlite\\_%' ESCAPE '\\'
    ''')
    cur_data = db.cursor()
    for table, in cur_table:
        cur_data.execute(f'SELECT * FROM {quote_identifier(table)} ORDER BY rowid')
        yield table, tuple(col[0] for col in cur_data.description), cur_data

with get_db() as db:
    ods_chunks = stream_write_ods(get_sheets(db))

Types

There are 8 possible data types in an Open Document Spreadsheet: boolean, currency, date, float, percentage, string, time, and void. 4 of these can be output by stream-write-ods, chosen automatically according to the following table.

Python type ODS type
boolean boolean
date date - without time component
datetime date - with time component
int float
float float
str string
NoneType string - as #NA

Limitations

ODS spreadsheets are essentially ZIP archives containing several member files. While in general ZIP archives can be up to 16EiB (exbibyte) in size using ZIP64, LibreOffice does not support ZIP64, and so ODS files are de-facto limited to 4GiB (gibibyte). This limit applies to the size of the entire compressed archive, the compressed size of each member file, and the uncompressed size of each member file.

Owner
Department for International Trade
Department for International Trade
Userbot Telegram + Music Voice Chats. Dibuat Untuk Bersenang - Senang , Dan Mempermudah Kegiatan. Created By Rio.

RIO - USERBOT Disclaimer Saya tidak bertanggung jawab atas penyalahgunaan bot ini. Bot ini dimaksudkan untuk bersenang-senang sekaligus membantu Anda

RioProjectX 1 Nov 10, 2021
A simple anti-ghostping python bot made using diskord.

Anti Ghostping A simple Anti-Ghostping python bot made with ❤ using Diskord Requirements No one will use this but, all you need for this bot is: Pytho

RyZe 2 Sep 12, 2022
A simple telegram bot to recognize lengthy voice files to text and vice versa with multiple language support.

Voicebot A simple Telegram bot to convert lengthy voice clips to text and vice versa with supporting languages. Mandatory Variables API_HASH - Yo

Renjith Mangal 12 Oct 21, 2022
Bot interpretation of the carbon.now.sh site

📒 Source code of the @PicodeBot 🧸 Developer: @hoosnick Run $ git clone https://github.com/hoosnick/picodebot.git $ pip install -r requirements.txt P

Husniddin Murodov 13 Oct 02, 2022
A simple discord bot based on python

A simple discord bot based on python

SENPAI LEGEND 2 Jul 24, 2022
Evernote SDK for Python

Evernote SDK for Python Evernote API version 1.28 This SDK is intended for use with Python 2.X For Evernote's beta Python 3 SDK see https://github.com

Evernote 612 Dec 30, 2022
wyscoutapi is an extremely basic API client for the Wyscout API (v2 & v3) for Python

wyscoutapi wyscoutapi is an extremely basic API client for the Wyscout API (v2 & v3). Usage Install with pip install wyscoutapi. To connect to the Wys

Ben Torvaney 11 Nov 22, 2022
A Discord bot that automatically saves SHSH blobs for all of your iOS devices.

AutoTSS AutoTSS is a Discord bot that automatically saves SHSH blobs for all of your iOS devices. Want a CLI automatic blob saver? Check out AutoTSS-c

adam 79 Dec 13, 2022
The best discord Nuk3r !

Discord - Nuker ☢️ Nuk3r ☢️ STEP 1 ✅ We go create discord bot ! [] Go on https://discord.com/developers/applications [] Set the name of your applica

2s.py 1 Apr 16, 2022
CSUL Discord Bot

Cruzeiro This is the same old bot running on the Discord Server of CSUL, but i've changed the code. It's better now. Discord.py Heroku How i did The b

Operaho 6 Jan 31, 2022
Monitor robot of Apple Store's products, using DingTalk notification.

概述 本项目应用主要用来监测Apple Store线下直营店货源情况,主要使用Python实现。 首先感谢iPhone-Pickup-Monitor项目带来的灵感,同时有些实现也直接使用了该项目的一些代码。 本项目在iPhone-Pickup-Monitor原有功能的基础上去掉了声音通知,但添加了多

Lennon Chin 159 Dec 09, 2022
A Telegram Bot which notifies the user when a vaccine is available on CoWin Platform.

Cowin Vaccine Availability Notifier Telegram Bot A bot that notifies the available vaccines at given district in realtime. Introduction • Requirements

Arham Shah 7 Jul 31, 2021
Discord bot built using Python. through this you can get information about the upcoming matches, scoreboard, live score

IPL-bot This is a Discord bot built using Python. through this you can get information about the upcoming matches, scoreboard, live score, and many mo

0 Dec 23, 2021
Telegram Group Management Bot based on Pyrogram

Komi-San Telegram Group Management Bot based on Pyrogram More updates coming soon Support Group Open a Pull request if you wana contribute Example for

33 Nov 07, 2022
Este programa tem como objetivo o cadastro dos usuários. Assim, caso a autenticação seja feita, permitir que o usuário entre em determinado sistema ou programa.

LoginPy Este programa tem como objetivo o cadastro dos usuários. Assim, caso a autenticação seja feita, permitir que o usuário entre em determinado si

Jonas Carvalho 4 Dec 23, 2021
Debugging with Stack Overflow: Web Search Behavior in Novice and Expert Programmers

Debugging with Stack Overflow: ICSE SEET, 2022 This is the GitHub repository associated with the 2022 ICSE SEET paper, Debugging with Stack Overflow:

Madeline Endres 1 Jan 31, 2022
Download apps and remove icloud

Download apps and remove icloud

A fork of discord.py meant to replace it

Texus A modern, easy to use, feature-rich, and async ready API wrapper for Discord written in Python. Key Features Modern Pythonic API using async and

Texus 1 Nov 18, 2021
A Python SDK for connecting devices to Microsoft Azure IoT services

V2 - We are now GA! This repository contains code for the Azure IoT SDKs for Python. This enables python developers to easily create IoT device soluti

Microsoft Azure 381 Dec 30, 2022
Darkflame Universe Account Manager

Darkflame Universe Account Manager This is a quick and simple web application intended for account creation and management for a DLU instance created

31 Nov 29, 2022