Define your JSON schema as Python dataclasses

Overview

codecov Build Status

nvelope

Define your JSON schema as Python dataclasses

Installation

pip install nvelope

The problem it solves

This is basically sommething like JSON-schema, but it works with static type checking, since the classes you define are just regular python dataclasses which can (and should) be type checked with mypy library.

It also lets not to just define the structure of your JSON data in a single place in your python code, but also to define custom checks and conversions from/to JSON for any type you want.

Original use case

Say you have two microservices communicating via JSON messages, both written in python.

You may define a shared package with the messages definition and use the model's .as_json() method on one end to serialize the message and .from_json() on the other to convert it into a DTO, checking and modifying the fields and their values along the way exactly how you defined it in a single place.

Combining this with static type checking (and maybe some unit tests) you can ensure that any message one microservice can send, the other can read as long as they use the same model to pack/unpack their JSONs.

Usage

Say you have a JSON representing a user in your app looking something like this

{
    "id": 530716139,
    "username": "johndoe",
    "language_code": "en"
}

You define an envelope for it

from dataclasses import dataclass
from typing import Optional

from nvelope import (Obj, int_conv, string_conv)

@dataclass      # note the @dataclass decorator, it is important
class User(Obj):
    _conversion = {
        "id": int_conv,
        "language_code": string_conv,
        "username": string_conv,
    }

    id: int
    language_code: str
    username: str

Now you have a model that knows how to read data from the JSON (not the raw string, actually, but to the types that are allowed by the standard json.dumps function e.g. dict, list, str, int, float, bool, None ) ...

user = User.from_json(
    {
        "id": 530716139,
        "username": "johndoe",
        "language_code": "en"
    }
)

... and knows how to convert itself into JSON

User(
    id=530716139,
    username="johndoe",
    language_code="en",
).as_json() 

# returns a dictionary {
#     "id": 530716139,
#     "username": "johndoe",
#     "language_code": "en"
# }

Compound envelopes

You can also define compound envelopes.

Say we want to define a message and include info about the sender. Having defined the User envelope, we can do it like this:

from nvelope import CompoundConv

@dataclass
class Message(Obj):
    _conversion = {
        "message_id": int_conv,
        "from_": CompoundConv(User),
        "text": string_conv,
    }

    from_: User
    text: str
    message_id: int

and use it the same way:

# reading an obj from json like this

Message.from_json(
    {
        "message_id": 44,
        "text": "hello there",
        "from_": {
            "id": 530716139,
            "username": "johndoe",
            "language_code": "en"
        }
    }
)

# and dumping an object to json like this
Message(
    message_id=44,
    text="whatever",
    from_=User(
        id=530716139,
        username="johndoe",
        language_code="en",
    )
).as_json()

Arrays

This is how you define arrays:

from nvelope import Arr


class Users(Arr):
    conversion = CompoundConv(User)


# Same API inherited from nvelope.Compound interface

Users.from_json(
    [
        {
            "id": 530716139,
            "username": "johndoe",
            "language_code": "en",
        },
        {
            "id": 452341341,
            "username": "ivandrago",
            "language_code": "ru",
        }
    ]
)

Users(
    [
        User(
            id=530716139,
            username="johndoe",
            language_code="en",
        ),
        User(
            id=452341341,
            username="ivandrago",
            language_code="ru",
        ),
    ]
).as_json()

Field aliases

At some point you may need to define an envelope for an API containing certain field names which cannot be used in python since they are reserved keywords.

There's a solution for this:

from nvelope import ObjWithAliases

@dataclass
class Comment(ObjWithAliases):
    _conversion = {
        "text": string_conv,
        "from_": int_conv,
    }
    
    
    _alias_to_actual = {
        "from_": "from",
    }
    
    text: str
    from_: User

In this case from key gets replaced by from_ in the python model.

Missing and optional fields

There's a difference between fields that can be set to None and fields which may be missing in the JSON at all.

This is how you specify that a some field may be missing from the JSON and that's OK:

from typing import Optional

from nvelope import MaybeMissing
from nvelope import OptionalConv

@dataclass
class Comment(ObjWithAliases):
    _alias_to_actual = {
        "from_": "from",
    }
    
    text: str
    img: Optional[str]          # this field can be set to None (null), but is must always be present in the JSON
    from_: MaybeMissing[User]   # this field can be missing from JSON body

    _conversion = {
        "text": string_conv,
        "img": OptionalConv(string_conv),   # note the wrapping with OptionalConv
        "from_": int_conv,
    }

This is how you check if the MaybeMissing field is actually missing

comment.from_.has()     # returns False if the field is missing

and this is how you get the value:

comment.value()     # raises an error if there's no value, 
                    # so it is recommended to check the output of .has()
                    #  before calling .value() 

Custom conversions

You may define a custom conversions inheriting from nvelope.nvelope.Conversion abstract base class or using nvelope.nvelope.ConversionOf class.

For example, this is how datetime_iso_format_conv is defined:

from nvelope import WithTypeCheck, ConversionOf

datetime_iso_format_conv = WithTypeCheck(
    datetime.datetime,
    ConversionOf(
        to_json=lambda v: v.isoformat(),
        from_json=lambda s: datetime.datetime.fromisoformat(s),
    ),
)

Say we want to jsonify a datetime field as POSIX timestamp, instead of storing it in ISO string format.

datetime_timestamp_conv = ConversionOf(
    to_json=lambda v: v.timestamp(),
    from_json=lambda s: datetime.datetime.fromtimestamp(s),
)

We could also add WithTypeCheck wrapper in order to add explicit check that the value passed into .from_json() is indeed float.

datetime_timestamp_conv = WithTypeCheck(
    float,
    ConversionOf(
        to_json=lambda v: v.timestamp(),
        from_json=lambda s: datetime.datetime.fromtimestamp(s),
    )
)
You might also like...
simplejson is a simple, fast, extensible JSON encoder/decoder for Python

simplejson simplejson is a simple, fast, complete, correct and extensible JSON http://json.org encoder and decoder for Python 3.3+ with legacy suppo

jq for Python programmers Process JSON and HTML on the command-line with familiar syntax.

jq for Python programmers Process JSON and HTML on the command-line with familiar syntax.

A JSON utility library for Python featuring Django-style queries and mutations.

JSON Enhanced JSON Enhanced implements fast and pythonic queries and mutations for JSON objects. Installation You can install json-enhanced with pip:

Python script for converting .json to .md files using Mako templates.

Install Just install poetry and update script dependencies Usage Put your settings in settings.py and .json data (optionally, with attachments) in dat

json|dict to python object

Pyonize convert json|dict to python object Setup pip install pyonize Examples from pyonize import pyonize

Editor for json/standard python data
Editor for json/standard python data

Editor for json/standard python data

Python script to extract news from RSS feeds and save it as json.

Python script to extract news from RSS feeds and save it as json.

A Python tool that parses JSON documents using JsonPath

A Python tool that parses JSON documents using JsonPath

Simple Python Library to convert JSON to XML
Simple Python Library to convert JSON to XML

json2xml Simple Python Library to convert JSON to XML

Comments
  • Recursive definition

    Recursive definition

    Is there a way for recursive definitions? So for example let's say we want to consider sections in an article. A section have a title and maybe some subsections. The corresponding json could look like:

    {
        "title": "This is a really nice title",
        "sections": [
            {
                "title": "Oh this title is even nicer"
                "sections: [
                    {
                         "title: "Not so nice title, no subsections"
                    }
                ]
            },
            {
                 "title": "Section without subsection"
            }
        ]
    

    So we could start with:

    @dataclass
    class Section(Obj):
        _conversion = {"title": string_conv}
    
        title: str
    

    But obviously the maybe subsections are missing. Is there a way to model that? Thanks.

    opened by donpatrice 4
  • Descriptors

    Descriptors

    From readme I see

    @dataclass    
    class User(Obj):
        _conversion = {
            "id": int_conv,
            "language_code": string_conv,
            "username": string_conv,
        }
    
        id: int
        language_code: str
        username: str
    

    Which makes me curious on why to define an argument named _conversion, is this a design decision for some reason?

    What about implementing that using the descriptor protocol?

    @dataclass    
    class User(Obj):
        id: int = IntField()
        language_code: str = StringField()
        username: str = StringField()
    

    And having all *Field to be implementation of descriptors such as:

    class IntField:
    
        def __set_name__(self, owner, name):
            self.public_name = name
            self.private_name = '_' + name
    
        def __get__(self, obj, objtype=None):
            value = getattr(obj, self.private_name)
            logging.info('Accessing %r giving %r', self.public_name, value)
            return int(value)  # conversion here on reading
    
        def __set__(self, obj, value):
            logging.info('Updating %r to %r', self.public_name, value)
            setattr(obj, self.private_name, int(value))  # conversion here on writing
    
    
    opened by rochacbruno 1
Releases(v1.1.0)
  • v1.1.0(May 30, 2022)

  • v1.0.0(Feb 3, 2022)

    • JSON schema generation in compound objects via .schema() method;
    • Conversion interface now requires .schema() method returning a JSON schema;
    • moved functionality of ObjWithAliases to Obj;
    • added possibility of storing undefined JSON fields in a model instance;
    • validated class decorator checking the correctness of a Obj and Arr subclass;
    • datetime_timestamp_conv to store a datetime as POSIX timestamp float.
    Source code(tar.gz)
    Source code(zip)
  • v0.4.0(Dec 10, 2021)

  • v0.3.1(Nov 23, 2021)

Owner
A big fan of high quality software engineering
API that provides Wordle (ES) solutions in JSON format

Wordle (ES) solutions API that provides Wordle (ES) solutions in JSON format.

Álvaro García Jaén 2 Feb 10, 2022
cysimdjson - Very fast Python JSON parsing library

Fast JSON parsing library for Python, 7-12 times faster than standard Python JSON parser.

TeskaLabs 235 Dec 29, 2022
A Python tool that parses JSON documents using JsonPath

A Python tool that parses JSON documents using JsonPath

8 Dec 18, 2022
Creates fake JSON files from a JSON schema

Use jsf along with fake data generators to provide consistent and meaningful fake data for your system.

Andy Challis 86 Jan 03, 2023
simdjson : Parsing gigabytes of JSON per second

JSON is everywhere on the Internet. Servers spend a *lot* of time parsing it. We need a fresh approach. The simdjson library uses commonly available SIMD instructions and microparallel algorithms to

16.3k Dec 29, 2022
Json utils is a python module that you can use when working with json files.

Json-utils Json utils is a python module that you can use when working with json files. it comes packed with a lot of featrues Features Converting jso

Advik 4 Apr 24, 2022
A fast streaming JSON parser for Python that generates SAX-like events using yajl

json-streamer jsonstreamer provides a SAX-like push parser via the JSONStreamer class and a 'object' parser via the ObjectStreamer class which emits t

Kashif Razzaqui 196 Dec 15, 2022
A JSON utility library for Python featuring Django-style queries and mutations.

JSON Enhanced JSON Enhanced implements fast and pythonic queries and mutations for JSON objects. Installation You can install json-enhanced with pip:

Collisio Technologies 4 Aug 22, 2022
A JSON API for returning Godspeak sentences. Based on the works of Terry A Davis (Rest in Peace, King)

GodspeakAPI A simple API for generating random words ("godspeaks"), inspired by the works of Terrence Andrew Davis (Rest In Peace, King). Installation

Eccentrici 3 Jan 24, 2022
JSON Schema validation library

jsonschema A JSON Schema validator implementation. It compiles schema into a validation tree to have validation as fast as possible. Supported drafts:

Dmitry Dygalo 309 Jan 01, 2023
Json GUI for No Man's Sky save file

NMS-Save-Parser Json GUI for No Man's Sky save file GUI python NMS_SAVE_PARSER.py [optional|save.hg] converter only python convert.py usage: conver

2 Oct 19, 2022
A Python application to transfer Zeek ASCII (not JSON) logs to Elastic/OpenSearch.

zeek2es.py This Python application translates Zeek's ASCII TSV logs into ElasticSearch's bulk load JSON format. For JSON logs, see Elastic's File Beat

Corelight, Inc. 28 Dec 22, 2022
Console to handle object storage using JSON serialization and deserealization.

Console to handle object storage using JSON serialization and deserealization. This is a team project to develop a Python3 console that emulates the AirBnb object management.

Ronald Alexander 3 Dec 03, 2022
Roamtologseq - A script loads a json export of a Roam graph and cleans it up for import into Logseq

Roam to Logseq The script loads a json export of a Roam graph and cleans it up f

Sebastian Pech 4 Mar 07, 2022
Simple, minimal conversion of Bus Open Data Service SIRI-VM data to JSON

Simple, minimal conversion of Bus Open Data Service SIRI-VM data to JSON

Andy Middleton 0 Jan 22, 2022
Marshall python objects to and from JSON

Pymarshaler - Marshal and Unmarshal Python Objects Disclaimer This tool is in no way production ready About Pymarshaler allows you to marshal and unma

Hernan Romer 9 Dec 20, 2022
A tools to find the path of a specific key in deep nested JSON.

如何快速从深层嵌套 JSON 中找到特定的 Key #公众号 在爬虫开发的过程中,我们经常遇到一些 Ajax 加载的接口会返回 JSON 数据。

kingname 56 Dec 13, 2022
JSON for Modern C++ Release Scripts

JSON for Modern C++ Release Scripts Preparations Install required tools: make install_requirements. Add required keys to config.json (apparently not c

Niels Lohmann 4 Sep 19, 2022
A daily updated JSON dataset of all the Open House London venues, events, and metadata

Open House London listings data All of it. Automatically scraped hourly with updates committed to git, autogenerated per-day CSV's, and autogenerated

Jonty Wareing 4 Jan 01, 2022
JsonParser - Parsing the Json file by provide the node name

Json Parser This project is based on Parsing the json and dumping it to CSV via

Ananta R. Pant 3 Aug 08, 2022