Simple embedded in memory json database

Overview

Build Status Coverage Version Supported Downloads License

dbj

dbj is a simple embedded in memory json database.

It is easy to use, fast and has a simple query language.

The code is fully documented, tested and beginner friendly.

Only the standard library is used and it works on Python 2.7, Python 3.4+, PyPy 2.7 and PyPy 3.6.

Usage

>> r = db.find('name == "John" or name == "Bob" and age > 10') >>> db.getmany(r) [{'name': 'Bob', 'age': 30}, {'name': 'John', 'age': 18}] >>> # Sort the result by age >>> r = db.sort(r, 'age') >>> db.getmany(r) [{'name': 'John', 'age': 18}, {'name': 'Bob', 'age': 30}] >>> # Sort can also be used from find directly >>> r = db.find('age >= 10', sortby='age') >>> db.getmany(r) [{'name': 'Ana Beatriz', 'age': 10}, {'name': 'John', 'age': 18}, {'name': 'Bob', 'age': 30}] >>> # One-liner: >>> db.getmany(db.find('age >= 10', sortby='age')) [{'name': 'Ana Beatriz', 'age': 10}, {'name': 'John', 'age': 18}, {'name': 'Bob', 'age': 30}] >>> db.save() True">
>>> from dbj import dbj
>>> db = dbj('mydb.json')

>>> # Insert using an auto generated uuid1 key
>>> db.insert({'name': 'John', 'age': 18})
'a71d90ce0c7611e995faf23c91392d78'

>>> # Insert using a supplied key, in this case '[email protected]'
>>> user = {'name': 'Ana Beatriz', 'age': 10}
>>> db.insert(user, '[email protected]')
'[email protected]'

>>> db.insert({'name': 'Bob', 'age': 30})
'cc6ddfe60c7611e995faf23c91392d78'

>>> db.get('a71d90ce0c7611e995faf23c91392d78')
{'name': 'John', 'age': 18}

>>> db.get('[email protected]')
{'name': 'Ana Beatriz', 'age': 10}

>>> db.find('age >= 18')
['a71d90ce0c7611e995faf23c91392d78', 'cc6ddfe60c7611e995faf23c91392d78']

>>> db.find('name == "ana beatriz"')
['[email protected]']

>>> r = db.find('name == "John" or name == "Bob" and age > 10')
>>> db.getmany(r)
[{'name': 'Bob', 'age': 30}, {'name': 'John', 'age': 18}]

>>> # Sort the result by age
>>> r = db.sort(r, 'age')
>>> db.getmany(r)
[{'name': 'John', 'age': 18}, {'name': 'Bob', 'age': 30}]

>>> # Sort can also be used from find directly
>>> r = db.find('age >= 10', sortby='age')
>>> db.getmany(r)
[{'name': 'Ana Beatriz', 'age': 10}, {'name': 'John', 'age': 18}, {'name': 'Bob', 'age': 30}]

>>> # One-liner:
>>> db.getmany(db.find('age >= 10', sortby='age'))
[{'name': 'Ana Beatriz', 'age': 10}, {'name': 'John', 'age': 18}, {'name': 'Bob', 'age': 30}]

>>> db.save()
True

Install

Install using pip:

$ pip install dbj

Examples

Check the available commands for a full list of supported methods.

Import the module and create a new database:

>>> from dbj import dbj
>>> db = dbj('mydb.json')

Insert a few documents with auto generated key:

>>> doc = {'name': 'John Doe', 'age': 18}
>>> db.insert(doc)
'7a5ebd420cb211e98a0ff23c91392d78'

>>> docs = [{'name': 'Beatriz', 'age': 30}, {'name': 'Ana', 'age': 10}]
>>> db.insertmany(docs)
2

Insert with a supplied key:

>>> doc = {'name': 'john', 'age': 20, 'country': 'Brasil'}
>>> db.insert(doc, '1')
1

>>> db.insert({'name': 'Bob', 'age': 40}, '2')
2

>>> db.getallkeys()
['7a5ebd420cb211e98a0ff23c91392d78', 'db21baf80cb211e98a0ff23c91392d78', 'db21edde0cb211e98a0ff23c91392d78', '1', '2']

Pop and delete:

>>> db.delete('1')
True

>>> db.poplast()
{'name': 'Bob', 'age': 40}

>>> db.size()
3

>>> db.getallkeys()
['7a5ebd420cb211e98a0ff23c91392d78', 'db21baf80cb211e98a0ff23c91392d78', 'db21edde0cb211e98a0ff23c91392d78']

Updating an existing document:

>>> db.insert({'name': 'Ethan', 'age': 40}, '1000')
'1000'

>>> db.get('1000')
{'name': 'Ethan', 'age': 40}

>>> db.update('1000', {'age': 50})
True

>>> db.get('1000')
{'name': 'Ethan', 'age': 50}

>>> db.update('1000', {'name': 'Ethan Doe', 'gender': 'male'})
True

>>> db.pop('1000')
{'name': 'Ethan Doe', 'age': 50, 'gender': 'male'}

Retrieving some documents:

>>> db.getall()
[{'name': 'John Doe', 'age': 18}, {'name': 'Beatriz', 'age': 30}, {'name': 'Ana', 'age': 10}]

>>> db.getfirst()
{'name': 'John Doe', 'age': 18}

>>> db.getlast()
{'name': 'Ana', 'age': 10}

>>> db.getrandom() # returns a random document
{'name': 'Ana', 'age': 10}

Check for existance:

>>> db.exists('7a5ebd420cb211e98a0ff23c91392d78')
True

Searchin and sorting:

>> db.getmany(r) [{'name': 'John Doe', 'age': 18}] >>> query = 'name == "john doe" or name == "ana" and age >= 10' >>> r = db.find(query) >>> db.getmany(r) [{'name': 'John Doe', 'age': 18}, {'name': 'Ana', 'age': 10}] >>> r = db.find('age < 40', sortby='age') >>> db.getmany(r) [{'name': 'Ana', 'age': 10}, {'name': 'John Doe', 'age': 18}, {'name': 'Beatriz', 'age': 30}]">
>>> r = db.sort(db.getallkeys(), 'name')
>>> db.getmany(r)
[{'name': 'Ana', 'age': 10}, {'name': 'Beatriz', 'age': 30}, {'name': 'John Doe', 'age': 18}]

>>> r = db.find('name ?= "john"')
>>> db.getmany(r)
[{'name': 'John Doe', 'age': 18}]

>>> query = 'name == "john doe" or name == "ana" and age >= 10'
>>> r = db.find(query)
>>> db.getmany(r)
[{'name': 'John Doe', 'age': 18}, {'name': 'Ana', 'age': 10}]

>>> r = db.find('age < 40', sortby='age')
>>> db.getmany(r)
[{'name': 'Ana', 'age': 10}, {'name': 'John Doe', 'age': 18}, {'name': 'Beatriz', 'age': 30}]

Save the database to disk:

>>> db.save()
True

To save a prettified json, use indent:

>>> db.save(indent=4)
True

Enable auto saving to disk after a insert, update or delete:

>>> db = dbj('mydb.json', autosave=True)

About the simple query language

The query for the find command uses the following pattern:

field operator value and/or field operator value...

Spaces are mandatory and used as a separator by the parser. For example, the following query will not work:

name=="John" and age >=18

A valid example:

name == "John Doe" and age >= 18

Strings must be enclosed by quotes. Quoted text can be searched using double quotes as the string delimiter, like:

name == ""Bob "B" Lee""

Please note that if value is a string, a search for text will be executed (using the string operatos below) and if value is a number, a number comparison search will be used.

The supported string operators are:

'==' -> Exact match. 'John' will not match 'John Doe' but will match 'john'
by default. If case sensitive is desired, just use find with sens=True. See
available commands below for the full find method signature.

'?=' -> Partial match. In this case, 'John' will match 'John Doe'.

'!=' -> Not equal operator.

The numbers comparison operators are:

'==', '!=', '<', '<=', '>', '>='

The supported logical operatos are:

and, or

Important changes

0.1.4:

  • The insert() method will raise a TypeError exception if the document dict is not json serializable.

Performance

Since the entire database is an OrderedDict in memory, performance is pretty good. On a cheap single core VM it can handle dozens of thousands operations per second.

A simple benchmark is included to get a roughly estimative of operations per second. Here is the result on a $5 bucks Linode VM running on Python 3.6:

$ python3.6 bench_dbj.py

--------------------------------

Inserting 100000 documents using auto generated uuid1 key...
Done! Time spent: 3.23s
Inserted: 100000
Rate: 30995 ops/s

--------------------------------

Clearing the database...
Done!

--------------------------------

Inserting 100000 documents using a supplied key...
Done! Time spent: 1.26s
Inserted: 100000
Rate: 79587 ops/s

--------------------------------

Retrieving 100000 documents one at a time...
Done! Time spent: 1.61s
Retrieved: 100000
Rate: 62136 ops/s

--------------------------------

Saving database to disk...
Done! Time spent: 1.09s

--------------------------------

Deleting 100000 documents one at a time...
Done! Time spent: 0.22s
Deleted: 100000
Rate: 450764 ops/s

--------------------------------

Removing file...
Done!

Peak memory usage: 57.37 MB

Available commands

insert(document, key=None) -> Create a new document on database.
Args:
document (dict): The document to be created.
key (str, optional): The document unique key. Defaults to uuid1.
Returns:
The document key.
insertmany(documents) -> Insert multiple documents on database.
Args:
documents (list): List containing the documents to insert.
Returns:
Number of inserted documents.
save(indent=None) -> Save database to disk.
Args:
indent (int or str, optional): If provided, save a prettified json with that indent level. 0, negative or "" will only insert newlines.
Returns:
True if successful.
clear() -> Remove all documents from database.
Returns:
True if successful.
size() -> Return the database size.
Returns:
Number of documents on database.
exists(key) -> Check if a document exists on database.
Args:
key (str): The document key.
Returns:
True or False if it does not exist.
delete(key) -> Delete a document on database.
Args:
key (str): The document key.
Returns:
True or False if it does not exist.
deletemany(keys) -> Delete multiple documents on database.
Args:
keys (list): List containing the keys of the documents to delete.
Returns:
Number of deleted documents.
update(key, values) -> Add/update values on a document.
Args:
key (str): The document key.
values (dict): The values to be added/updated.
Returns:
True or False if document does not exist.
updatemany(keys, values) -> Add/update values on multiple documents.
Args:
keys (list): List containing the keys of the documents to update.
values (dict): The values to be added/updated.
Returns:
Number of updated documents.
get(key) -> Get a document on database.
Args:
key (str): The document key.
Returns:
The document or False if it does not exist.
getmany(keys) -> Get multiple documents from database.
Args:
keys (list): List containing the keys of the documents to retrieve.
Returns:
List of documents.
getall() -> Return a list containing all documents on database.
Returns:
List with all database documents.
getallkeys() -> Return a list containing all keys on database.
Returns:
List with all database keys.
getrandom() -> Get a random document on database.
Returns:
A document or False if database is empty.
getfirst() -> Get the first inserted document on database.
Returns:
The first inserted document or False if database is empty.
getlast() -> Get the last inserted document on database.
Returns:
The last inserted document or False if database is empty.
getfirstkey() -> Get the first key on database.
Returns:
The first key or False if database is empty.
getlastkey() -> Get the last key on database.
Returns:
The last key or False if database is empty.
pop(key) -> Get the document from database and remove it.
Args:
key (str): The document key.
Returns:
The document or False if it does not exist.
popfirst() -> Get the first inserted document on database and remove it.
Returns:
The first inserted document or False if database is empty.
poplast() -> Get the last inserted document on database and remove it.
Returns:
The last inserted document or False if database is empty.
sort(keys, field, reverse=False) -> Sort the documents using the field provided.
Args:
keys (list): List containing the keys of the documents to sort.
field (str): Field to sort.
reverse (bool, optional): Reverse search. Defaults to False.
Returns:
Sorted list with the documents keys.
findtext(field, text, exact=False, sens=False, inverse=False, asc=True) -> Simple text search on the provided field.
Args:
field (str): The field to search.
text (str): The value to be searched.
exact (bool, optional): Exact text match. Defaults to False.
sens (bool, optional): Case sensitive. Defaults to False.
inverse (bool, optional): Inverse search, return the documents that do not match the search. Defaults to False.
asc (bool, optional): Ascii conversion before matching, this matches text like 'cafe' and 'café'. Defaults to True.
Returns:
List with the keys of the documents that matched the search.
findnum(expression) -> Simple number comparison search on provided field.
Args:
expression (str): The comparison expression to use, e.g., "age >= 18". The pattern is 'field operator number'.
Returns:
List with the keys of the documents that matched the search.
find(query, sens=False, asc=True, sortby=None, reverse=False) -> Simple query like search.
Args:
query (str): The query to use.
sens (bool, optional): Case sensitive. Defaults to False.
asc (bool, optional): Ascii conversion before matching, this matches text like 'cafe' and 'café'. Defaults to True.
sortby (string, optional): Sort using the provided field.
reverse (bool, optional): Reverse sort. Defaults to False.
Returns:
List with the keys of the documents that matched the search.
Owner
Pedro Gonring
Pedro Gonring
A fast ordered NoSQL database.

MerkavaDB Note This is still in active development. Things will change. If you are interested in helping out, or would like to see any particular feat

Adam Hopkins 6 Sep 29, 2022
PathfinderMonsterDatabase - A database of all monsters in Pathfinder 1e, created by parsing aonprd.com

PathfinderMonsterDatabase A database of all monsters in Pathfinder 1e, created by parsing aonprd.com Setup Run the following line to install all requi

Yoni Lerner 11 Jun 12, 2022
Simple embedded in memory json database

dbj dbj is a simple embedded in memory json database. It is easy to use, fast and has a simple query language. The code is fully documented, tested an

Pedro Gonring 25 Aug 12, 2022
MyReplitDB - the most simplistic and easiest wrapper to use for replit's database system.

MyReplitDB is the most simplistic and easiest wrapper to use for replit's database system. Installing You can install it from the PyPI Or y

kayle 4 Jul 03, 2022
Python function to extract all the rows from a SQLite database file while iterating over its bytes, such as while downloading it

Python function to extract all the rows from a SQLite database file while iterating over its bytes, such as while downloading it

Department for International Trade 16 Nov 09, 2022
This is a simple graph database in SQLite, inspired by

This is a simple graph database in SQLite, inspired by "SQLite as a document database".

Denis Papathanasiou 1.2k Jan 03, 2023
Enfilade: Tool to Detect Infections in MongoDB Instances

Enfilade: Tool to Detect Infections in MongoDB Instances

Aditya K Sood 7 Feb 21, 2022
AWS Tags As A Database is a Python library using AWS Tags as a Key-Value database.

AWS Tags As A Database is a Python library using AWS Tags as a Key-Value database. This database is completely free* 💸

Oren Leung 42 Nov 25, 2022
A Persistent Embedded Graph Database for Python

Cog - Embedded Graph Database for Python cogdb.io New release: 2.0.5! Installing Cog pip install cogdb Cog is a persistent embedded graph database im

Arun Mahendra 214 Dec 30, 2022
Shelf DB is a tiny document database for Python to stores documents or JSON-like data

Shelf DB Introduction Shelf DB is a tiny document database for Python to stores documents or JSON-like data. Get it $ pip install shelfdb shelfquery S

Um Nontasuwan 35 Nov 03, 2022
This repo contains the backend of the KMK project

KMK Backend This repository contains the backend part of the KMK project Demo Watch it on Youtube Getting started Pre-commit hooks After you cloned th

21 Nov 26, 2022
A super easy, but really really bad DBMS

Dumb DB Are you looking for a reliable database management system? Then you've come to the wrong place. This is a very small database management syste

Elias Amha 5 Dec 28, 2022
Youtube Kanalinda tanittigim ve Programladigim SQLite3 ile calisan Kütüphane Programi

SQLite3 Kütüphane Uygulamasi SQLite3 ile calisan Kütüphane Arayüzü Yükleme Yerel veritabani olusacaktir. Yaptiginiz islemler kaybolmaz! Temel Gereksin

Mikael Pikulski 6 Aug 13, 2022
Simple json type database for python3

What it is? Simple json type database for python3! What about speed? The speed is great! All data is stored in RAM until saved. How to install? pip in

3 Feb 11, 2022
EmployeeDB - Advanced Redis search functionalities on Python applied on an Employee management backend app

EmployeeDB - Advanced Redis search functionalities on Python applied on an Employee management backend app

Ahmad Bazzi 58 Oct 10, 2022
ClutterDB - Extremely simple JSON database made for infrequent changes which behaves like a dict

extremely simple JSON database made for infrequent changes which behaves like a dict this was made for ClutterBot

Clutter Development 1 Jan 12, 2022
A Simple , ☁️ Lightweight , 💪 Efficent JSON based database for 🐍 Python.

A Simple, Lightweight, Efficent JSON based DataBase for Python The current stable version is v1.6.1 pip install pysondb==1.6.1 Support the project her

PysonDB 282 Jan 07, 2023
Given a metadata file with relevant schema, an SQL Engine can be run for a subset of SQL queries.

Mini-SQL-Engine Given a metadata file with relevant schema, an SQL Engine can be run for a subset of SQL queries. The query engine supports Project, A

Prashant Raj 1 Dec 03, 2021
pickleDB is an open source key-value store using Python's json module.

pickleDB pickleDB is lightweight, fast, and simple database based on the json module. And it's BSD licensed! pickleDB is Fun import pickledb

Harrison Erd 738 Jan 04, 2023
Tools for analyzing Git history using SQLite

git-history Tools for analyzing Git history using SQLite Installation Install this tool using pip: $ pip install git-history Usage This tool can be r

Simon Willison 128 Jan 02, 2023