A reusable Django model field for storing ad-hoc JSON data

Overview

jsonfield

https://circleci.com/gh/rpkilby/jsonfield.svg?style=shield

jsonfield is a reusable model field that allows you to store validated JSON, automatically handling serialization to and from the database. To use, add jsonfield.JSONField to one of your models.

Note: django.contrib.postgres now supports PostgreSQL's jsonb type, which includes extended querying capabilities. If you're an end user of PostgreSQL and want full-featured JSON support, then it is recommended that you use the built-in JSONField. However, jsonfield is still useful when your app needs to be database-agnostic, or when the built-in JSONField's extended querying is not being leveraged. e.g., a configuration field.

Requirements

jsonfield aims to support all current versions of Django, however the explicity tested versions are:

  • Python: 3.6, 3.7, 3.8
  • Django: 2.2, 3.0

Installation

pip install jsonfield

Usage

from django.db import models
from jsonfield import JSONField

class MyModel(models.Model):
    json = JSONField()

Querying

As stated above, JSONField is not intended to provide extended querying capabilities. That said, you may perform the same basic lookups provided by regular text fields (e.g., exact or regex lookups). Since values are stored as serialized JSON, it is highly recommended that you test your queries to ensure the expected results are returned.

Handling null values

A model field's null argument typically controls whether null values may be stored in its column by setting a not-null constraint. However, because JSONField serializes its values (including nulls), this option instead controls how null values are persisted. If null=True, then nulls are not serialized and are stored as a null value in the database. If null=False, then the null is instead stored in its serialized form.

This in turn affects how null values may be queried. Both fields support exact matching:

MyModel.objects.filter(json=None)

However, if you want to use the isnull lookup, you must set null=True.

class MyModel(models.Model):
    json = JSONField(null=True)

MyModel.objects.filter(json__isnull=True)

Note that as JSONField.null does not prevent nulls from being stored, achieving this must instead be handled with a validator.

Advanced Usage

By default python deserializes json into dict objects. This behavior differs from the standard json behavior because python dicts do not have ordered keys. To overcome this limitation and keep the sort order of OrderedDict keys the deserialisation can be adjusted on model initialisation:

import collections

class MyModel(models.Model):
    json = JSONField(load_kwargs={'object_pairs_hook': collections.OrderedDict})

Other Fields

jsonfield.JSONCharField

Subclasses models.CharField instead of models.TextField.

Running the tests

The test suite requires tox.

$ pip install tox

Then, run the tox command, which will run all test jobs.

$ tox

Or, to test just one job (for example Django 2.0 on Python 3.6):

$ tox -e py36-django20

Release Process

  • Update changelog
  • Update package version in setup.py
  • Check supported versions in setup.py and readme
  • Create git tag for version
  • Upload release to PyPI test server
  • Upload release to official PyPI server
$ pip install -U pip setuptools wheel twine
$ rm -rf dist/ build/
$ python setup.py sdist bdist_wheel
$ twine upload -r test dist/*
$ twine upload dist/*

Changes

Take a look at the changelog.

Comments
  • Problems withs PostGres 9.3, JSON Field and Query Distinct

    Problems withs PostGres 9.3, JSON Field and Query Distinct

    As mentionned in #47, using ".distinct()", which is used by the admin panel creating users but also in several other cases in django processes, triggers the same bug as #47 describes.

    A workaround has been found by @mkhattab : The work around for this in the Django admin is to subclass QuerySet and Manager and override the distinct method [to remove the JSON fields from it].

    What would be the definite fix ?

    opened by ewjoachim 20
  • Can not store ordinary Python strings in JSONfield

    Can not store ordinary Python strings in JSONfield

    Since version 0.9.4 I am not able to store ordinary Python strings in a JSONfield. Up to version 0.9.2 this worked without problems. Did I miss something? From my point of view, it should not matter, if I store ordinary strings, lists or dicts in a JSONfield.

    opened by jrief 20
  • AttributeError in Creator class

    AttributeError in Creator class

    I updated to Django 1.11 and I get AttributeError from subclassing.py:33.

    In Django 1.11 (django/db/models/options.py:890) the get method of the Creator class is implicitly called. The code in subclassing.py is a copy/paste from Django and seems to be <1.7 and Django's code has updated since then. Simply copying the code from Django 1.7 seems to solve my problems.

    opened by Majsvaffla 16
  • JSONField not deserialized when used from inside a polymorphic inherited model

    JSONField not deserialized when used from inside a polymorphic inherited model

    Hi,

    I have models like this:

    class BaseFeed(PolymorphicModel):
        errors = JSONField(default=list, blank=True)
    
    
    class RssAtomFeed(BaseFeed):
        pass
    

    Note: polymorphic is here. Say I do:

    f1 = BaseFeed.objects.create()
    f2 = RssAtomFeed.objects.create()
    

    Then I got:

    f1.errors
    []
    
    f2.errors
    u'[]'
    

    And this is pretty annoying because it will obviously fail in many places because all my code expects a list.

    I have read #92, but I didn't find any obvious/easy clue on how to solve this issue.

    Any other hint I can try?

    For the record, my models are much complex than this. You can view the base feed here and the rss/atom feed here. I've tried with or without default, with or without load_kwargs={'object_pairs_hook': collections.OrderedDict}, this doesn't change anything.

    I've tried the other JSONField implementation and the problem doesn't occur, even after a south migration (I'm on Django 1.6).

    regards,

    opened by Karmak23 11
  • Problem when iterating if default value is '

    Problem when iterating if default value is '""' on postgresql-9.3

    In [1]: from my_app.models import MyModel

    In [2]: [fo for fo in MyModel.objects.all()]

    ValidationError Traceback (most recent call last) in () ----> 1 [fo for fo in MyModel.objects.all()]

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in iter(self) 94 - Responsible for turning the rows into model objects. 95 """ ---> 96 self._fetch_all() 97 return iter(self._result_cache) 98

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in _fetch_all(self) 852 def _fetch_all(self): 853 if self._result_cache is None: --> 854 self._result_cache = list(self.iterator()) 855 if self._prefetch_related_lookups and not self._prefetch_done: 856 self._prefetch_related_objects()

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/query.pyc in iterator(self) 228 obj = model_cls(*_dict(zip(init_list, row_data))) 229 else: --> 230 obj = model(_row_data) 231 232 # Store the source database of the object

    /home/florent/.virtualenvs//local/lib/python2.7/site-packages/django/db/models/base.pyc in init(self, _args, *_kwargs) 345 # without changing the logic. 346 for val, field in zip(args, fields_iter): --> 347 setattr(self, field.attname, val) 348 else: 349 # Slower, kwargs-ready version.

    /home/florent/Work//django-jsonfield/jsonfield/subclassing.pyc in set(self, obj, value) 39 # we can definitively tell if a value has already been deserialized 40 # More: https://github.com/bradjasper/django-jsonfield/issues/33 ---> 41 obj.dict[self.field.name] = self.field.pre_init(value, obj) 42 43

    /home/florent/Work//django-jsonfield/jsonfield/fields.pyc in pre_init(self, value, obj) 75 return json.loads(value, **self.load_kwargs) 76 except ValueError: ---> 77 raise ValidationError(_("Enter valid JSON")) 78 79 return value

    ValidationError: [u'Enter valid JSON']

    opened by toopy 11
  • Serialize ''(empty) string to empty database value

    Serialize ''(empty) string to empty database value

    Uses ""(empty) string in database to store empty json ([]).

    You can use empty value as default for your table fields. You will have no overhead in database if you have no data.

    opened by Romamo 11
  • empty string or null in db

    empty string or null in db

    Hello,

    Wouldn't it be nice to gracefully handle empty string or NULL when reading from db? It could default to empty dictionary: {}.

    I've added the field to my model manually to the DB table and it did not work with empty string I had to update my_table set settings='{}'.

    I know I can set the '{}' via the south migrations, but it would still be a nice default behavior.

    opened by adrianandreias 9
  • DBError: could not identify an equality operator for type json when annotating a model with JSONField

    DBError: could not identify an equality operator for type json when annotating a model with JSONField

    I'm working in Django 1.5.4 with PostgreSQL 9.3, and i get this error on a query like this:

    ModelWithJsonField.objects.annotate(num=Count('field_to_count_by'))
    

    Traceback:

    Traceback (most recent call last):
      File "<console>", line 1, in <module>
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 93, in __repr__
        data = list(self[:REPR_OUTPUT_SIZE + 1])
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 108, in __len__
        self._result_cache.extend(self._iter)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/query.py", line 317, in iterator
        for row in compiler.results_iter():
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 775, in results_iter
        for rows in self.execute_sql(MULTI):
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 840, in execute_sql
        cursor.execute(sql, params)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/util.py", line 41, in execute
        return self.cursor.execute(sql, params)
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 58, in execute
        six.reraise(utils.DatabaseError, utils.DatabaseError(*tuple(e.args)), sys.exc_info()[2])
      File "/Users/jondoe/Project/lib/python2.7/site-packages/django/db/backends/postgresql_psycopg2/base.py", line 54, in execute
        return self.cursor.execute(query, args)
    DatabaseError: could not identify an equality operator for type json
    

    Ideas? I saw another issue with similar error, going to try and find out what causes this.

    opened by neara 9
  • JSONField fails in MySQL

    JSONField fails in MySQL

    I recently added the JSONField to one of my Models and worked fine since both my local machine runs on Postgres and my Heroku instance too, since moving it over to production I have found that everytime I try and bring up an item containg a default value for MySQL it raises a validationerror.

    class Item(models.Model): ... ... json = JSONField(default='{}')

    raise ValidationError(msg) ValidationError: [u"'' is not a valid JSON string."]

    opened by appel268576 9
  • Issues between versions

    Issues between versions

    Hi,

    We have recently updated from jsonfield2 to jsonfield and we are experiencing issues when we have a dict data being passed in get_or_create().

    The idea is we were using jsonfield2, jsonfield2==3.0.2 and when we update to latest jsonfield==3.1.0 the issue appears. In order to reproduce it you need to have a model lets say:

    class MyModel:
        field_1 = models.CharField
        field_2 = JsonField()
    
       Meta:
          unique(field_1)
    
    MyModel.objects.create(field_1='test', field_2={'foo':'bar'})
    

    And while still on jsonfiield2==3.0.2 version create an entry in the database ( we use postgres ).

    Now uninstall jsonfield2 and install latest jsonfield and try to do

    MyModel.objects.get_or_create(field_1='test', field_2={'foo':'bar'}) - It will fail with IntegrityError and you will get "duplicate key value violates unique constraint" message. This is because if you just call MyModel.objects.get(field_1='test', field_2={'foo':'bar'}) - it raises ObjectDoesNotExist, and then the above tries to call create but there is already a entry in the database with those params and there is a unique constraint in our case.

    The most important thing is that this MyModel.objects.get(field_1='test', field_2={'foo':'bar'}) raises ObjectDoesNotExist when the entry clearly exists in the table

    opened by ghost 8
  • Support polymorphic objects

    Support polymorphic objects

    Fix polymorphic object issue: https://github.com/dmkoch/django-jsonfield/issues/101

    With polymorphic objects I get a AttributeError on obj.pk. obj.id works fine. pk and id has in the tests always the same value.

    opened by ubaumann 8
  • Don't autofill fields with null

    Don't autofill fields with null

    After 92613991d76429c65bd35aebea3470c0baf26520, a blank field is automatically populated with null. This is bad for null=True fields, since those should be left empty when unpopulated. This PR preserves the behaviour from before.

    opened by quantum5 0
  • Update tests

    Update tests

    I was testing some PR that you have for jsonfield

    https://github.com/rpkilby/jsonfield/pull/262 This fixes the warnings and the name of a file that causes the circle ci test to fail.

    https://github.com/rpkilby/jsonfield/pull/261 And this one adds test for django 3.1 and 3.2

    I think it would be nice to add them to the main branch. I tried both PR together and work fine (https://app.circleci.com/pipelines/github/MaferMazu/jsonfield?branch=mfmz%2Ffix-ci-and-test-django3.2)

    Hopefully you can add them so that the library remains updated.

    Any doubt or advance I am at your service.

    Thanks for your consideration.

    opened by MaferMazu 0
  • Filter warnings from Django during capture.

    Filter warnings from Django during capture.

    Django has warnings in asyncio that are also captured in the tests. Filter warnings from django.utils.asyncio module. This was fixed in Django but not yet released : https://github.com/django/django/commit/623c8cd8f41a99f22d39b264f7eaf7244417000b .

    Fedora issue: https://bugzilla.redhat.com/show_bug.cgi?id=1962449

    opened by tirkarthi 1
  • docs: fix simple typo, explicity -> explicitly

    docs: fix simple typo, explicity -> explicitly

    There is a small typo in README.rst.

    Should read explicitly rather than explicity.

    Semi-automated pull request generated by https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md

    opened by timgates42 0
  • Not enough space to persist data

    Not enough space to persist data

    I was trying to save a big json object but i couldn't accomplish it since this field generate a Long type field on my Oracle DB. there is a solution to do it or it is neccesary apply a change.

    opened by jasonlazo 0
Owner
Ryan P Kilby
beep boop
Ryan P Kilby
PEP-484 stubs for Django

pep484 stubs for Django This package contains type stubs and a custom mypy plugin to provide more precise static types and type inference for Django f

TypedDjango 1.1k Dec 30, 2022
A Django web application that allows you to be in the loop about everything happening in your neighborhood.

A Django web application that allows you to be in the loop about everything happening in your neighborhood. From contact information of different handyman to meeting announcements or even alerts.

Kennedy Ngugi Mwaura 3 Dec 11, 2022
Simple reproduction of connection leak with celery/django/gevent

Redis connection leak with celery/django/gevent Reproduces celery issue at https://github.com/celery/celery/issues/6819 using gevented django web serv

2 Apr 03, 2022
Django Starter is a simple Skeleton to start with a Django project.

Django Starter Template Description Django Starter is a simple Skeleton to start

Numan Ibn Mazid 1 Jan 10, 2022
This is raw connection between redis server and django python app

Django_Redis This repository contains the code for this blogpost. Running the Application Clone the repository git clone https://github.com/xxl4tomxu9

Tom Xu 1 Sep 15, 2022
A Redis cache backend for django

Redis Django Cache Backend A Redis cache backend for Django Docs can be found at http://django-redis-cache.readthedocs.org/en/latest/. Changelog 3.0.0

Sean Bleier 1k Dec 15, 2022
Silk is a live profiling and inspection tool for the Django framework.

Silk is a live profiling and inspection tool for the Django framework. Silk intercepts and stores HTTP requests and database queries before presenting them in a user interface for further inspection:

Jazzband 3.7k Jan 02, 2023
Rosetta is a Django application that eases the translation process of your Django projects

Rosetta Rosetta is a Django application that facilitates the translation process of your Django projects. Because it doesn't export any models, Rosett

Marco Bonetti 909 Dec 26, 2022
Django Girls Tutorial Workshop

Django Girls Tutorial Workshop A log of activities during the workshop. this is an H2 git remote add origin https://github.com/ahuimanu/django_girls_t

Jeffry Babb 1 Oct 27, 2021
This is a personal django website for forum posts

Django Web Forum This is a personal django website for forum posts It includes login, registration and forum posts with date time. Tech / Framework us

5 May 12, 2022
Website desenvolvido em Django para gerenciamento e upload de arquivos (.pdf).

Website para Gerenciamento de Arquivos Features Esta é uma aplicação full stack web construída para desenvolver habilidades com o framework Django. O

Alinne Grazielle 8 Sep 22, 2022
A simple plugin to attach a debugger in Django on runserver command.

django-debugger A simple plugin to attach a debugger in Django during runserver Installation pip install django-debugger Usage Prepend django_debugger

Sajal Shrestha 11 Nov 15, 2021
django-dashing is a customisable, modular dashboard application framework for Django to visualize interesting data about your project. Inspired in the dashboard framework Dashing

django-dashing django-dashing is a customisable, modular dashboard application framework for Django to visualize interesting data about your project.

talPor Solutions 703 Dec 22, 2022
Developer-friendly asynchrony for Django

Django Channels Channels augments Django to bring WebSocket, long-poll HTTP, task offloading and other async support to your code, using familiar Djan

Django 5.5k Jan 06, 2023
An orgizational tool to keep track of tasks/projects and the time spent on them.

Django-Task-Manager Task Tracker using Python Django About The Project This project is an orgizational tool to keep track of tasks/projects and the ti

Nick Newton 1 Dec 21, 2021
Utility for working with recurring dates in Django.

django-recurrence django-recurrence is a utility for working with recurring dates in Django. Documentation is available at https://django-recurrence.r

408 Jan 06, 2023
A set of high-level abstractions for Django forms

django-formtools Django's "formtools" is a set of high-level abstractions for Django forms. Currently for form previews and multi-step forms. This cod

Jazzband 621 Dec 30, 2022
Full control of form rendering in the templates.

django-floppyforms Full control of form rendering in the templates. Authors: Gregor Müllegger and many many contributors Original creator: Bruno Renié

Jazzband 811 Dec 01, 2022
Getdp-project - A Django-built web app that generates a personalized banner of events to come

getdp-project https://get-my-dp.herokuapp.com/ A Django-built web app that gener

CODE 4 Aug 01, 2022
Basic implementation of Razorpay payment gateway 💳 with Django

Razorpay Payment Integration in Django 💥 In this project Razorpay payment gateway 💳 is integrated with Django by breaking down the whole process int

ScaleReal 12 Dec 12, 2022