Django models and endpoints for working with large images -- tile serving

Last update: Dec 17, 2022

Comments

EPSG and its effect on COG and Tiles Opacity

Great work with the library.

I have an issue with opacity/clarity of tiles, thumbnails and overall Geotiffs/COGs as well as COGs not getting tiled.

Opacity Issue: When using the swagger API to get a thumbnail of an existing COG, if I include "EPSG:3857" on the request, I get a washed out image

However, if I omit the EPSG Code, I get a clear image thumbnail. I have tested with multiple COGs and raw GeoTiffs

In Django Admin, the same Geotiffs and COGs are also washed out

I have also used a separate Leaflet html page to access the /api/maps/{id}/{z}/{x}/{y}.png and I get the same washed out images

<html>
  <head>
    <meta name="viewport" content="initial-scale=1.0, user-scalable=no" />
    <meta charset="utf-8" />
    <style>
      html,
      body {
        height: 100%;
        margin: 0;
        padding: 0;
      }
      #map {
        height: 100%;
      }
    </style>
    <link
      rel="stylesheet"
      href="https://unpkg.com/[email protected]/dist/leaflet.css"
      integrity="sha512-hoalWLoI8r4UszCkZ5kL8vayOGVae1oxXe/2A4AO6J9+580uKHDO3JdHb7NzwwzK5xr/Fs0W40kiNHxM9vyTtQ=="
      crossorigin=""
    />
    <script
      src="https://unpkg.com/[email protected]/dist/leaflet.js"
      integrity="sha512-BB3hKbKWOc9Ez/TAwyWxNXeoV9c1v6FIeYiBieIWkpLjauysF18NzgR1MBNBXf8/KABdlkX68nAhlwcDFLGPCQ=="
      crossorigin=""
    ></script>
  </head>
  <body>
    <div id="map"></div>
    <script>
      var map = L.map("map").setView(
        [-0.24511062205544606, 34.87011481076246],
        16
      );
      L.tileLayer("https://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png", {
        attribution:
          '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors',
        maxZoom: 18,
      }).addTo(map);
      L.tileLayer(
        "http://localhost:8000/api/maps/1/tiles/{z}/{x}/{y}.png?projection=EPSG%3A3857",
        {
          attribution:
            '&copy; <a href="https://www.openstreetmap.org/copyright">OpenStreetMap</a> contributors',
          maxZoom: 23,
          opacity: 1,
        }
      ).addTo(map);

    </script>
  </body>
</html>

However, in the above example, if I do not append the EPSG code on the url in html, then I get an error (no map rendered) and a 400 bad request

As for individual tiles, if I include the EPSG Code in the request body in swagger UI, i get a washed out tile

However, if I omit the EPSG code, I get a bad request error

An example COG metadata is

{
  "geospatial": true,
  "levels": 8,
  "sizeX": 32465,
  "sizeY": 28586,
  "sourceLevels": 8,
  "sourceSizeX": 32465,
  "sourceSizeY": 28586,
  "tileWidth": 256,
  "tileHeight": 256,
  "bounds": {
    "ll": {
      "x": 34.85392278135806,
      "y": -0.23501171685313998
    },
    "ul": {
      "x": 34.85392278135806,
      "y": -0.2273338589976692
    },
    "lr": {
      "x": 34.86258419062559,
      "y": -0.23501171685313998
    },
    "ur": {
      "x": 34.86258419062559,
      "y": -0.2273338589976692
    },
    "srs": "EPSG:4326",
    "xmin": 34.85392278135806,
    "xmax": 34.86258419062559,
    "ymin": -0.23501171685313998,
    "ymax": -0.2273338589976692
  },
  "sourceBounds": {
    "ll": {
      "x": 34.85392278135806,
      "y": -0.23501171685313998
    },
    "ul": {
      "x": 34.85392278135806,
      "y": -0.2273338589976692
    },
    "lr": {
      "x": 34.86258419062559,
      "y": -0.23501171685313998
    },
    "ur": {
      "x": 34.86258419062559,
      "y": -0.2273338589976692
    },
    "srs": "+proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs",
    "xmin": 34.85392278135806,
    "xmax": 34.86258419062559,
    "ymin": -0.23501171685313998,
    "ymax": -0.2273338589976692
  },
  "bands": {
    "1": {
      "min": 55,
      "max": 255,
      "mean": 193.05740298087738,
      "stdev": 72.13546456636203,
      "units": "metre",
      "interpretation": "red",
      "maskband": 4
    },
    "2": {
      "min": 63,
      "max": 255,
      "mean": 194.14802446569178,
      "stdev": 70.6158617103163,
      "units": "metre",
      "interpretation": "green",
      "maskband": 4
    },
    "3": {
      "min": 41,
      "max": 255,
      "mean": 182.4484673790776,
      "stdev": 84.39825325395805,
      "units": "metre",
      "interpretation": "blue",
      "maskband": 4
    },
    "4": {
      "min": 0,
      "max": 255,
      "mean": 109.17366774465692,
      "stdev": 126.17604981162934,
      "units": "metre",
      "interpretation": "alpha"
    }
  },
  "magnification": null,
  "mm_x": 2.969893339423529,
  "mm_y": 2.969893339423529,
  "proj4": [
    "+proj=longlat +ellps=WGS84 +towgs84=0,0,0,0,0,0,0 +no_defs"
  ],
  "frames": false
}

bug

opened by geoffreynyaga 7

Add back Base64 styling support

This reverts commit 3c74b7f37a5d91f68048fe5a907eedb2ead734f0.

Take the following style object:

{
    "bands": [
        {
            "frame": 0,
            "palette": "#ffffff"
        },
        {
            "frame": 1,
            "palette": "#ffffff"
        },
        {
            "frame": 2,
            "palette": "#ffffff"
        },
        {
            "frame": 3,
            "palette": "#ffffff"
        },
        {
            "frame": 4,
            "palette": "#ffffff"
        },
        {
            "frame": 5,
            "palette": "#ffffff"
        },
        {
            "frame": 6,
            "palette": "#ffffff"
        },
        {
            "frame": 7,
            "palette": "#ffffff"
        },
        {
            "frame": 8,
            "palette": "#ffffff"
        },
        {
            "frame": 9,
            "palette": "#ffffff"
        },
        {
            "frame": 10,
            "palette": "#ffffff"
        },
        {
            "frame": 11,
            "palette": "#ffffff"
        }
    ]
}

Base64 encoding of this object is smaller than the URI encoded version (and is so in all similar examples I have tried)

JSON Stringify (length 397):

{"bands":[{"frame":0,"palette":"#ffffff"},{"frame":1,"palette":"#ffffff"},{"frame":2,"palette":"#ffffff"},{"frame":3,"palette":"#ffffff"},{"frame":4,"palette":"#ffffff"},{"frame":5,"palette":"#ffffff"},{"frame":6,"palette":"#ffffff"},{"frame":7,"palette":"#ffffff"},{"frame":8,"palette":"#ffffff"},{"frame":9,"palette":"#ffffff"},{"frame":10,"palette":"#ffffff"},{"frame":11,"palette":"#ffffff"}]}

Base64 (length 532):

eyJiYW5kcyI6W3siZnJhbWUiOjAsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MSwicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjoyLCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjMsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6NCwicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjo1LCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjYsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6NywicGFsZXR0ZSI6IiNmZmZmZmYifSx7ImZyYW1lIjo4LCJwYWxldHRlIjoiI2ZmZmZmZiJ9LHsiZnJhbWUiOjksInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MTAsInBhbGV0dGUiOiIjZmZmZmZmIn0seyJmcmFtZSI6MTEsInBhbGV0dGUiOiIjZmZmZmZmIn1dfQ==

encodeURIComponent (length 721):

%7B%22bands%22%3A%5B%7B%22frame%22%3A0%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A1%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A2%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A3%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A4%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A5%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A6%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A7%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A8%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A9%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A10%2C%22palette%22%3A%22%23ffffff%22%7D%2C%7B%22frame%22%3A11%2C%22palette%22%3A%22%23ffffff%22%7D%5D%7D

If syles get more complex, they may quickly approach the common limit of 1024 characters for query strings. Having support for Base64 will give users a little more wiggle room to make sure their styles don't exceed limits set by their web server.

Related: https://github.com/atlascope/atlascope/pull/139#pullrequestreview-946184928

opened by banesullivan 2

Use django cache with large-image
requires https://github.com/girder/large_image/pull/876

resolves #2

To do:

[x] add some sort of test that will validate that large-image is really using the Django cache
opened by banesullivan 1
Use custom format handler and regroup endpoints

Resolves #22

~This is still a work in progress as these changes highlight an issue in drf-yasg I need to fully document/capture~ not addressing this

This introduces breaking changes to the API endpoints:

Endpoints are now grouped into info, data, and tiles subgroups.

I could implement a deprecated version of the API, but considering we're still in the early days and most projects are pinning versions. This shouldn't matter too much and I will do the migration manually

opened by banesullivan 1
Add specification of color to UI palette choice

The palette UI currently only allows for the selection of an MPL colormap - modify this to support hex colors (and/or everything large-image can support)

opened by banesullivan 1
Can we embed warnings/error in tile response body?
Sometimes we get warnings like:

Tiff image is missing many lower resolution levels (3). It will be inefficient to read lower resolution tiles.

ref https://sentry.io/organizations/django-large-image/issues/3131476233/?project=6272210&query=is%3Aunresolved

and in the future, we may warn if an image is not a properly tiled COG or pyramidal tiff (see #4).

It would be great if we could embed these warnings in the tile and/or metadata endpoint responses. While in many cases, these warnings will go unnoticed, in some of our custom applications we can monitor these and even create handlers to provide a toast to the user that says something along the lines of "Performance warning: -insert warning-"
opened by banesullivan 1
Performance issues

I'm seeing significant poor performance on the production demo. This could be due to the images not being proper COGs or something odd with how files are being checked out, causing requests to not be parallel

opened by banesullivan 1
FileExistsError with os.symlink
https://github.com/girder/django-large-image/blob/43c9ef62d83c4d6f33d3b49cc41f2c678f296584/django_large_image/utilities.py#L137

FileExistsError: [Errno 17] File exists: '/home/geoff/LargeImage/media/data/test.tif' -> '/tmp/django-large-image/file_cache/ImageFile-1/test.tif'
bug
opened by banesullivan 0
Add renderers and format extension handling
Can we add renderers (see details at bottom) and remove the duplicate _png, _jpeg, and _tif variants of many of the endpoints so that they are one endpoint that accepts different formats? I know this is possible, but I haven't been able to get it working with DRF 😕

For example, we have the thumbnail endpoint split as two endpoints:

/image-file/{id}/thumbnail.png

/image-file/{id}/thumbnail.jpeg

Can we combine these to: /image-file/{id}/thumbnail.{format}

Thus reducing the method shims for these endpoints like

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L38

and

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L48

So that we are only using

https://github.com/girder/django-large-image/blob/09ef6e823d78de9a304cebbba453cb4a46b53ac2/django_large_image/rest/data.py#L25

with the format argument.

So far I have been unsuccessful in getting this to work. What's annoying is that the default .json and .api "formats" work for these endpoints, but I cannot add any custom formats/renderers like .png, .jpeg, or .tif

from rest_framework.renderers import BaseRenderer as RFBaseRenderer class BaseRenderer(RFBaseRenderer): render_style = 'binary' charset = None def render(self, data, media_type=None, renderer_context=None): return data class PNGRenderer(BaseRenderer): media_type = 'image/png' format = 'png' class JPEGRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'jpeg' class JPGRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'jpg' image_renderers = [PNGRenderer, JPEGRenderer, JPGRenderer] class TifRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'tif' class TiffRenderer(BaseRenderer): media_type = 'image/jpeg' format = 'tiff' image_data_renderers = image_renderers + [TifRenderer, TiffRenderer]
opened by banesullivan 0
Change VSI failed logging to warning

https://github.com/ResonantGeoData/django-large-image/blob/87b6175cb08aff01367ff264815c5891fe9c5411/django_large_image/rest/core.py#L86

This is showing up as an error on Sentry. I should change this to a warning and perhaps more gracefully handle it to ensure only GDAL is used with this - might come with #11

opened by banesullivan 0
Style min/max value of "" is not valid; using "auto"

https://sentry.io/organizations/django-large-image/issues/3126500235/?project=6272210&query=is%3Aunresolved

Fix handling of empty min/max values in query parameters to avoid this warning

opened by banesullivan 0
Add COG validation endpoint

Large-image's GDAL source has a validateCOG method which should be wrapped into an endpoint to validate if an image is a COG

https://girder.github.io/large_image/_build/large_image_source_gdal/large_image_source_gdal.html#large_image_source_gdal.GDALFileTileSource.validateCOG

This will not be available/valid for all tile sources, so we will need a clever way to limit it to GDAL sources and error out when not a GDAL source

opened by banesullivan 0
Set min/max help text to value range
The min/max fields in the UI do not show the min/max range for each band and it can be tough to decipher what the value range is.

It would be great if we could do two things:

Set the help text to include the actual min/max values for the selected band

When initially hitting the up/down arrows, it is incremented from the actual min/max values
opened by banesullivan 0
`frames` in `/info/metadata` but not in `/tiles/metadata`

See https://github.com/atlascope/atlascope/pull/173#issuecomment-1163453575

Need to make sure the tiles_metadata endpoint is using the metadata wrapper in the tilesource module and not grabbing that directly from large_image

opened by banesullivan 0

Releases(0.8.0)

0.8.0(Jun 17, 2022)
New Features

Notable

Major UI overhaul

Copy metadata button

Modern UI styling

Advanced multi-band compositing with custom colors

Connects large-image's caching mechanisms to utilize the Django cache framework

Specify the named cache as the LARGE_IMAGE_CACHE_NAME, otherwise uses the 'default' cache

New tiffdump endpoint utilizing tifftools to extract information from Tiff files

Support for URI encoded styles in request URL parameters (documented in README)

Improved internal mechanisms for opening tile sources for easier overrides downstream

User can specify the source class to use through the source URL parameter

Better format handing for endpoints that serve image data (single endpoint where PNG, JPEG, or TIFF are specified for the desired format)

Additional

Support max_width and max_height specification for generating thumbnails

Improvements to SSR template overriding

mypy type annotations throughout API

New TileMetadataSerializer for the tiles/metadata endpoint

Improved API error handling

Better OpenAPI RESTful arguments documentation

Demos

Standalone Docker image demo app: https://github.com/girder/django-large-image/pkgs/container/django-large-image-demo

Usage with django-raster: https://github.com/ResonantGeoData/django-raster-demo

Noted support for GeoDjango's GDALRaster in README

Document how to convert Pyramidal Tiffs (COGs) with Celery tasks

Deprecations & Changes

internal_metadata -> metadata_internal

Metadata endpoints are now under the info/ prefix

e.g., metadata -> info/metadata

Data endpoints are now under the data/ prefix

e.g., thumbnail.png -> data/thumbnail.(?P<fmt>png|jpg|jpeg)

Source code(tar.gz)
Source code(zip)
0.4.0(Apr 10, 2022)
This release has breaking changes

There are now 4 ViewSet mixin classes for use with both detail and non-detail viewsets in DRF.

The following are the provided mixin classes and their use case:

LargeImageMixin: for use with a standard, non-detail ViewSet. Users must implement get_path()

LargeImageDetailMixin: for use with a detail viewset like GenericViewSet. Users must implement get_path()

LargeImageFileDetailMixin: (most commonly used) for use with a detail viewset like GenericViewSet where the associated model has a FileField storing the image data.

LargeImageVSIFileDetailMixin: (geospatial) for use with a detail viewset like GenericViewSet where the associated model has a FileField storing the image data that is intended to be read with GDAL. This will access the data over GDAL's Virtual File System interface (a VSI path).

Most users will want to use LargeImageFileDetailMixin

Other changes:

Style parameters can now be included as a JSON blob in the request body following the format specified by large-image

The thumbnail, tile, and region endpoints now each have .png, .jpeg, and .tif variants where appropriate

DRF ValidationErrors are raised for user-caused API errors (return 400 status on endpoint)

More usage examples

Source code(tar.gz)
Source code(zip)
0.2.4(Apr 2, 2022)
django-large-image is here to solve your image tile serving needs in Django! Take a look at the README for some example implementations and give it a try in your project with image data.

Highlights

Easy to use mixin interface

Rich set of RESTful endpoints to extract information from large image formats

Serve image tiles in the slippy maps standard (z/x/y.png)

Produce thumbnails of large images on the fly

Works with FileField interfaces

Works with URL files (through GDAL's VFS/VSI layer)

Included viewer for admin interface or server-rendered templates

OpenAPI documentation included

Source code(tar.gz)
Source code(zip)

Owner

Resonant GeoData

Geospatial data cataloging with Django

GitHub Repository

Simple XML-RPC and JSON-RPC server for modern Django

django-modern-rpc Build an XML-RPC and/or JSON-RPC server as part of your Django project. Major Django and Python versions are supported Main features

82 Dec 04, 2022

Django backend of Helium's planner application

Helium Platform Project Prerequisites Python (= 3.6) Pip (= 9.0) MySQL (= 5.7) Redis (= 3.2) Getting Started The Platform is developed using Pytho

17 Dec 14, 2022

Comprehensive Markdown plugin built for Django

Django MarkdownX Django MarkdownX is a comprehensive Markdown plugin built for Django, the renowned high-level Python web framework, with flexibility,

738 Dec 21, 2022

Buckshot++ is a new algorithm that finds highly stable clusters efficiently.

Buckshot++: An Outlier-Resistant and Scalable Clustering Algorithm. (Inspired by the Buckshot Algorithm.) Here, we introduce a new algorithm, which we

1 Jul 02, 2022

django+bootstrap5 实现的个人博客

项目状态：正在开发中【目前已基本可用】项目地址： https://github.com/find456789/django_blog django_blog django+bootstrap5 实现的个人博客特点文章的历史版本管理（随时回退） rss、atom markdown 评论功能

3 Nov 16, 2021

Media-Management with Grappelli

Django FileBrowser Media-Management with Grappelli. The FileBrowser is an extension to the Django administration interface in order to: browse directo

913 Dec 28, 2022

A slick ORM cache with automatic granular event-driven invalidation.

Cacheops A slick app that supports automatic or manual queryset caching and automatic granular event-driven invalidation. It uses redis as backend for

1.7k Jan 03, 2023

A drop-in replacement for django's ImageField that provides a flexible, intuitive and easily-extensible interface for quickly creating new images from the one assigned to the field.

django-versatileimagefield A drop-in replacement for django's ImageField that provides a flexible, intuitive and easily-extensible interface for creat

490 Dec 13, 2022

Create a netflix-like service using Django, React.js, & More.

Create a netflix-like service using Django. Learn advanced Django techniques to achieve amazing results like never before.

67 Dec 08, 2022

A prettier way to see Django requests while developing

35 Dec 02, 2022

django-reversion is an extension to the Django web framework that provides version control for model instances.

django-reversion django-reversion is an extension to the Django web framework that provides version control for model instances. Requirements Python 3

2.8k Jan 02, 2023

Realtime data read and write without page refresh using Ajax in Django.

Realtime read-write with AJAX Hey,this is the basic implementation type of ajax realtime read write from the database. where you can insert or view re

3 Dec 13, 2022

Django API without Django REST framework.

Django API without DRF This is a API project made with Django, and without Django REST framework. This project was done with: Python 3.9.8 Django 3.2.

3 Jan 19, 2022

Use heroicons in your Django and Jinja templates.

heroicons Use heroicons in your Django and Jinja templates. Requirements Python 3.6 to 3.9 supported. Django 2.2 to 3.2 supported. Are your tests slow

52 Dec 14, 2022

Atualizando o projeto APIs REST Django REST 2.0

APIs REST Django REST 3.0-KevinSoffa Atualização do projeto APIs REST Django REST 2.0-Kevin Soffa Melhorando e adicionando funcionalidades O que já fo

2 Dec 13, 2022

A handy tool for generating Django-based backend projects without coding. On the other hand, it is a code generator of the Django framework.

Django Sage Painless The django-sage-painless is a valuable package based on Django Web Framework & Django Rest Framework for high-level and rapid web

51 Sep 15, 2022

Django Audit is a simple Django app that tracks and logs requests to your application.

django-audit Django Audit is a simple Django app that tracks and logs requests to your application. Quick Start Install django-audit pip install dj-au

6 Dec 01, 2022

A simple Django middleware for Duo V4 2-factor authentication.

django-duo-universal-auth A lightweight middleware application that adds a layer on top of any number of existing authentication backends, enabling 2F

1 Jan 10, 2022

Organize Django settings into multiple files and directories. Easily override and modify settings. Use wildcards and optional settings files.

Organize Django settings into multiple files and directories. Easily override and modify settings. Use wildcards in settings file paths and mark setti

940 Jan 03, 2023

I managed to attach the Django Framework to my Telegram Bot and set a webhook

I managed to attach the Django Framework to my Telegram Bot and set a webhook. I've been developing it from 10th of November 2021 and I want to have a basic working prototype.

2 Sep 08, 2022

Django models and endpoints for working with large images -- tile serving

Related tags

Overview

Django Large Image