Read and write rasters in parallel using Rasterio and Dask

Last update: Aug 30, 2022

Related tags

Overview

dask-rasterio

dask-rasterio provides some methods for reading and writing rasters in parallel using Rasterio and Dask arrays.

Usage

Read a multiband raster

>>> from dask_rasterio import read_raster

>>> array = read_raster('tests/data/RGB.byte.tif')
>>> array
dask.array<stack, shape=(3, 718, 791), dtype=uint8, chunksize=(1, 3, 791)>

>>> array.mean()
dask.array<mean_agg-aggregate, shape=(), dtype=float64, chunksize=()>
>>> array.mean().compute()
40.858976977533935

Read a single band from a raster

>>> from dask_rasterio import read_raster

>>> array = read_raster('tests/data/RGB.byte.tif', band=3)
>>> array
dask.array<raster, shape=(718, 791), dtype=uint8, chunksize=(3, 791)>

Write a singleband or multiband raster

>>> from dask_rasterio import read_raster, write_raster

>>> array = read_raster('tests/data/RGB.byte.tif')

>>> new_array = array & (array > 100)
>>> new_array
dask.array<and_, shape=(3, 718, 791), dtype=uint8, chunksize=(1, 3, 791)>

>>> prof = ... # reuse profile from tests/data/RGB.byte.tif...
>>> write_raster('processed_image.tif', new_array, **prof)

Chunk size

Both read_raster and write_raster accept a block_size argument that acts as a multiplier to the block size of rasters. The default value is 1, which means the dask array chunk size will be the same as the block size of the raster file. You will have to adjust this value depending on the specification of your machine (how much memory do you have, and the block size of the raster).

Install

Install with pip:

pip install dask-rasterio

Development

This project is managed by Poetry. If you do not have it installed, please refer to Poetry instructions.

Now, clone the repository and run poetry install. This will create a virtual environment and install all required packages there.

Run poetry run pytest to run all tests.

Run poetry build to build package on dist/.

Issue tracker

Please report any bugs and enhancement ideas using the GitHub issue tracker:

https://github.com/dymaxionlabs/dask-rasterio/issues

Feel free to also ask questions on our Gitter channel, or by email.

Help wanted

Any help in testing, development, documentation and other tasks is highly appreciated and useful to the project.

For more details, see the file CONTRIBUTING.md.

License

Source code is released under a BSD-2 license. Please refer to LICENSE.md for more information.

Download and process satellite imagery in Python using Sentinel Hub services.

Description The sentinelhub Python package allows users to make OGC (WMS and WCS) web requests to download and process satellite images within your Py

659 Dec 23, 2022

An API built to format given addresses using Python and Flask.

An API built to format given addresses using Python and Flask. About The API returns properly formatted data, i.e. removing duplicate fields, distingu

1 Feb 27, 2022

This is a simple python code to get IP address and its location using python

IP address & Location finder @DEV/ED : Pavan Ananth Sharma Dependencies: ip2geotools Note: use pip install ip2geotools to install this in your termin

2 Jul 5, 2022

A package built to support working with spatial data using open source python

EarthPy EarthPy makes it easier to plot and manipulate spatial data in Python. Why EarthPy? Python is a generic programming language designed to suppo

414 Dec 23, 2022

Using SQLAlchemy with spatial databases

GeoAlchemy GIS Support for SQLAlchemy. Introduction GeoAlchemy is an extension of SQLAlchemy. It provides support for Geospatial data types at the ORM

109 Dec 1, 2022

Solving the Traveling Salesman Problem using Self-Organizing Maps

Solving the Traveling Salesman Problem using Self-Organizing Maps This repository contains an implementation of a Self Organizing Map that can be used

3.1k Dec 31, 2022

Example of animated maps in matplotlib + geopandas using entire time series of congressional district maps from UCLA archive. rendered, interactive version below

5 May 18, 2022

Hapi is a Python library for building Conceptual Distributed Model using HBV96 lumped model & Muskingum routing method

Current build status All platforms: Current release info Name Downloads Version Platforms Hapi - Hydrological library for Python Hapi is an open-sourc

15 Dec 26, 2022

Daily social mapping project in November 2021. Maps made using PyGMT whenever possible.

20 Nov 24, 2022

Comments

the time to write a dask array in tif is too long?

I have a image which have 40000cols and 40000 rows ,while I use data = read_raster(filename, band=1), and the use witer_raster(out_filename, data), it need about one minutes, I want to know can it be quickly

opened by DeZhao-Zhang 2
Add a Gitter chat badge to README.md

dymaxionlabs/dask-rasterio now has a Chat Room on Gitter

@munshkr has just created a chat room. You can visit it here: https://gitter.im/dymaxionlabs/dask-rasterio.

This pull-request adds this badge to your README.md:

If my aim is a little off, please let me know.

Happy chatting.

PS: Click here if you would prefer not to receive automatic pull-requests from Gitter in future.

opened by gitter-badger 1
Does dask-rasterio support masked array?
I'm working with dask masked array, and was wondering what would be the translation of these lines?

import rasterio with rasterio.open(inputFile) as source: # this is a 3D numpy array, with dimensions [band, row, col] src_array = source.read(masked=True)

Thank you for your cool lib!
opened by Becheler 0

TypeError: self._hds cannot be converted to a Python object for pickling

Seems that rasterio's _hds object is no more serializable

distributed.protocol.pickle - INFO - Failed to serialize ("('filled-2f9fe0560be0502eda038fa941309294', 0, 0)", <dask_rasterio.write.RasterioDataset object at 0x7f8f9deac828>, (slice(0, 748, None), slice(0, 22415, None)), <unlocked _thread.lock object at 0x7f8f9cb2af58>, False). Exception: self._hds cannot be converted to a Python object for pickling
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
~/miniconda3/envs/jupyter/lib/python3.6/site-packages/distributed/protocol/pickle.py in dumps(x)
     37     try:
---> 38         result = pickle.dumps(x, protocol=pickle.HIGHEST_PROTOCOL)
     39         if len(result) < 1000:

~/miniconda3/envs/jupyter/lib/python3.6/site-packages/rasterio/_io.cpython-36m-x86_64-linux-gnu.so in rasterio._io.DatasetWriterBase.__reduce_cython__()

TypeError: self._hds cannot be converted to a Python object for pickling

opened by arkanoid87 3

Releases(0.2.1)

0.2.1(Jun 9, 2018)
Fixes a test assertion

Source code(tar.gz)
Source code(zip)
0.2.0(Jun 9, 2018)
Add band keyword argument to read_raster

Source code(tar.gz)
Source code(zip)
0.1.0(Jun 6, 2018)

Source code(tar.gz)
Source code(zip)

Owner

Dymaxion Labs

Creating new value from geospatial imagery with deep learning

GitHub Repository

Python renderer for OpenStreetMap with custom icons intended to display as many map features as possible

Map Machine project consists of Python OpenStreetMap renderer: SVG map generation, SVG and PNG tile generation, Röntgen icon set: unique CC-BY 4.0 map

0 Dec 18, 2022

Geospatial Image Processing for Python

GIPPY Gippy is a Python library for image processing of geospatial raster data. The core of the library is implemented as a C++ library, libgip, with

83 Aug 19, 2022

Create Siege configuration files from Cloud Optimized GeoTIFF.

cogeo-siege Documentation: Source Code: https://github.com/developmentseed/cogeo-siege Description Create siege configuration files from Cloud Optimiz

3 Dec 01, 2022

Introduction to Geospatial Analysis in Python

Introduction to Geospatial Analysis in Python This repository is in support of a talk on geospatial data. Data To recreate all of the examples, the da

6 Oct 19, 2022

Python Data. Leaflet.js Maps.

folium Python Data, Leaflet.js Maps folium builds on the data wrangling strengths of the Python ecosystem and the mapping strengths of the Leaflet.js

6k Jan 02, 2023

ProjPicker (projection picker) is a Python module that allows the user to select all coordinate reference systems (CRSs)

ProjPicker ProjPicker (projection picker) is a Python module that allows the user to select all coordinate reference systems (CRSs) whose extent compl

4 Feb 06, 2022

Python bindings and utilities for GeoJSON

geojson This Python library contains: Functions for encoding and decoding GeoJSON formatted data Classes for all GeoJSON Objects An implementation of

763 Dec 26, 2022

Spectral decomposition for characterizing long-range interaction profiles in Hi-C maps

Inspectral Spectral decomposition for characterizing long-range interaction prof

6 Dec 13, 2022

This repository contains the scripts to derivate the ENU and ECEF coordinates from the longitude, latitude, and altitude values encoded in the NAD83 coordinates.

1 Feb 07, 2022

A Django application that provides country choices for use with forms, flag icons static files, and a country field for models.

Django Countries A Django application that provides country choices for use with forms, flag icons static files, and a country field for models. Insta

1.2k Jan 03, 2023

Interactive Maps with Geopandas

Create Interactive maps 🗺️ with your geodataframe Geopatra extends geopandas for interactive mapping and attempts to wrap the goodness of amazing map

46 Aug 16, 2022

A Python package for delineating nested surface depressions from digital elevation data.

Welcome to the lidar package lidar is Python package for delineating the nested hierarchy of surface depressions in digital elevation models (DEMs). I

166 Jan 03, 2023

Fiona reads and writes geographic data files

Fiona Fiona reads and writes geographic data files and thereby helps Python programmers integrate geographic information systems with other computer s

987 Jan 04, 2023

Calculate & view the trajectory and live position of any earth-orbiting satellite

satellite-visualization A cross-platform application to calculate & view the trajectory and live position of any earth-orbiting satellite in 3D. This

3 Jan 08, 2022

When traveling in the backcountry during winter time, updating yourself on current and recent weather data is important to understand likely avalanche danger.

Weather Data When traveling in the backcountry during winter time, updating yourself on current and recent weather data is important to understand lik

0 Jan 02, 2022

Zora is a python program that searches for GeoLocation info for given CIDR networks , with options to search with API or without API

Zora Zora is a python program that searches for GeoLocation info for given CIDR networks , with options to search with API or without API Installing a

1 Oct 26, 2021

LicenseLocation - License Location With Python

LicenseLocation Hi,everyone! ❤ 🧡 💛 💚 💙 💜 This is my first project! ✔ Actual

1 Jan 25, 2022