A django integration for huey task queue that supports multi queue management

Related tags

Djangodjango-huey
Overview

Version

django-huey

This package is an extension of huey contrib djhuey package that allows users to manage multiple queues.

Installation

Using pip package manager run:

# pip install Django  if not installed
# pip install huey    if not installed
pip install django-huey

Note: use a virtualenv to isolate your dependencies. Note 2: django and huey must be installed.

Then, in your settings.py file add django_huey to the INSTALLED_APPS:

INSTALLED_APPS = [
	...
    'django_huey',
]

Configuration

In settings.py you must add the HUEYS setting:

HUEYS = {
    'first': {#this name will be used in decorators below
        'huey_class': 'huey.RedisHuey',  
        'name': 'first_tasks',  
        'consumer': {
            'workers': 2,
            'worker_type': 'thread',
        },
    },
    'emails': {#this name will be used in decorators below
        'huey_class': 'huey.RedisHuey',  
        'name': 'emails_tasks',  
        'consumer': {
            'workers': 5,
            'worker_type': 'thread',
        },
    }
}

Note: This setting is incompatible with HUEY setting.

Usage

Now you will be able to run multiple queues using:

python manage.py run_djangohuey --queue first
python manage.py run_djangohuey --queue emails

Each queue must be run in a different terminal.

Configuring tasks

You can use usual huey decorators to register tasks, but they must be imported from django_huey as shown below:

from django_huey import db_task, task

@db_task(queue='first')
	# perform some db task

@task(queue='emails')
	# send some emails

All the args and kwargs defined in huey decorators should work in the same way, if not, let us know.

You might also like...
Bootstrap 3 integration with Django.

django-bootstrap3 Bootstrap 3 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 3. Want to use Bootstr

Bootstrap 4 integration with Django.

django-bootstrap 4 Bootstrap 4 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 4. Requirements Pytho

Plug and play continuous integration with django and jenkins

django-jenkins Plug and play continuous integration with Django and Jenkins Installation From PyPI: $ pip install django-jenkins Or by downloading th

Django admin CKEditor integration.

Django CKEditor NOTICE: django-ckeditor 5 has backward incompatible code moves against 4.5.1. File upload support has been moved to ckeditor_uploader.

TinyMCE integration for Django

django-tinymce django-tinymce is a Django application that contains a widget to render a form field as a TinyMCE editor. Quickstart Install django-tin

Bootstrap 3 integration with Django.

django-bootstrap3 Bootstrap 3 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 3. Want to use Bootstr

Django + Next.js integration

Django Next.js Django + Next.js integration From a comment on StackOverflow: Run 2 ports on the same server. One for django (public facing) and one fo

This is a repository for collecting global custom management extensions for the Django Framework.

Django Extensions Django Extensions is a collection of custom extensions for the Django Framework. Getting Started The easiest way to figure out what

File and Image Management Application for django
File and Image Management Application for django

Django Filer django Filer is a file management application for django that makes handling of files and images a breeze. Contributing This is a an open

Comments
  • HueyException <task> not found in TaskRegistry

    HueyException not found in TaskRegistry

    When using django-huey, the consumer is able to correctly find the registered tasks, however when running the Django server and trying to enqueue a task, I get an "HueyException not found in TaskRegistry" error. From doing wayy too much debugging, I finally tracked this down to the fact that the huey package initializes itself with RegistryA, and then django_huey initializes itself with a separate RegistryB. So when trying to enqueue a task, it attempts to find the task in RegistryA which is empty, because all of the tasks are in RegistryB.

    I use import django_huey as huey everywhere, and then use @huey.task(queue="test"), but I'm still getting this error. I'm not sure what I'm doing wrong 😞

    opened by wgordon17 7
  • django-huey-monitor not working with djangohuey

    django-huey-monitor not working with djangohuey

    I use djangohuey --queue [name of queue] to run consumer, this runs my task, but the django-huey-monitor does not track the tasks, or are not created in the huey_monitor database tables. Which configuration am I missing?

    opened by EvelynArinaitwe 2
  • Unable to use `django-huey` with `SqliteHuey`

    Unable to use `django-huey` with `SqliteHuey`

    Settings:

    DJANGO_HUEY = {
        'default': 'xxx',
        'queues': {
            'xxx': {
                'huey_class': 'huey.SqliteHuey',
                'name': 'xxx_tasks',
                'consumer': {
                    'workers': 4,
                    'worker_type': 'thread',
                },
            },
            'yyy': { 
                'huey_class': 'huey.SqliteHuey',
                'name': 'yyy_tasks',
                'consumer': {
                    'workers': 2,
                    'worker_type': 'thread',
                },
            }
        }
    }
    

    Exception raised:

    huey.exceptions.ConfigurationError: "redis" python module not found, cannot use Redis storage backend. Run "pip install redis" to install.
    

    redis should be optional.

    opened by fabiocaccamo 2
  • Cannot configure djangohuey with different workers.

    Cannot configure djangohuey with different workers.

    Is there a way to configure the djangohuey settings, so that a specific worker is assigned to a specific queue? At the moment, I implement different queues with different number of workers in each consumer, but anytime a worker is free, it is taken on by any of the queue tasks that are waiting. what I want is for example: worker1 listen to only queue_a and worker2&worker3 listen to queue_b, regardless of which worker is free. Is this possible? Here are my settings:

    DJANGO_HUEY = {
        'default': 'first', #this name must match with any of the queues defined below.
        'queues': {
            'queue_a': {
                'huey_class': 'huey.RedisHuey',
                'name': 'tasks_a',
                "immediate": False,
                'results': True,
                'blocking': True,
                'connection': {
                        'host': 'localhost',
                        'port': 6379,
                        'db': 0,
                        'connection_pool': None,
                        'read_timeout': 1,
                        'url': None,  
                    },
                'consumer': {
                    'workers': 1,
                    'worker_type': 'thread',
                },
            },
            'queue_b': {
                'huey_class': 'huey.RedisHuey',
                'name': 'tasks_b',
                "immediate": False,
                'blocking': True,
                
                'consumer': {
                    'workers': 2,
                    'worker_type': 'thread',
                },
            },
        }
    }
    
    opened by EvelynArinaitwe 1
Releases(v1.1.1)
  • v1.1.1(Feb 7, 2022)

  • v1.1.0(Jan 19, 2022)

  • v1.0.1(Jan 14, 2022)

    1.0.1 - 2022-01-14

    Added

    • Close db connections before task body. https://github.com/coleifer/huey/commit/e77acf307bfdade914ab7f91c65dbbc183af5d8f
    Source code(tar.gz)
    Source code(zip)
  • v1.0.0(May 19, 2021)

    1.0.0 - 2021-05-19

    Note: This release contains breaking changes, see them below with the migration instructions.

    Added

    • Allow definition of a default queue.

    Changed

    • HUEYS django setting renamed to DJANGO_HUEY.
    • Change command run_djangohuey to djangohuey.
    Source code(tar.gz)
    Source code(zip)
  • v0.2.0(Apr 18, 2021)

    0.2.0 - 2021-04-18

    Added

    Nothing added this release

    Changed

    Nothing changed this release

    Fixed

    • When a huey name was not provided, default django db name was used. Now it's defaulted to queue name.

    Removed

    • Removed incompatibility with HUEY setting used by huey project.
    Source code(tar.gz)
    Source code(zip)
  • v0.1.1(Apr 4, 2021)

Owner
GAIA Software
Cooperativa de Desarrollo de Software | Quilmes, Buenos Aires, Argentina
GAIA Software
Integarting Celery with Django to asynchronous tasks 📃

Integrating 🔗 Celery with Django via Redis server ,To-Do asynchronously 👀task without stopping the main-flow 📃 of Django-project . It increase your speed 🚀 and user experience 🤵 of website

Rushi Patel 4 Jul 15, 2022
pdm-django: Django command shortcuts for PDM

pdm-django: Django command shortcuts for PDM A plugin that gives you command shortcuts for developing with PDM. pdm run python manage.py runserver -

Neutron Sync 2 Aug 11, 2022
Automatically upgrade your Django projects.

django-upgrade Automatically upgrade your Django projects. Installation Use pip: python -m pip install django-upgrade Python 3.8 to 3.10 supported. Or

Adam Johnson 525 Dec 29, 2022
A simple porfolio with Django, Bootstrap and Sqlite3

Django Portofolio Example this is a basic portfolio in dark mode Installation git clone https://github.com/FaztWeb/django-portfolio-simple.git cd djan

Fazt Web 16 Sep 26, 2022
Indonesia's negative news detection using gaussian naive bayes with Django+Scikir Learn

Introduction Indonesia's negative news detection using gaussian naive bayes build with Django and Scikit Learn. There is also any features, are: Input

Harifzi Ham 1 Dec 30, 2021
Automatic class scheduler for Texas A&M written with Python+Django and React+Typescript

Rev Registration Description Rev Registration is an automatic class scheduler for Texas A&M, aimed at easing the process of course registration by gen

Aggie Coding Club 21 Nov 15, 2022
Bootstrap 4 integration with Django.

django-bootstrap 4 Bootstrap 4 integration for Django. Goal The goal of this project is to seamlessly blend Django and Bootstrap 4. Requirements Pytho

Zostera B.V. 980 Dec 29, 2022
A Django app that allows visitors to interact with your site as a guest user without requiring registration.

django-guest-user A Django app that allows visitors to interact with your site as a guest user without requiring registration. Largely inspired by dja

Julian Wachholz 21 Dec 17, 2022
A simple djagno music website.

Mrock A simple djagno music website. I used this template and I translated it to eng. Also some changes commited. My Live Domo : https://mrock.pythona

Hesam N 1 Nov 30, 2021
A pickled object field for Django

django-picklefield About django-picklefield provides an implementation of a pickled object field. Such fields can contain any picklable objects. The i

Gintautas Miliauskas 167 Oct 18, 2022
A Django web application that shortens long URLs. This is a demo project to show off my tech abilities.

Django URL Shortener This project is just a complete and production-ready URL shortener web application to show off my tech and coding abilities. Impo

Seyyed Ali Ayati 5 Jan 26, 2022
A middleware to log the requests and responses using loguru.

Django Loguru The extension was based on another one and added some extra flavours. One of the biggest problems with the apps is the logging and that

Tiago Silva 9 Oct 11, 2022
Sistema de tratamento e análise de grandes volumes de dados através de técnicas de Data Science

Sistema de tratamento e análise de grandes volumes de dados através de técnicas de data science Todos os scripts, gráficos e relatórios de todas as at

Arthur Quintanilha Neto 1 Sep 05, 2022
A prettier way to see Django requests while developing

A prettier way to see Django requests while developing

Adam Hill 35 Dec 02, 2022
webfest Django project @innovaccer

inno-doctor webfest Django project @innovaccer setup guide create new directory for project clone the repo with url into the directory make sure pytho

Rohit sahu 6 Oct 28, 2022
Use heroicons in your Django and Jinja templates.

heroicons Use heroicons in your Django and Jinja templates. Requirements Python 3.6 to 3.9 supported. Django 2.2 to 3.2 supported. Are your tests slow

Adam Johnson 52 Dec 14, 2022
Template de desarrollo Django

Template de desarrollo Django Python Django Docker Postgres Nginx CI/CD Descripción del proyecto : Proyecto template de directrices para la estandariz

Diego Esteban 1 Feb 25, 2022
A Django web application that allows you to be in the loop about everything happening in your neighborhood.

A Django web application that allows you to be in the loop about everything happening in your neighborhood. From contact information of different handyman to meeting announcements or even alerts.

Kennedy Ngugi Mwaura 3 Dec 11, 2022
Mobile Detect is a lightweight Python package for detecting mobile devices (including tablets).

Django Mobile Detector Mobile Detect is a lightweight Python package for detecting mobile devices (including tablets). It uses the User-Agent string c

Botir 6 Aug 31, 2022
Easily share data across your company via SQL queries. From Grove Collab.

SQL Explorer SQL Explorer aims to make the flow of data between people fast, simple, and confusion-free. It is a Django-based application that you can

Grove Collaborative 2.1k Dec 30, 2022