a spacial-temporal pattern detection system for home automation

Related tags

Deep Learningargos
Overview

Argos

docker pulls

a spacial-temporal pattern detection system for home automation. Based on OpenCV and Tensorflow, can run on raspberry pi and notify HomeAssistant via MQTT or webhooks.

Demo

Have a spare raspberry pi or jetson nano (or old laptop/mac mini) lying around? Have wifi connected security cams in your house (or a raspi camera)? Want to get notified when someone exits or enters your main door? When someone waters your plants (or forgets to)? When your dog hasn't been fed food in a while, or hasn't eaten? When someone left the fridge door open and forgot? left the gas stove running and forgot? when birds are drinking from your dog's water bowl? Well, you're not alone, and you're at the right place :)

Architecture

argos

  • Take a video input (a raspberry pi camera if run on a rpi, an RTMP stream of a security cam, or a video file)
  • Run a simple motion detection algorithm on the stream, applying minimum box thresholds, negative masks and masks
  • Run object detection on either the cropped frame where motion was detected or even the whole frame if needed, using tensorflow object detection API. There is support for both tensorflow 1 and 2 as well as tensorflow lite, and custom models as well
  • Serves a flask webserver to allow you to see the motion detection and object detection in action, serve a mpeg stream which can be configured as a camera in HomeAssistant
  • Object detection is also highly configurable to threshold or mask out false positives
  • Object detection features an optional "detection buffer' which can be used to get the average detection in moving window of frames before reporting the maximum cumulative average detection
  • Supports sending notifications to HomeAssistant via MQTT or webhooks. Webhook notification send the frame on which the detection was triggered, to allow you to create rich media notifications from it via the HA android or iOS apps.
  • Pattern detection: both the motion-detector and object-detector send events to a queue which is monitored and analyzed by a pattern detector. You can configure your own "movement patterns" - e.g. a person is exiting a door or entering a door, or your dog is going to the kitchen. It keeps a configurable history of states (motion detected in a mask, outside a mask, object detected (e.g. person), etc.) and your movement patterns are pre-configured sequence of states which identify that movement. door_detect.py provides a movement pattern detector to detect if someone is entering or exiting a door
  • All of the above functionality is provided by running stream.py. There's also serve.py which serves as an object detection service which can be called remotely from a low-grade CPU device like a raspberry pi zero w which cannot run tensorflow lite on its own. The motion detector can still be run on the pi zero, and only object detection can be done remotely by calling this service, making a distributed setup.
  • Architected to be highly concurrent and asynchronous (uses threads and queue's between all the components - flask server, motion detector, object detector, pattern detector, notifier, mqtt, etc)
  • Has tools to help you generate masks, test and tune the detectors, etc.
  • Every aspect of every detector can be tuned in the config files (which are purposefully kept as python classes and not yaml), every aspect is logged with colored output on the console for you to debug what is going on.

Installation

On a pi, as a systemd service
cd ~
git clone https://github.com/angadsingh/argos
sudo apt-get install python3-pip
sudo apt-get install python3-venv
pip3 install --upgrade pip
python3 -m venv argos-venv/
source argos-venv/bin/activate
pip install https://github.com/bitsy-ai/tensorflow-arm-bin/releases/download/v2.4.0/tensorflow-2.4.0-cp37-none-linux_armv7l.whl
pip install wheel
pip install -r argos/requirements.txt

#only required for tf2
git clone https://github.com/tensorflow/models.git
cd models/research/object_detection/packages/tf2
python -m pip install . --no-deps

make a systemd service to run it automatically

cd ~/argos
sudo cp resources/systemd/argos_serve.service /etc/systemd/system/
sudo cp resources/systemd/argos_stream.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable argos_serve.service
sudo systemctl enable argos_stream.service
sudo systemctl start argos_serve
sudo systemctl start argos_stream

see the logs

journalctl --unit argos_stream.service -f
As a docker container

You can use the following instructions to install argos as a docker container (e.g. if you already use docker on your rpi for hassio-supervised, or you intend to install it on your synology NAS which has docker, or you just like docker)

Install docker (optional)

curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

Run argos as a docker container

Note: replace the docker tag name below for your cpu architecture

image example device notes
angadsingh/argos:armv7 raspberry pi 2/3/4+
angadsingh/argos:x86_64 PC, Mac
angadsingh/argos:x86_64_gpu PC, Mac tensorflow with gpu support. run with docker flag --runtime=nvidia

stream.py:

docker run --rm -p8081:8081 -v configs:/configs \
						-v /home/pi/detections:/output_detections \
						-v /home/pi/argos-ssh:/root/.ssh angadsingh/argos:armv7 \
						/usr/src/argos/stream.py --ip 0.0.0.0 --port 8081 \
						--config configs.your_config

serve.py:

docker run --rm -p8080:8080 -v configs:/configs \
						-v /home/pi/upload:/upload angadsingh/argos:armv7 \
						/usr/src/argos/serve.py --ip 0.0.0.0 --port 8080 \
						--config configs.your_config  --uploadfolder "/upload"

make a systemd service to run it automatically. these services automatically download the latest docker image and run them for you: (note: you'll have to change the docker tag inside the service file for your cpu architecture)

sudo wget https://raw.githubusercontent.com/angadsingh/argos/main/resources/systemd/argos_serve_docker.service -P /etc/systemd/system/
sudo wget https://raw.githubusercontent.com/angadsingh/argos/main/resources/systemd/argos_stream_docker.service -P /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable argos_serve_docker.service
sudo systemctl enable argos_stream_docker.service
sudo systemctl start argos_serve_docker
sudo systemctl start argos_stream_docker

see the logs

journalctl --unit argos_serve_docker.service -f
journalctl --unit argos_stream_docker.service -f

Usage

stream.py - runs the motion detector, object detector (with detection buffer) and pattern detector

stream.py --ip 0.0.0.0 --port 8081 --config configs.config_tflite_ssd_example
Method Endpoint Description
Browse / will show a web page with the real time processing of the input video stream, and a separate video stream showing the object detector output
GET /status status shows the current load on the system
GET /config shows the config
GET /config?= will let you edit any config parameter without restarting the service
GET /image returns the latest frame as a JPEG image (useful in HA generic camera platform)
GET /video_feed streams an MJPEG video stream of the motion detector (useful in HA generic camera platform)
GET /od_video_feed streams an MJPEG video stream of the object detector

serve.py

serve.py --ip 0.0.0.0 --port 8080 --config configs.config_tflite_ssd_example --uploadfolder upload
Method Endpoint Description
POST /detect params:

file: the jpeg file to run the object detector on
threshold: object detector threshold (override config.tf_accuracy_threshold)
nmask: base64 encoded negative mask to apply. format: (xmin, ymin, xmax, ymax)

Home assistant automations

ha_automations/notify_door_movement_at_entrance.yaml - triggered by pattern detector ha_automations/notify_person_is_at_entrance.yaml - triggered by object detector

both of these use HA webhooks. i used MQTT earlier but it was too delayed and unreliable for my taste. the project still supports MQTT though and you'll have to make mqtt sensors in HA for the topics you're sending the notifications to here.

Configuration

both stream.py and serve.py share some configuration for the object detection, but stream.py builds on top of that with a lot more configuration for the motion detector, object detection buffer, pattern detector, and stream input configuration, etc. The example config documents the meaning of all the parameters

Performance

This runs at the following FPS with every component enabled:

device component fps
raspberry pi 4B motion detector 18 fps
raspberry pi 4B object detector (tflite) 5 fps

I actually run multiple of these for different RTMP cameras, each at 1 fps (which is more than enough for all real time home automation use cases)

Note:

This is my own personal project. It is not really written in a readable way with friendly abstractions, as that wasn't the goal. The goal was to solve my home automation problem quickly so that I can get back to real work :) So feel free to pick and choose snippets of code as you like or the whole solution if it fits your use case. No compromises were made in performance or accuracy, only 'coding best practices'. I usually keep such projects private but thought this is now meaty enough to be usable to someone else in ways I cannot imagine, so don't judge this project on its maturity or reuse readiness level ;) . Feel free to fork this project and make this an extendable framework if you have the time.

If you have any questions feel free to raise a github issue and i'll respond as soon as possible

Special thanks to these resources on the web for helping me build this.

Owner
Angad Singh
Angad Singh
Official pytorch implement for “Transformer-Based Source-Free Domain Adaptation”

Official implementation for TransDA Official pytorch implement for “Transformer-Based Source-Free Domain Adaptation”. Overview: Result: Prerequisites:

stanley 54 Dec 22, 2022
GitHub repository for the ICLR Computational Geometry & Topology Challenge 2021

ICLR Computational Geometry & Topology Challenge 2022 Welcome to the ICLR 2022 Computational Geometry & Topology challenge 2022 --- by the ICLR 2022 W

42 Dec 13, 2022
Pacman-AI - AI project designed by UC Berkeley. Designed reflex and minimax agents for the game Pacman.

Pacman AI Jussi Doherty CAP 4601 - Introduction to Artificial Intelligence - Fall 2020 Python version 3.0+ Source of this project This repo contains a

Jussi Doherty 1 Jan 03, 2022
A "gym" style toolkit for building lightweight Neural Architecture Search systems

A "gym" style toolkit for building lightweight Neural Architecture Search systems

Jack Turner 12 Nov 05, 2022
Libraries, tools and tasks created and used at DeepMind Robotics.

dm_robotics: Libraries, tools, and tasks created and used for Robotics research at DeepMind. Package overview Package Summary Transformations Rigid bo

DeepMind 273 Jan 06, 2023
Implementation of BI-RADS-BERT & The Advantages of Section Tokenization.

BI-RADS BERT Implementation of BI-RADS-BERT & The Advantages of Section Tokenization. This implementation could be used on other radiology in house co

1 May 17, 2022
This is an unofficial implementation of the paper “Student-Teacher Feature Pyramid Matching for Unsupervised Anomaly Detection”.

This is an unofficial implementation of the paper “Student-Teacher Feature Pyramid Matching for Unsupervised Anomaly Detection”.

haifeng xia 32 Oct 26, 2022
Automatic Calibration for Non-repetitive Scanning Solid-State LiDAR and Camera Systems

ACSC Automatic extrinsic calibration for non-repetitive scanning solid-state LiDAR and camera systems. System Architecture 1. Dependency Tested with U

KINO 192 Dec 13, 2022
Official code repository for ICCV 2021 paper: Gravity-Aware Monocular 3D Human Object Reconstruction

GraviCap Official code repository for ICCV 2021 paper: Gravity-Aware Monocular 3D Human Object Reconstruction. Gravity-Aware Monocular 3D Human-Object

Rishabh Dabral 15 Dec 09, 2022
ConvMixer unofficial implementation

ConvMixer ConvMixer 非官方实现 pytorch 版本已经实现。 nets 是重构版本 ,test 是官方代码 感兴趣小伙伴可以对照看一下。 keras 已经实现 tf2.x 中 是tensorflow 2 版本 gelu 激活函数要求 tf=2.4 否则使用入下代码代替gelu

Jian Tengfei 8 Jul 11, 2022
Compact Bilinear Pooling for PyTorch

Compact Bilinear Pooling for PyTorch. This repository has a pure Python implementation of Compact Bilinear Pooling and Count Sketch for PyTorch. This

Grégoire Payen de La Garanderie 234 Dec 07, 2022
An implementation of a discriminant function over a normal distribution to help classify datasets.

CS4044D Machine Learning Assignment 1 By Dev Sony, B180297CS The question, report and source code can be found here. Github Repo Solution 1 Based on t

Dev Sony 6 Nov 09, 2021
Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network

Speech Separation Using an Asynchronous Fully Recurrent Convolutional Neural Network This repository is the official implementation of Speech Separati

Kai Li (李凯) 116 Nov 09, 2022
A large-image collection explorer and fast classification tool

IMAX: Interactive Multi-image Analysis eXplorer This is an interactive tool for visualize and classify multiple images at a time. It written in Python

Matias Carrasco Kind 23 Dec 16, 2022
Graph Regularized Residual Subspace Clustering Network for hyperspectral image clustering

Graph Regularized Residual Subspace Clustering Network for hyperspectral image clustering

Yaoming Cai 5 Jul 18, 2022
Cross-modal Deep Face Normals with Deactivable Skip Connections

Cross-modal Deep Face Normals with Deactivable Skip Connections Victoria Fernández Abrevaya*, Adnane Boukhayma*, Philip H. S. Torr, Edmond Boyer (*Equ

72 Nov 27, 2022
Code and Experiments for ACL-IJCNLP 2021 Paper Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering.

Code and Experiments for ACL-IJCNLP 2021 Paper Mind Your Outliers! Investigating the Negative Impact of Outliers on Active Learning for Visual Question Answering.

Sidd Karamcheti 50 Nov 16, 2022
RCDNet: A Model-driven Deep Neural Network for Single Image Rain Removal (CVPR2020)

RCDNet: A Model-driven Deep Neural Network for Single Image Rain Removal (CVPR2020) Hong Wang, Qi Xie, Qian Zhao, and Deyu Meng [PDF] [Supplementary M

Hong Wang 6 Sep 27, 2022
Dimension Reduced Turbulent Flow Data From Deep Vector Quantizers

Dimension Reduced Turbulent Flow Data From Deep Vector Quantizers This is an implementation of A Physics-Informed Vector Quantized Autoencoder for Dat

DreamSoul 3 Sep 12, 2022
Machine Learning From Scratch. Bare bones NumPy implementations of machine learning models and algorithms with a focus on accessibility. Aims to cover everything from linear regression to deep learning.

Machine Learning From Scratch About Python implementations of some of the fundamental Machine Learning models and algorithms from scratch. The purpose

Erik Linder-Norén 21.8k Jan 09, 2023