GazeScroller - Using Facial Movements to perform Hands-free Gesture on the system

Overview

GazeScroller

Using Facial Movements to perform Hands-free Gesture on the system

Abstract

As our world is getting digitized on an fast rate, every person is having a device that is making life better. Also, there is a considerate amount of the society that do not have interactions as others to these devices. One such example are the quadriplegic people (people suffering from paralysis) which constitute to 5.4 million people people in the world*. Our aim here is to make them interact with the digital world. In this project, facial movements of the person's face is fed to the system on real-time and a certain list of operations can be performed on the system using these facial actions.Additionally, we will extend this system to mini-games on the internet like the Dino Game. Finally, I have evaluated the system by five people and found that they have positively to the system. These results imply that we can generalise this system to the entire world.

Approach

The project captures live stream of the video via webcam of the system. It then maps the face to 68 landmark points via the library Dlib. The movements of the points corresponding to the eye and nose are monitored continously. The functionalities covered in the project include : • Detect blink of one eye to enable/disable scrolling. • Detect the scroll movement based on the movement of the point on the nose. Using Blink to toggle scroll and head direction to scroll

Background Study

Blinking is an involuntary action of a human being.Blinks can be spontaneous, reflex and voluntary, and eye blink rate depends on various factors including environmental factors, type of activity.

In order to segregate natural blink of the eye with the intentional blink of one eye of the user for functionality 1 as discussed above, I have studied the eye width ratios of by conducting experiments study over 5 users with each subject testing for 10 times. This data analysis is used to understand to difference in the eye width ratio between both the eyes to when a user blinks one of the eye. Secondly, the intentional blink of the eye is put on a threshold for 3 frames to detect blink. These procedures helped detect the intentional one eye blink from the natural blink of the eyes. The information from the Fig 1 gives us the details of the eye ratio and the delta (difference between the eye ratios). We take the mean and use them as a reference in our code as threshold.

Technical Tools :

• Dlib - a library used to detect face per frame via webcam • Python - language to write the code • landmarksPoints.dat file - this file is used to superimpose landmarks onto the face detected. • pynput - library to invoke keyboard and mouse keys.

System Setup :

By using the tools of mentioned above, we get the face of the user per frame superimposed by landmark points. Calculations for each frame include :

rightEyeWidthRatio = height of the right eye/ width of the right eye leftEyeWidthRatio = height of the left eye/ width of the left eye delta = abs(leftEyeWidthRatio - rightEyeWidthRatio) Whenever a user blinks one eye, following cases are checked • Check 1 : if delta > threshold of delta taken from fig.1 • Check 2 : if leftEyeWidthRatio < threshold value of blink and frame count is 3. • If Check 1 and Check 2 true , trigger Blink and enable scrolling. UX Aspects : Trigger notifications in the system when scrolling is toggled.

Discussion & Future Scope:

In the present work I have not made much effort into perfectly the model and in CV. I have worked towards the thresholds and correlating to the use case I mentioned in the abstract. If substantial work is detecting the exact eye wink using ML models, the system would be much better. The false blinks being recorded is because we lack a model here. In the future scope , we can use this feature to build interactive games to the quadriplegic people to improve their psychological status too.

Conclusion :

All the subjects who have tested responded positively to the system and felt good about it. Therefore, we can say that our system is performing good to scroll pages using the nose and to capture the blink of the eye as a toggle gesture.

Hence, such a model will be beneficial to quadriplegic people and help them to interact with the digital world.Since the false blinks are low, the system is good to be used. It can be further perfected with ML models to give better accuracy to be used by the quadriplegic people.

Spatial color quantization in Rust

rscolorq Rust port of Derrick Coetzee's scolorq, based on the 1998 paper "On spatial quantization of color images" by Jan Puzicha, Markus Held, Jens K

Collyn O'Kane 37 Dec 22, 2022
Prototype python implementation of the ome-ngff table spec

Prototype python implementation of the ome-ngff table spec

Kevin Yamauchi 8 Nov 20, 2022
Graph neural network message passing reframed as a Transformer with local attention

Adjacent Attention Network An implementation of a simple transformer that is equivalent to graph neural network where the message passing is done with

Phil Wang 49 Dec 28, 2022
Solutions and questions for AoC2021. Merry christmas!

Advent of Code 2021 Merry christmas! 🎄 🎅 To get solutions and approximate execution times for implementations, please execute the run.py script in t

Wilhelm Ågren 5 Dec 29, 2022
This is the code repository for the paper A hierarchical semantic segmentation framework for computer-vision-based bridge column damage detection

Bridge-damage-segmentation This is the code repository for the paper A hierarchical semantic segmentation framework for computer-vision-based bridge c

Jingxiao Liu 5 Dec 07, 2022
[CVPR 2022 Oral] Crafting Better Contrastive Views for Siamese Representation Learning

Crafting Better Contrastive Views for Siamese Representation Learning (CVPR 2022 Oral) 2022-03-29: The paper was selected as a CVPR 2022 Oral paper! 2

249 Dec 28, 2022
Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation"

EgoNet Official project website for the CVPR 2021 paper "Exploring intermediate representation for monocular vehicle pose estimation". This repo inclu

Shichao Li 138 Dec 09, 2022
Boundary IoU API (Beta version)

Boundary IoU API (Beta version) Bowen Cheng, Ross Girshick, Piotr Dollár, Alexander C. Berg, Alexander Kirillov [arXiv] [Project] [BibTeX] This API is

Bowen Cheng 177 Dec 29, 2022
This python-based package offers a way of creating a parametric OpenMC plasma source from plasma parameters.

openmc-plasma-source This python-based package offers a way of creating a parametric OpenMC plasma source from plasma parameters. The OpenMC sources a

Fusion Energy 10 Oct 18, 2022
[ACM MM 2021] Multiview Detection with Shadow Transformer (and View-Coherent Data Augmentation)

Multiview Detection with Shadow Transformer (and View-Coherent Data Augmentation) [arXiv] [paper] @inproceedings{hou2021multiview, title={Multiview

Yunzhong Hou 27 Dec 13, 2022
Code for paper "A Critical Assessment of State-of-the-Art in Entity Alignment" (https://arxiv.org/abs/2010.16314)

A Critical Assessment of State-of-the-Art in Entity Alignment This repository contains the source code for the paper A Critical Assessment of State-of

Max Berrendorf 16 Oct 14, 2022
🐦 Quickly annotate data from the comfort of your Jupyter notebook

🐦 pigeon - Quickly annotate data on Jupyter Pigeon is a simple widget that lets you quickly annotate a dataset of unlabeled examples from the comfort

Anastasis Germanidis 647 Jan 05, 2023
The Agriculture Domain of ERPNext comes with features to record crops and land

Agriculture The Agriculture Domain of ERPNext comes with features to record crops and land, track plant, soil, water, weather analytics, and even trac

Frappe 21 Jan 02, 2023
[NeurIPS2021] Code Release of Learning Transferable Perturbations

Learning Transferable Adversarial Perturbations This is an official release of the paper Learning Transferable Adversarial Perturbations. The code is

Krishna Kanth 17 Nov 11, 2022
Tensorflow implementation of "Learning Deconvolution Network for Semantic Segmentation"

Tensorflow implementation of Learning Deconvolution Network for Semantic Segmentation. Install Instructions Works with tensorflow 1.11.0 and uses the

Fabian Bormann 224 Apr 15, 2022
Official repository of "BasicVSR++: Improving Video Super-Resolution with Enhanced Propagation and Alignment"

BasicVSR_PlusPlus (CVPR 2022) [Paper] [Project Page] [Code] This is the official repository for BasicVSR++. Please feel free to raise issue related to

Kelvin C.K. Chan 227 Jan 01, 2023
This is the repository for our paper SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking

SimpleTrack This is the repository for our paper SimpleTrack: Understanding and Rethinking 3D Multi-object Tracking. We are still working on writing t

TuSimple 189 Dec 26, 2022
Library extending Jupyter notebooks to integrate with Apache TinkerPop and RDF SPARQL.

Graph Notebook: easily query and visualize graphs The graph notebook provides an easy way to interact with graph databases using Jupyter notebooks. Us

Amazon Web Services 501 Dec 28, 2022
NEATEST: Evolving Neural Networks Through Augmenting Topologies with Evolution Strategy Training

NEATEST: Evolving Neural Networks Through Augmenting Topologies with Evolution Strategy Training

Göktuğ Karakaşlı 16 Dec 05, 2022
Automatically Build Multiple ML Models with a Single Line of Code. Created by Ram Seshadri. Collaborators Welcome. Permission Granted upon Request.

Auto-ViML Automatically Build Variant Interpretable ML models fast! Auto_ViML is pronounced "auto vimal" (autovimal logo created by Sanket Ghanmare) N

AutoViz and Auto_ViML 397 Dec 30, 2022