A live streaming chatroom involving multiple modalities, such as voice, gesture, and facial expression

Overview

GitHub issues GitHub forks GitHub stars GitHub license

HiLive

A live streaming chatroom involving multiple modalities, such as voice, gesture, and facial expression.

Introduction

We focus on demonstrating the design, as well as highlighting the advantages and our considerations on design features, using the knowledge and design principles learned in CS3483.

As mentioned in our previous design reports, the project is motivated by the limited chatroom in the live stream. This interface is designed for a better chatroom experience in a live streaming platform based on Twitch, which is a popular video game live streaming platform. It also could be further generalized and adopted by video platforms like YouTube. Viewers usually type some messages in the chatroom and send some emojis to communicate with others and vibrant the atmosphere in the streaming room. However, the monotonous form of interaction has long been criticized by users. Therefore, we propose HiLive, where people can not only send emojis that are generalized according to their recognized facial expression but also say something and send the transcribed message directly without typing it out themselves. With the gesture recognition function, users are no longer bound to the keyboard, therefore, they can have an immersive live streaming interaction experience.

Features

Front Page

The front page levereage the use if Twitch API, and the OAuth authentication system. Users can login and register to the website with Twitch account. And the front page also provides a list of avaliable Live streams for users to click into it.


Recommended Channels Component

The component is completely reusable, it only takes a data array from the official Twitch API and shuffles the results (Some sort of recommended channels "algorithm")


Facial Expression Recognition

The facial expression recognition to emoji is a interesting feature that can let user interact with streamers directly. We have trained a model with more than 2000 images for different emoji categories. The prdiction outcome is in real-time and can be seen in the demo video we provided. The following figure capture some of the facial expression recognition results.

angry image smile image surprise image tease image

Speech to Text Component

Speech to Text used the Google Cloud Speech Recognition API to transcribe the voice to text and sent it to the chatroom. The website will capture the users voice with MDN Web Audio API, which means it will streaming the user's voice and activate the speech-to-text once receving the real-time speech data in byte format.

Gesture Recognition Component

The gesture recognition using the repository from Real-time Hand Gesture Recognition with 3D CNNs. Here is the demo video of the outcome.

simulation results

Figure: A real-time simulation of the architecture with input video from EgoGesture dataset (on left side) and real-time (online) classification scores of each gesture (on right side) are shown, where each class is annotated with different color.

Built With:

  • Next.JS
  • Typescript
  • Axios
  • Styled-Components
  • TwitchAPI
  • Deployed to Vercel

Run this locally

To run this project locally, you'll need Node.js installed.

Install dependencies preferably with yarn but you can also use npm install

Create a .env file in the root of the folder based on .env.example.

Run your Next.JS App with yarn dev or npm run dev

Go to localhost:3000 and check out this amazing clone

Owner
Ryan Yen
Love & Passion can conquer the world~ QAQ
Ryan Yen
The OUCH Project - OUCH Server/Client

This software simulates OUCH Server/Client communication through a script which initialises a central server and another script which simulates clients connecting to the server.

Jahin Z. 2 Dec 10, 2022
Build surface water network for MODFLOW's SFR Package

Surface water network Creates surface water network, which can be used to create MODFLOW's SFR. Python packages Python 3.6+ is required. Required geop

Mike Taves 20 Nov 22, 2022
Qtas(Quite a Storage)is an experimental distributed storage system developed by Q-team in BJFU Advanced Computer Network sources.

Qtas(Quite a Storage)is a experimental distributed storage system developed by Q-team in BJFU Advanced Computer Network sources.

Jiaming Zhang 3 Jan 12, 2022
Medusa is a cross-platform agent compatible with both Python 3.8 and Python 2.7.

Medusa Medusa is a cross-platform agent compatible with both Python 3.8 and Python 2.7. Installation To install Medusa, you'll need Mythic installed o

Mythic Agents 123 Nov 09, 2022
Bittensor - an open, decentralized, peer-to-peer network that functions as a market system for the development of artificial intelligence

At Bittensor, we are creating an open, decentralized, peer-to-peer network that functions as a market system for the development of artificial intelligence.

Opentensor 169 Dec 30, 2022
telnet implementation over TCP socket with python

This a P2P implementation of telnet. This program transfers data on TCP sockets as plain text

10 May 19, 2022
Ping IP addresses and domains in parallel to find the accessible and inaccessible ones.

🚀 IPpy Parallel testing of IP addresses and domains in python. Reads IP addresses and domains from a CSV file and gives two lists of accessible and i

Shivam Mathur 54 May 21, 2022
Start a simple TCP Listener on a specified IP Address and Port Number and receive incoming connections.

About Start a simple TCP Listener on a specified IP Address and Port Number and receive incoming connections. Download Clone using git in terminal(git

AgentGeneric 5 Feb 24, 2022
Tool for ROS 2 IP Discovery + System Monitoring

Monitor the status of computers on a network using the DDS function of ROS2.

Ar-Ray 33 Apr 03, 2022
The module that allows the collection of data sampling, which is transmitted with WebSocket via WIFI or serial port for CSV file.

The module that allows the collection of data sampling, which is transmitted with WebSocket via WIFI or serial port for CSV file.

Nelson Wenner 2 Apr 01, 2022
Python script to stop qBittorrent from torrenting without VPN for users with static IP.

Python script to stop qBittorrent from torrenting without VPN for users with static IP.

voidoak_ 1 Oct 25, 2021
No-dependency, single file NNTP server library for developing modern, rfc3977-compliant (bridge) NNTP servers.

nntpserver.py No-dependency, single file NNTP server library for developing modern, rfc3977-compliant (bridge) NNTP servers for python =3.7. Develope

Manos Pitsidianakis 44 Nov 14, 2022
Connection package to a raspberry or any other machine using ssh, it simplifies the deployment scripts and monitoring.

Connection package to a raspberry or any other machine using ssh, it simplifies the deployment scripts and monitoring.

Dashstrom 7 Mar 29, 2022
Apple Store Stock Notifier monitors the availability of selected Apple devices in selected Apple stores, and sends you a notification when devices are available!

Apple Store Stock Notifier This software will immediately send you a notification via Telegram when one of your coveted Apple Devices is available in

Floris-Jan Willemsen 25 Dec 05, 2022
Fast and configurable script to get and check free HTTP, SOCKS4 and SOCKS5 proxy lists from different sources and save them to files

Fast and configurable script to get and check free HTTP, SOCKS4 and SOCKS5 proxy lists from different sources and save them to files. It can also get geolocation for each proxy and check if proxies a

Almaz 385 Dec 31, 2022
track IP Address

ipX Table of Contents ipX Welcome Features Uses Author 📝 License Welcome find the location of an IP address. Specifically, you can get the following

Ali Shahid 15 Sep 26, 2022
A TCP Chatroom built with python and TCP/IP sockets, consisting of a server and multiple clients which can connect with the server and chat with each other.

A TCP Chatroom built with python and TCP/IP sockets, consisting of a server and multiple clients which can connect with the server and chat with each other. It also provides an Admin role with featur

3 May 22, 2022
Domain To Api [ PYTHON ]

Domain To IP Usage You Open Terminal For Run The Program python ip.py Input & Output Input Your List e.g domain.txt Output ( For Save Output File )

It's Me Jafar 0 Dec 12, 2021
Heroku Cloudflare App Domain

Heroku Cloudflare App Domain Creating branded herokuapp.com-like domains using Cloudflare, based on the app name (eg my-app-prod.example.com). Feature

Torchbox 2 Oct 04, 2022
Flashes keyboard leds on incoming/outgoing network packets

LED Net Capture Flashes keyboard leds on incoming/outgoing network packets Usage Requires root priviledges to run usage: ledcapture.py [-h] --keyboard

Dan Habot 56 Oct 27, 2022