A live streaming chatroom involving multiple modalities, such as voice, gesture, and facial expression

Overview

GitHub issues GitHub forks GitHub stars GitHub license

HiLive

A live streaming chatroom involving multiple modalities, such as voice, gesture, and facial expression.

Introduction

We focus on demonstrating the design, as well as highlighting the advantages and our considerations on design features, using the knowledge and design principles learned in CS3483.

As mentioned in our previous design reports, the project is motivated by the limited chatroom in the live stream. This interface is designed for a better chatroom experience in a live streaming platform based on Twitch, which is a popular video game live streaming platform. It also could be further generalized and adopted by video platforms like YouTube. Viewers usually type some messages in the chatroom and send some emojis to communicate with others and vibrant the atmosphere in the streaming room. However, the monotonous form of interaction has long been criticized by users. Therefore, we propose HiLive, where people can not only send emojis that are generalized according to their recognized facial expression but also say something and send the transcribed message directly without typing it out themselves. With the gesture recognition function, users are no longer bound to the keyboard, therefore, they can have an immersive live streaming interaction experience.

Features

Front Page

The front page levereage the use if Twitch API, and the OAuth authentication system. Users can login and register to the website with Twitch account. And the front page also provides a list of avaliable Live streams for users to click into it.


Recommended Channels Component

The component is completely reusable, it only takes a data array from the official Twitch API and shuffles the results (Some sort of recommended channels "algorithm")


Facial Expression Recognition

The facial expression recognition to emoji is a interesting feature that can let user interact with streamers directly. We have trained a model with more than 2000 images for different emoji categories. The prdiction outcome is in real-time and can be seen in the demo video we provided. The following figure capture some of the facial expression recognition results.

angry image smile image surprise image tease image

Speech to Text Component

Speech to Text used the Google Cloud Speech Recognition API to transcribe the voice to text and sent it to the chatroom. The website will capture the users voice with MDN Web Audio API, which means it will streaming the user's voice and activate the speech-to-text once receving the real-time speech data in byte format.

Gesture Recognition Component

The gesture recognition using the repository from Real-time Hand Gesture Recognition with 3D CNNs. Here is the demo video of the outcome.

simulation results

Figure: A real-time simulation of the architecture with input video from EgoGesture dataset (on left side) and real-time (online) classification scores of each gesture (on right side) are shown, where each class is annotated with different color.

Built With:

  • Next.JS
  • Typescript
  • Axios
  • Styled-Components
  • TwitchAPI
  • Deployed to Vercel

Run this locally

To run this project locally, you'll need Node.js installed.

Install dependencies preferably with yarn but you can also use npm install

Create a .env file in the root of the folder based on .env.example.

Run your Next.JS App with yarn dev or npm run dev

Go to localhost:3000 and check out this amazing clone

Owner
Ryan Yen
Love & Passion can conquer the world~ QAQ
Ryan Yen
IP Pinger - This tool allows you to enter an IP and check if its currently connected to a host

IP Pinger - This tool allows you to enter an IP and check if its currently connected to a host

invasion 3 Feb 18, 2022
PySocks lets you send traffic through SOCKS proxy servers.

PySocks lets you send traffic through SOCKS proxy servers. It is a modern fork of SocksiPy with bug fixes and extra features. Acts as a drop-i

1.1k Dec 07, 2022
This is a Client-Server-System which can share the screen from the server to client and in the other direction.

Screenshare-Streaming-Python This is a Client-Server-System which can share the screen from the server to client and in the other direction. You have

VFX / Videoeffects Creator 1 Nov 19, 2021
A repository to spoof ARP table of any devices and successfully establish Man in the Middle(MITM) attack using Python3 in Linux

arp_spoofer A repository to spoof ARP table of any devices and successfully establish Man in the Middle(MITM) attack using Python3 in Linux Usage: git

Surya Das N 1 Oct 30, 2021
Simple threaded Python Rickroll server. Listens on port 23 by default.

Terminal Rickroll Simple threaded Python Rickroll server. Listens on port 23 by default. Rickroll video made using Video-To-Ascii and the standard ric

AG 10 Sep 13, 2022
A simple hosts picker for Microsoft Services

A simple Python scrip for you to select the fastest IP for Microsoft services.

Konnyaku 394 Dec 17, 2022
An advanced real time threat intelligence framework to identify threats and malicious web traffic on the basis of IP reputation and historical data.

ARTIF is a new advanced real time threat intelligence framework built that adds another abstraction layer on the top of MISP to identify threats and malicious web traffic on the basis of IP reputatio

CRED 225 Dec 31, 2022
A Python server and client app that tracks player session times and server status

MC Outpost A Python server and client application that tracks player session times and server status About MC Outpost provides a session graph and ser

Grant Scrits 0 Jul 23, 2021
A project that forwards data it receives in a URL POST Request to a Discord Webhook link

Mailman Mailman is a project that basically just forwards data it receives in a URL POST Request to a Discord Webhook link and act as a sort of messag

Prakhar Trivedi 2 Mar 14, 2022
Nexum is an open-source, remote administration tool written in Python 3

A full-featured remote administration tool written in Python 3. The goal of this project is to make the use of a remote administration tool as simple

z3phyrus 2 Nov 26, 2021
Asyncer, async and await, focused on developer experience

Asyncer, async and await, focused on developer experience. Documentation: https:

Sebastián Ramírez 895 Dec 28, 2022
Exfiltrate files using the HTTP protocol version ("HTTP/1.0" is a 0 and "HTTP/1.1" is a 1)

http-protocol-exfil Use the HTTP protocol version to send a file bit by bit ("HTTP/1.0" is a 0 and "HTTP/1.1" is a 1). It uses GET requests so the Blu

Ricardo Ruiz 23 Apr 30, 2022
CORS Bypass Proxy Cloud Function

CORS Bypass Proxy Cloud Function

Elayamani K 1 Oct 23, 2021
This program ingests a Cisco "sh ip arp" as a text file and produces the list of vendors seen in the file

IP-ARP-Vendor_lookup This program ingests a Cisco "sh ip arp" as a text file and produces the list of vendors seen in the file Why? Answers the questi

Stew Alexander 1 Dec 24, 2022
stellar-add-guest is a small tool to generate a new guest for Stellar Wireless (Enterprise mode) in OmniVista 2500 hosted on OmniSwitch with AOS Release 8

stellar-add-guest is a small tool to generate a new guest for Stellar Wireless (Enterprise mode) in OmniVista 2500 hosted on OmniSwitch with AOS Release 8.

BennyE 3 Jan 24, 2022
A python tool auto change proxy or ip after dealy time set by user

Auto proxy Ghost This tool auto change proxy or ip after dealy time set by user how to run 1. Install required file ./requirements.sh 2.Enter command

Harsh Tagra 0 Feb 23, 2022
High capacity, high availability, well connected, fast lightning node.

LND ⚡ Routing High capacity, high availability, well connected, fast lightning node. We aim to become a top liquidity provider for the lightning netwo

18 Dec 16, 2022
Dnspython is a DNS toolkit for Python.

dnspython is a DNS toolkit for Python. It supports almost all record types. It can be used for queries, zone transfers, and dynamic updates. It supports TSIG authenticated messages and EDNS0.

Bob Halley 2.1k Jan 06, 2023
Ping Verification Python Script

Python Script Port Scanner Script WHAT IS IT? Port scanner script using Python. HOW IT WORKS Once the script has been executed, it will request the ta

AC 0 Dec 12, 2021
SMS Based Headless Browsing

Browse the internet without a network connection - Submission for ConUHacks VI

Zafir Khalid 2 Feb 07, 2022