A live streaming chatroom involving multiple modalities, such as voice, gesture, and facial expression

Overview

GitHub issues GitHub forks GitHub stars GitHub license

HiLive

A live streaming chatroom involving multiple modalities, such as voice, gesture, and facial expression.

Introduction

We focus on demonstrating the design, as well as highlighting the advantages and our considerations on design features, using the knowledge and design principles learned in CS3483.

As mentioned in our previous design reports, the project is motivated by the limited chatroom in the live stream. This interface is designed for a better chatroom experience in a live streaming platform based on Twitch, which is a popular video game live streaming platform. It also could be further generalized and adopted by video platforms like YouTube. Viewers usually type some messages in the chatroom and send some emojis to communicate with others and vibrant the atmosphere in the streaming room. However, the monotonous form of interaction has long been criticized by users. Therefore, we propose HiLive, where people can not only send emojis that are generalized according to their recognized facial expression but also say something and send the transcribed message directly without typing it out themselves. With the gesture recognition function, users are no longer bound to the keyboard, therefore, they can have an immersive live streaming interaction experience.

Features

Front Page

The front page levereage the use if Twitch API, and the OAuth authentication system. Users can login and register to the website with Twitch account. And the front page also provides a list of avaliable Live streams for users to click into it.


Recommended Channels Component

The component is completely reusable, it only takes a data array from the official Twitch API and shuffles the results (Some sort of recommended channels "algorithm")


Facial Expression Recognition

The facial expression recognition to emoji is a interesting feature that can let user interact with streamers directly. We have trained a model with more than 2000 images for different emoji categories. The prdiction outcome is in real-time and can be seen in the demo video we provided. The following figure capture some of the facial expression recognition results.

angry image smile image surprise image tease image

Speech to Text Component

Speech to Text used the Google Cloud Speech Recognition API to transcribe the voice to text and sent it to the chatroom. The website will capture the users voice with MDN Web Audio API, which means it will streaming the user's voice and activate the speech-to-text once receving the real-time speech data in byte format.

Gesture Recognition Component

The gesture recognition using the repository from Real-time Hand Gesture Recognition with 3D CNNs. Here is the demo video of the outcome.

simulation results

Figure: A real-time simulation of the architecture with input video from EgoGesture dataset (on left side) and real-time (online) classification scores of each gesture (on right side) are shown, where each class is annotated with different color.

Built With:

  • Next.JS
  • Typescript
  • Axios
  • Styled-Components
  • TwitchAPI
  • Deployed to Vercel

Run this locally

To run this project locally, you'll need Node.js installed.

Install dependencies preferably with yarn but you can also use npm install

Create a .env file in the root of the folder based on .env.example.

Run your Next.JS App with yarn dev or npm run dev

Go to localhost:3000 and check out this amazing clone

Owner
Ryan Yen
Love & Passion can conquer the world~ QAQ
Ryan Yen
Modern Denial-of-service ToolKit for python

๐Ÿ’ฃ Impulse Modern Denial-of-service ToolKit ๐Ÿ’ป Main window ๐Ÿ“ก Methods: Method Target Description SMS PHONE Sends a massive amount of SMS messages and

1 Nov 29, 2021
Tool to transfer credential files from Firefox to your local machine to decrypt offline.

Firefox-Dumper Firefox Dumper identifies the current user's Firefox profile directory and exfiltrates the credential files to the attacker's FTP serve

Joe Helle 22 Sep 10, 2022
NanoChat - nano chat server and client

NanoChat This is a work in progress! NanoChat is an application for connecting with your friends using Python that uses ONLY default Python libraries.

Miss Bliss 1 Nov 13, 2021
Ip-Tracker: a script written in python for tracking Someone using targets ip-Tracker address

๐Ÿ”ฐ ๐•€๐•ก-๐•‹๐•ฃ๐•’๐•”๐•œ๐•–๐•ฃ ๐Ÿ”ฐ Ip-Tracker is a script written in python for tracking Someone using targets ip-Tracker address It was made by Spider Anongre

Spider Anongreyhat 15 Dec 02, 2022
Simple Port Scanner With Socket Module In Python 3x

PortScanner Simple Port Scanner With Socket Module In Python 3x How To Install Requirements Of This Port Scanner sudo apt install python3;sudo apt ins

1 Nov 23, 2021
Test - Python project for Collection Server and API Server

QProjectPython Collection Server ์™€ API Server ๋ฅผ ์œ„ํ•œ Python ํ”„๋กœ์ ํŠธ ์ž…๋‹ˆ๋‹ค. [FastAPI์ฐธ๊ณ ]

1 Jan 03, 2022
Evaluation of TCP BBRv1 in wireless networks

The Network Simulator, Version 3 Table of Contents: An overview Building ns-3 Running ns-3 Getting access to the ns-3 documentation Working with the d

3 Nov 01, 2021
A repository to spoof ARP table of any devices and successfully establish Man in the Middle(MITM) attack using Python3 in Linux

arp_spoofer A repository to spoof ARP table of any devices and successfully establish Man in the Middle(MITM) attack using Python3 in Linux Usage: git

Surya Das N 1 Oct 30, 2021
Apple Store Stock Notifier monitors the availability of selected Apple devices in selected Apple stores, and sends you a notification when devices are available!

Apple Store Stock Notifier This software will immediately send you a notification via Telegram when one of your coveted Apple Devices is available in

Floris-Jan Willemsen 25 Dec 05, 2022
sshuttle: where transparent proxy meets VPN meets ssh

Transparent proxy server that works as a poor man's VPN. Forwards over ssh. Doesn't require admin. Works with Linux and MacOS. Supports DNS tunneling.

9.4k Jan 09, 2023
LGPL Pure Python OPC-UA Client and Server

LGPL Pure Python OPC-UA Client and Server

Free OPC-UA Library 1.2k Jan 04, 2023
InfraGenie is allows you to split out your infrastructure project into separate independent pieces, each with its own terraform state.

๐Ÿงž InfraGenie InfraGenie is allows you to split out your infrastructure project into separate independent pieces, each with its own terraform state. T

Digger 53 Nov 23, 2022
a safe proxy over tls

TlsProxys ๅŸบไบŽTLSๅ่ฎฎ็š„httpๆต้‡ไปฃ็† ๅฎ‰่ฃ… โ€ป ้œ€่ฆpython3.7+ linux: python3.9 -m pip install TlsProxys windows: pip install TlsProxys ๅŸบๆœฌ็”จๆณ• ๆœๅŠกๅ™จ็ซฏ: $ tpserver [command]

56 Nov 30, 2022
A simple software which can use to make a server in local network

home-nas it is simple software which can use to make a server in local network, it has a web site on it which can use by multipale system, i use nginx

R ansh joseph 1 Nov 10, 2021
An improved version of the original AutoDD

AutoDD = Automatically does the "due diligence" for you. If you want to know what stocks people are talking about on reddit, this little program might help you.

Steven Zhu 169 Oct 05, 2022
a decompilation of NAP36 the widevine removal software for amz and nf used by p2p groups until it stoped working due to it using expired cdm keys

NAP36 a decompilation of NAP36 the widevine removal software for amz and nf used by p2p groups until it stoped working due to it useing expired cdm ke

9 Aug 29, 2021
Build custom OSINT tools and APIs (Ping, Traceroute, Scans, Archives, DNS, Scrape, Whois, Metadata & built-in database for more info) with this python package

Build custom OSINT tools and APIs with this python package - It includes different OSINT modules (Ping, Traceroute, Scans, Archives, DNS, Scrape, Whoi

QeeqBox 52 Jan 06, 2023
A network address manipulation library for Python

netaddr A system-independent network address manipulation library for Python 2.7 and 3.5+. (Python 2.7 and 3.5 support is deprecated). Provides suppor

711 Jan 05, 2023
Find information about an IP address, such as its location, ISP, hostname, region, country, and city.

Find information about an IP address, such as its location, ISP, hostname, region, country, and city. An IP address can be traced, tracked, and located.

Sachit Yadav 2 Jul 09, 2022
This is a python based command line Network Scanner utility, which input as an argument for the exact IP address or the relative IP Address range you wish to do the Network Scan for and returns all the available IP addresses with their MAC addresses on your current Network.

This is a python based command line Network Scanner utility, which input as an argument for the exact IP address or the relative IP Address range you wish to do the Network Scan for and returns all t

Abhinandan Khurana 1 Feb 09, 2022