Telegram bot/scraper to get the latest NUS vacancy reports.

Overview

Initial setup:

  1. Clone this repo
  2. Create a db.ini file in the root of the project directory
  3. Inside the file, include the following sections with key-value pairs:
[postgresql]
host=
   
    
database=
    
     
user=
     
      
password=
      
       

[telebot]
token=
       
         parse_mode=None 
       
      
     
    
   

example:

[postgresql]
host=localhost
database=master
user=postgres
password=123456

[telebot]
token=abcdgaksf:23641273uehwbdeh
parse_mode=None

The database port is 5432 by default.

To include other database hosts, just add the same 5 lines with a different section header like so:

[postgresql]
host=
   
    
database=
    
     
user=
     
      
password=
      
       

[hostnumber2]
host=
       
         database=
        
          user=
         
           password=
           
          
         
        
       
      
     
    
   

Vacancy Reports Folder

This folder contains all the vacancy reports from previous rounds. The naming convention is: {year} Sem {semester} Round {round}.pdf Example: 2020 Sem 2 Round 1.pdf

For the year variable, if the academic year is AY19/20, then year will be 2019. Always take the lower year in an academic year.

Inspiration

During modreg, we wanted some way of quickly looking up old vacancy reports to gauge the trend in the number of vacancies left for a particular mod.

There was hardly any archive of past year vacancy reports (except for 1 reddit thread) and hence the idea for a vacancy report scraper/database and a python bot to query the database

What it does

The frontend is a bot that queries the postgresql database. On the back, pdfs of old vacancy reports are fed through a scraper to generate the relevant tables and stored into the database.

How we built it

  1. scrape pdfs using tabula
  2. perform data cleaning on the scraped data
  3. insert clean data into postgresql database
  4. write some functions to query the database
  5. have a python bot invoke these functions

Challenges we ran into

dealing with panda dataframes

Accomplishments that we're proud of

  • hosting a postgresql db
  • good workflow implemented into the scraper so more vacancy reports can be added into the database as they come.

What we learned

python-telegram-bot, postgresql, data cleaning

What's next for modrekt vacancy reports bot (mvrb)

  • we are missing vacancy reports for sem 1!!!
  • expand the different ways data can be queried
  • move away from text-based to image-based data visualisation for better viewing experience
  • perform normalization on the database
Owner
Chee Hong
NUS Year 2 CS student. Matriculated in 2020.
Chee Hong
🛒 Bot de lista de compras compartilhada para o Telegram

Lista de Compras Lista de compras de Cuducos e Flávia. Comandos do bot Comando Descrição /add item Adiciona item à lista de compras /remove item

Eduardo Cuducos 4 Jan 15, 2022
Visual Weather api. Returns beautiful pictures with the current weather.

VWapi Visual Weather api. Returns beautiful pictures with the current weather. Installation: sudo apt update -y && sudo apt upgrade -y sudo apt instal

Hotaru 33 Nov 13, 2022
Telegram bot to stream videos in telegram voicechat for both groups and channels

Telegram bot to stream videos in telegram voicechat for both groups and channels. Supports live streams, YouTube videos and telegram media. With record stream support, Schedule streams, and many more

ALBY 9 Feb 20, 2022
PRAW, an acronym for "Python Reddit API Wrapper", is a python package that allows for simple access to Reddit's API.

PRAW: The Python Reddit API Wrapper PRAW, an acronym for "Python Reddit API Wrapper", is a Python package that allows for simple access to Reddit's AP

Python Reddit API Wrapper Development 3k Dec 29, 2022
This is an implementation example of a bot that periodically sends predictions to the alphasea-agent.

alphasea-example-model alphasea-example-modelは、 alphasea-agent に対して毎ラウンド、予測を投稿するプログラムです。 Numeraiのexample modelに相当します。 準備 alphasea-example-modelの動作には、

AlphaSea 11 Jul 28, 2022
Seems Like Everyone Is Posting This, Thought I Should Too, Tokens Get Locked Upon Creation And Im Not Going To Fix For Several Reasons

Member-Booster Seems Like Everyone Is Posting This, Thought I Should Too, Tokens Get Locked Upon Creation And Im Not Going To Fix For Several Reasons

Mintyz 1 Dec 28, 2021
IOGen - An Open source discord token generator

_____ ____ _____ |_ _/ __ \ / ____| | || | | | |

0xVichy#1234 85 Nov 03, 2022
A small Python app to create Notion pages from Jira issues

Jira to Notion This little program will capture a Jira issue and create a corresponding Notion subpage. Mac users can fetch the current issue from the

Dr. Kerem Koseoglu 12 Oct 27, 2022
A simple healthcheck wrapper to monitor Kafka.

kafka-healthcheck A simple healthcheck wrapper to monitor Kafka. Kafka Healthcheck is a simple server that provides a singular API endpoint to determi

Rodrigo Nicolas Garcia 3 Oct 17, 2022
AirDrive lets you store unlimited files to cloud for free. Upload & download files from your personal drive at any time using its super-fast API.

AirDrive lets you store unlimited files to cloud for free. Upload & download files from your personal drive at any time using its super-fast API.

Sougata 4 Jul 12, 2022
Telegram Radio - A User-bot who continuously play random audio files (from the famous telegram music channel @mveargasm) in the intended voice chat.

MvEargasmDJ: This is my submission for the Telegram Radio Project of Baivaru. Which required a userbot to continiously play random audio files from th

eyaadh 24 Nov 12, 2022
A Telegram Music Bot written in Python using Pyrogram and Py-Tgcalls

A Telegram Music Bot written in Python using Pyrogram and Py-Tgcalls. This is Also The Source Code of The UserBot Which is Playing Music in @S1-BOTS Support Group ❤️

SAF ONE 224 Dec 28, 2022
Most Simple & Powefull web3 Trade Bot (WINDOWS LINUX) Suport BSC ETH

Most Simple & Powefull Trade Bot (WINDOWS LINUX) What Are Some Pros And Cons Of Owning A Sniper Bot? While having a sniper bot is typically an advanta

GUI BOT 4 Jan 25, 2022
ARKHAM X GOD MULTISPAM BOT

ARKHAM-X-GOD-MULTISPAM-BOT 𝗗𝗘𝗣𝗟𝗢𝗬 𝗨𝗣𝗧𝗢 30 𝗕𝗢𝗧𝗦 𝗜𝗡 𝗔 𝗦𝗜𝗡𝗚𝗟?

ArkhamXGod 2 Jan 08, 2022
Faster Twitch Alerts is a highly customizable, lightning-fast alternative to Twitch's slow mobile notification system

Faster Twitch Alerts What is "Faster Twitch Alerts"? Faster Twitch Alerts is a highly customizable, lightning-fast alternative to Twitch's slow mobile

6 Dec 22, 2022
An interactive and multi-function Telegram bot, made especially for Telegram groups.

PyKorone An interaction and fun bot for Telegram groups, having some useful and other useless commands. Created as an experiment and learning bot but

Amano Team 17 Nov 12, 2022
SC4.0 - BEST EXPERIENCE · HEX EDITOR · Discord Nuker · Plugin Adder · Cheat Engine

smilecreator4 This site is for people who want to hack or want to learn it! Furthermore, this program does not work without turning off Antivirus or W

1 Jan 04, 2022
Decode the Ontario proof of vaccination QR code

Decode the contents of the Ontario Proof of Vaccination (the "Smart Health Card QR Code") Output This is from my QR code, hopefully fully redacted alt

Wesley Ellis 4 Oct 22, 2021
Discord bot that generates boba drinks. Submission for sunhacks 2021

boba-bot Team Poggies' submission for Sunhacks 2021. Find our project page on Devpost, and a video demonstration can be found on YouTube. Commands $he

Joshua Tenorio 3 Nov 02, 2022
The Official Twilio SendGrid Led, Community Driven Python API Library

The default branch name for this repository has been changed to main as of 07/27/2020. This library allows you to quickly and easily use the SendGrid

Twilio SendGrid 1.4k Jan 07, 2023