Telegram bot/scraper to get the latest NUS vacancy reports.

Overview

Initial setup:

  1. Clone this repo
  2. Create a db.ini file in the root of the project directory
  3. Inside the file, include the following sections with key-value pairs:
[postgresql]
host=
   
    
database=
    
     
user=
     
      
password=
      
       

[telebot]
token=
       
         parse_mode=None 
       
      
     
    
   

example:

[postgresql]
host=localhost
database=master
user=postgres
password=123456

[telebot]
token=abcdgaksf:23641273uehwbdeh
parse_mode=None

The database port is 5432 by default.

To include other database hosts, just add the same 5 lines with a different section header like so:

[postgresql]
host=
   
    
database=
    
     
user=
     
      
password=
      
       

[hostnumber2]
host=
       
         database=
        
          user=
         
           password=
           
          
         
        
       
      
     
    
   

Vacancy Reports Folder

This folder contains all the vacancy reports from previous rounds. The naming convention is: {year} Sem {semester} Round {round}.pdf Example: 2020 Sem 2 Round 1.pdf

For the year variable, if the academic year is AY19/20, then year will be 2019. Always take the lower year in an academic year.

Inspiration

During modreg, we wanted some way of quickly looking up old vacancy reports to gauge the trend in the number of vacancies left for a particular mod.

There was hardly any archive of past year vacancy reports (except for 1 reddit thread) and hence the idea for a vacancy report scraper/database and a python bot to query the database

What it does

The frontend is a bot that queries the postgresql database. On the back, pdfs of old vacancy reports are fed through a scraper to generate the relevant tables and stored into the database.

How we built it

  1. scrape pdfs using tabula
  2. perform data cleaning on the scraped data
  3. insert clean data into postgresql database
  4. write some functions to query the database
  5. have a python bot invoke these functions

Challenges we ran into

dealing with panda dataframes

Accomplishments that we're proud of

  • hosting a postgresql db
  • good workflow implemented into the scraper so more vacancy reports can be added into the database as they come.

What we learned

python-telegram-bot, postgresql, data cleaning

What's next for modrekt vacancy reports bot (mvrb)

  • we are missing vacancy reports for sem 1!!!
  • expand the different ways data can be queried
  • move away from text-based to image-based data visualisation for better viewing experience
  • perform normalization on the database
Owner
Chee Hong
NUS Year 2 CS student. Matriculated in 2020.
Chee Hong
PRAW, an acronym for "Python Reddit API Wrapper", is a python package that allows for simple access to Reddit's API.

PRAW: The Python Reddit API Wrapper PRAW, an acronym for "Python Reddit API Wrapper", is a Python package that allows for simple access to Reddit's AP

Python Reddit API Wrapper Development 3k Dec 29, 2022
Support for Competitive Coding badges to add in Github readme or portfolio websites.

Support for Competitive Coding badges to add in Github readme or portfolio websites.

Akshat Aggarwal 2 Feb 14, 2022
Python script for download course from platzi.com

Platzi Downloader Tool Esta es una pequeña herramienta que hace mucho y que te ahorra una gran cantidad de trabajo a la hora de descargar cursos de Pl

Devil64-Dev 21 Sep 22, 2022
Very Sempil Bot Auto Filter bot

SAMANTHA_BOT Very Sempil Bot Auto Filter bot ##[ # 𝐂𝐋𝐈𝐂𝐊 𝐁𝐄𝐋𝐎𝐖 𝐈𝐌𝐀𝐆𝐄 𝐓𝐎 𝐃𝐄𝐏𝐋𝐎𝐘 👇 👇 👇 Auto Filter Manuel Filter IMDB Admin Co

DARK WEBLOAD 3 Jun 27, 2022
Cleiton Leonel 4 Apr 22, 2022
hydrotoolbox is a Python script for hydrologic calculations and analysis or by function calls within Python.

hydrotoolbox is a Python script for hydrologic calculations and analysis or by function calls within Python.

Tim Cera 4 Aug 20, 2022
Simple script to ban bots at Twitch chats using a text file as a source.

AUTOBAN 🇺🇸 English version Simple script to ban bots at Twitch chats using a text file as a source. How to use Windows Go to releases for further in

And Paiva 5 Feb 06, 2022
An Telegram Bot By @AsmSafone To Stream Videos in Telegram Voice Chat. This is Also The Source Code of The Bot Which is Being Used In @SafoTheBot Group! ❤️

Telegram Video Player Bot (Beta) An Telegram Bot By @AsmSafone To Stream Videos in Telegram Voice Chat. Special Features Supports Live Streaming From

SAF ONE 206 Jan 03, 2023
Python Business Transactions Library - ContractsPY

Python Business Transactions Library - ContractsPY Declare and define business transactions in Python. Use the contracts library to validate business

Arzu Huseynov 7 Jun 21, 2022
A discord bot written in discord.py to manage custom roles assigned to boosters of your server.

BBotty A discord bot written in discord.py to manage custom roles assigned to boosters of your server. v0.0.1-alpha released! This version is incomple

Oui002 1 Nov 27, 2021
A GUI Weather Application written with Python

weather-box - A GUI Weather Application written with Python Made with ❤️ by Suresh Mishra

Suresh Mishra 2 Dec 18, 2021
🖥️ Python - P1 Monitor API Asynchronous Python Client

🖥️ Asynchronous Python client for the P1 Monitor

Klaas Schoute 9 Dec 12, 2022
A repository of publicly verifiable token Sale contracts

Token-Sale-Plutus-Contract A repository of publicly verifiable token sale and royalty contracts. This will be the storage solution since it is easily

Logical Mechanism 29 Aug 18, 2022
Aqui está disponível GRATUITAMENTE, um bot de discord feito em python, saiba que, terá que criar seu bot como aplicação, e utilizar seu próprio token, e lembrando, é um bot básico, não se utiliza Cogs nem slash commands nele!

BotDiscordPython Aqui está disponível GRATUITAMENTE, um bot de discord feito em python, saiba que, terá que criar seu bot como aplicação, e utilizar s

Matheus Muguet 4 Feb 05, 2022
🤖 Chegg answers requested and sent by the Discord BOT to the targeted user.

Chegg BOT Description "I believe that open-source resources are a must for everyone around. Especially in the field of education. As Chegg c

Vusal Ismayilov 33 Aug 20, 2021
Automate saving your Discover Weekly Playlist using Python.

SpotWeekly Automate saving your Discover Weekly Playlist using Python. Made with 3 and FastAPI. The saved playlist link is sent to my discord server

shourya 6 Jan 03, 2022
Requests based multi-threaded script for increasing followers on Spotify

Proxyless Spotify Follow Bot Requests based multi-threaded script for increasing followers on Spotify. Click here to report bugs. Usage Download ZIP h

397 Jan 03, 2023
Automate HoYoLAB Genshin Daily Check-In Using Github Actions

Genshin Daily Check-In 🤖 Automate HoYoLAB Daily Check-In Using Github Actions KOR, ENG Instructions Fork the repository Go to Settings - Secrets Cli

Leo Kim 41 Jun 24, 2022
This is a telegram bot hosted by a Raspberry Pi equipped with a temperature and humidity sensor. The bot is capable of sending plots and readings.

raspy-temperature-bot This is a telegram bot hosted by a Raspberry Pi equipped with a temperature and humidity sensor. The bot is capable of sending p

31 May 22, 2022
An simple python script for remove rockstar account for fivem, very useful for cheating or change account for unban from an server, remember change steam account.

This tool is used for be unbanned from servers, u need disconnect the discord, use other steam account and uninstall xbox for be unbanned 100%, it only work for unban in server, not global

MiguDEV 4 Oct 10, 2022