Newsemble is an API that provides easy access to the current news for programmatic analysis

Last update: Dec 16, 2022

Overview

📰 Newsemble 📰

An API for fetching the current news.

🔖 About 🔖

Newsemble is an API that provides easy access to the current news for programmatic analysis. It has been built using Python, BeautifulSoup and MongoDB.
The data is scraped from these news websites every hour, stored in a database on the cloud and whenever requested, the most recent articles are promptly served.
Developers can make use of this API to fetch current data with each article having the following fields:
Headlines, Content, Source, Link and Time.

💻 Technologies

Newsemble is created with:

Python 3
Flask
PyMongo
BeautifulSoup

📂 File Structure and Description

app.py - Flask code for the API
scraper.py - Collection of scrapers for the various news sites.
db.py - Connecting and Using MongoDB
utils.py - Utility Functions
scheduler.py - Scheduler
Procfile - For Deployment
requirements.txt - Python Requirments

🛠️ Pipeline

🚀 Getting-started

This project can be accessed by using following setup

Links

Links	Description
www.newsemble.ml/news	Link to fetch all the data from all sources
www.newsemble.ml/news/toi	Link to fetch data from Times of India
www.newsemble.ml/news/th	Link to fetch data from The Hindu
www.newsemble.ml/news/tie	Link to fetch data from The Indian Express
www.newsemble.ml/news/ndtv	Link to fetch data from NDTV news
www.newsemble.ml/news/it	Link to fetch data from India Today

Request format

$ import requests
$ url = "http://www.newsemble.ml/news/"
$ requests.get(url).json()

Response format

{   
    ‘link’      :  $source_link$,
    ‘content’   :  $content_text$,    
    ‘source’    :  $news_source$,
    ‘title’     :  $headline$, 
    ‘time       :  $date_time_of_article$  
 }

Sample output

⚙️ Currently Supported Sites

🙏 Thanks!

All contributions are welcome and appreciated. 👍
If you liked this project, or found it useful in any way, please drop a 🌟 !

✍️ Authors ✍️

✒️ Rishabh Gupta
✒️ Vishal Singhania
✒️ Roshan Kumar

You might also like...

music downloader written in python. (Uses jiosaavn API)

35 Jul 20, 2022

Mobile based API for Crunchyroll BETA (and Downloader).

Mobile based API for Crunchyroll BETA (and Downloader). Not restricted on servers and NO CLOUDFLARE

27 Dec 11, 2022

Pypixiv - A fully-typed, asynchronous api wrapper for pixiv

pypixiv this library is a fully-typed, asynchronous api wrapper for pixiv. featu

2 Nov 16, 2022

This project is helps to download contents from Streamtape by utilizing the API

It scrapes Streamtape api and download contents from the site.

5 Dec 28, 2022

A python script that discovers hidden YouTube API clients. Just a research project.

YouTube-Internal-Clients A script that discovers hidden internal clients of the YouTube (Innertube) API using bruteforce methods. The script tries cli

97 Jan 2, 2023

Simple Python script to download images and videos from public subreddits without using Reddit's API 😎

Subreddit Media Downloader Download images and videos from any public subreddit without using Reddit's API Made with ❤ by Nico 💬 About: This script a

106 Jan 7, 2023

This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular intervals.It sends out the most recent news at random!

Nepali-news-notifier This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular in

1 Feb 11, 2022

News-app - This is a news web app for reading news from different sources and topics

1 Feb 2, 2022

Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and other financial information. This repository provides an SDK for developing applications to access the NCDS.

Nasdaq Cloud Data Service (NCDS) Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and ot

8 Dec 1, 2022

The windML framework provides an easy-to-use access to wind data sources within the Python world, building upon numpy, scipy, sklearn, and matplotlib. Renewable Wind Energy, Forecasting, Prediction

windml Build status : The importance of wind in smart grids with a large number of renewable energy resources is increasing. With the growing infrastr

125 Dec 24, 2022

Repo Home WPDrawBot - (Repo, Home, WP) A powerful programmatic 2D drawing application for MacOS X which generates graphics from Python scripts. (graphics, dev, mac)

DrawBot DrawBot is a powerful, free application for macOS that invites you to write Python scripts to generate two-dimensional graphics. The built-in

342 Dec 27, 2022

A curated list of programmatic weak supervision papers and resources

118 Jan 2, 2023

Learning trajectory representations using self-supervision and programmatic supervision.

Trajectory Embedding for Behavior Analysis (TREBA) Implementation from the paper: Jennifer J. Sun, Ann Kennedy, Eric Zhan, David J. Anderson, Yisong Y

58 Jan 6, 2023

Programmatic interface to Synapse services for Python

A Python client for Sage Bionetworks' Synapse, a collaborative, open-source research platform that allows teams to share data, track analyses, and collaborate

54 Dec 23, 2022

Django API that scrapes and provides the last news of the city of Carlos Casares by semantic way (RDF format).

"Casares News" API Api that scrapes and provides the last news of the city of Carlos Casares by semantic way (RDF format). Usage Consume the articles

6 May 12, 2022

This python module can analyse cryptocurrency news for any number of coins given and return a sentiment. Can be easily integrated with a Trading bot to keep an eye on the news.

Python script that analyses news headline or body sentiment and returns the overall media sentiment of any given coin. It can take multiple coins an

185 Dec 22, 2022

NLP project that works with news (NER, context generation, news trend analytics)

СоАвтор СоАвтор – платформа и открытый набор инструментов для редакций и журналистов-фрилансеров, который призван сделать процесс создания контента ма

38 Jan 4, 2023

Your own movie streaming service. Easy to install, easy to use. Download, manage and watch your favorite movies conveniently from your browser or phone. Install it on your server, access it anywhere and enjoy.

Vigilio Your own movie streaming service. Easy to install, easy to use. Download, manage and watch your favorite movies conveniently from your browser

141 Jan 6, 2023

💰 An Alfred Workflow that provides current price of cryptocurrency

Coin Ticker for Alfred Workflow An Alfred Workflow that provides current price and status about cryptocurrency from cryptocompare.com. Supports Alfred

14 Nov 17, 2022

Comments

Bump lxml from 4.5.0 to 4.6.5
Bumps lxml from 4.5.0 to 4.6.5.

Changelog

Sourced from lxml's changelog.

4.6.5 (2021-12-12)

Bugs fixed

A vulnerability (GHSL-2021-1038) in the HTML cleaner allowed sneaking script content through SVG images (CVE-2021-43818).

A vulnerability (GHSL-2021-1037) in the HTML cleaner allowed sneaking script content through CSS imports and other crafted constructs (CVE-2021-43818).

4.6.4 (2021-11-01)

Features added

GH#317: A new property system_url was added to DTD entities. Patch by Thirdegree.

GH#314: The STATIC_* variables in setup.py can now be passed via env vars. Patch by Isaac Jurado.

4.6.3 (2021-03-21)

Bugs fixed

A vulnerability (CVE-2021-28957) was discovered in the HTML Cleaner by Kevin Chung, which allowed JavaScript to pass through. The cleaner now removes the HTML5 formaction attribute.

4.6.2 (2020-11-26)

Bugs fixed

A vulnerability (CVE-2020-27783) was discovered in the HTML Cleaner by Yaniv Nizry, which allowed JavaScript to pass through. The cleaner now removes more sneaky "style" content.

4.6.1 (2020-10-18)

... (truncated)

Commits

a9611ba Fix a test in Py2.

a3eacbc Prepare release of 4.6.5.

b7ea687 Update changelog.

69a7473 Cleaner: cover some more cases where scripts could sneak through in specially...

54d2985 Fix condition in test decorator.

4b220b5 Use the non-depcrecated TextTestResult instead of _TextTestResult (GH-333)

d85c6de Exclude a test when using the macOS system libraries because it fails with li...

cd4bec9 Add macOS-M1 as wheel build platform.

fd0d471 Install automake and libtool in macOS build to be able to install the latest ...

f233023 Cleaner: Remove SVG image data URLs since they can embed script content.

Additional commits viewable in compare view

Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.

Dependabot commands and options

You can trigger Dependabot actions by commenting on this PR:

@dependabot rebase will rebase this PR

@dependabot recreate will recreate this PR, overwriting any edits that have been made to it

@dependabot merge will merge this PR after your CI passes on it

@dependabot squash and merge will squash and merge this PR after your CI passes on it

@dependabot cancel merge will cancel a previously requested merge and block automerging

@dependabot reopen will reopen this PR if it is closed

@dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually

@dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)

@dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)

@dependabot use these labels will set the current labels as the default for future PRs for this repo and language

@dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language

@dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language

@dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

You can disable automated security fix PRs for this repo from the Security Alerts page.

dependencies
opened by dependabot[bot] 0

Releases(v1.0)

v1.0(Jul 17, 2021)

Source code(tar.gz)
Source code(zip)
v0.1(Jul 16, 2021)

Source code(tar.gz)
Source code(zip)

Owner

Rishabh

CSE Undergrad at IIIT-D.

GitHub Repository http://www.newsemble.ml/news

Tool To download Amazon 4k SDR HDR 1080, CDM IS Not Included

WV-AMZN-4K-RIPPER Tool To download Amazon 4k SDR HDR 1080, CDM IS Not Included For CDM You can Mail :- 11 Dec 23, 2021

Download all posts and comments in a subreddit

subreddit downloader This subreddit downloader downloads all posts and comments in a subreddit For a tutorial to use this program please follow this m

6 Dec 16, 2022

Fully automated download and parsing for Texas A&M University's Registrar's grade distribution PDFs for years 2014+.

Fully automated download and parsing for Texas A&M University's Registrar's grade distribution PDFs for years 2014+. Adds the parsing results to a mySQL database.

1 Sep 28, 2022

Code to scrape , download and upload to youtube daily

Youtube_Automated_Channel Code to scrape , download and upload to youtube daily INSTRUCTIONS Download the Github Repository Download and install Pytho

2 Dec 19, 2021

squid-dl is a massively parallel yt-dlp-based YouTube downloader.

squid-dl squid-dl is a massively parallel yt-dlp-based YouTube downloader. Installation Run the setup.py, which will install squid-dl and its two depe

51 Jan 05, 2023

Python script to automate youtube-dl downloads

Automated Download Tool !! Project status I am writing a new version of this program, which will solve several errors. The new version only supports G

21 Sep 22, 2022

Itchio Downloader Tool with python

Itchio Downloader Tool Install pip install git+https://github.com/emersont1/itchio Download All Games in library from account python -m itchio.downloa

69 Dec 05, 2022

Open Source application for downloading and playing music.

Musifre Greetings For HackHeist(Wartex) Judges: Synopsis, Promotion Video & Product Functioning Video are present in Documentation Folder. A Star woul

9 Mar 22, 2022

YouTube Video Search Engine For Python

YouTube-Video-Search-Engine Introduction With the increasing demand for electronic devices, it is hard for people to choose the best products from mul

1 Dec 21, 2021

抖音批量下载助手

303 Jan 05, 2023

Download a large file from Google Drive (curl/wget fails because of the security notice).

gdown Download a large file from Google Drive. Description Download a large file from Google Drive. If you use curl/wget, it fails with a large file b

2.7k Jan 09, 2023

Convert BMS songs to osu! With options to convert keysounds and convert to 7key.

bmx2osu Convert BMS to osu! With options to: convert keysounds to one song file using BMX2WAV include 7k version change Overall Difficulty and HP Drai

7 Nov 28, 2022

a simple ehentai downloader with jpg 2 pdf

Simple_Ehentai_DownLoader a simple ehentai downloader with jpg 2 pdf 中文介绍 Environment python3.8 How to use before you start,there are some tips. the q

6 Dec 11, 2022

This is a tool to allow downloading any links from r/Roms Magethread

ILYFPR This is a tool to allow downloading any links from r/Roms Magethread Limitations: It downloads ALL roms of the system you choose. This will be

4 Nov 04, 2021

Download Web-10K data by querying Bing Image Search

gpv2-web10k This repository contains the script to download images from the Web-10K dataset. The script takes in a list of queries, queries Bing Image

8 Sep 06, 2022

Download Apple Music Cover Artwork in the best Quality by providing an Apple Music Link. It downloads the jpg, png and webp version since they often differ from another.

amogus.py - Version 0.0.5 amogus - Apple Music Hi-Res Artwork Fetcher this is my first real python tool so sorry if its bad amogus is a Python script

46 Jan 09, 2023

Newsemble is an API that provides easy access to the current news for programmatic analysis

Related tags

Overview

📰 Newsemble 📰

🔖 About 🔖

🗒️ Table of contents

💻 Technologies

📂 File Structure and Description

🛠️ Pipeline

🚀 Getting-started

⚙️ Currently Supported Sites

🙏 Thanks!

✍️ Authors ✍️

You might also like...

music downloader written in python. (Uses jiosaavn API)

Mobile based API for Crunchyroll BETA (and Downloader).

Pypixiv - A fully-typed, asynchronous api wrapper for pixiv

This project is helps to download contents from Streamtape by utilizing the API

A python script that discovers hidden YouTube API clients. Just a research project.

Simple Python script to download images and videos from public subreddits without using Reddit's API 😎

This script just scrapes the most recent Nepali news from Kathmandu Post and notifies the user about current events at regular intervals.It sends out the most recent news at random!

News-app - This is a news web app for reading news from different sources and topics

Nasdaq Cloud Data Service (NCDS) provides a modern and efficient method of delivery for realtime exchange data and other financial information. This repository provides an SDK for developing applications to access the NCDS.

The windML framework provides an easy-to-use access to wind data sources within the Python world, building upon numpy, scipy, sklearn, and matplotlib. Renewable Wind Energy, Forecasting, Prediction

Repo Home WPDrawBot - (Repo, Home, WP) A powerful programmatic 2D drawing application for MacOS X which generates graphics from Python scripts. (graphics, dev, mac)

A curated list of programmatic weak supervision papers and resources

Learning trajectory representations using self-supervision and programmatic supervision.

Programmatic interface to Synapse services for Python

Django API that scrapes and provides the last news of the city of Carlos Casares by semantic way (RDF format).

This python module can analyse cryptocurrency news for any number of coins given and return a sentiment. Can be easily integrated with a Trading bot to keep an eye on the news.

NLP project that works with news (NER, context generation, news trend analytics)

Your own movie streaming service. Easy to install, easy to use. Download, manage and watch your favorite movies conveniently from your browser or phone. Install it on your server, access it anywhere and enjoy.

💰 An Alfred Workflow that provides current price of cryptocurrency

Comments

Bump lxml from 4.5.0 to 4.6.5

4.6.5 (2021-12-12)

Bugs fixed

4.6.4 (2021-11-01)

Features added

4.6.3 (2021-03-21)

Bugs fixed

4.6.2 (2020-11-26)

Bugs fixed

4.6.1 (2020-10-18)

Releases(v1.0)

v1.0(Jul 17, 2021)

v0.1(Jul 16, 2021)

Owner

Rishabh

Tool To download Amazon 4k SDR HDR 1080, CDM IS Not Included

Download all posts and comments in a subreddit

Fully automated download and parsing for Texas A&M University's Registrar's grade distribution PDFs for years 2014+.

Code to scrape , download and upload to youtube daily

squid-dl is a massively parallel yt-dlp-based YouTube downloader.

Python script to automate youtube-dl downloads

Itchio Downloader Tool with python

Open Source application for downloading and playing music.

YouTube Video Search Engine For Python

抖音批量下载助手

Download a large file from Google Drive (curl/wget fails because of the security notice).

Convert BMS songs to osu! With options to convert keysounds and convert to 7key.

a simple ehentai downloader with jpg 2 pdf

This is a tool to allow downloading any links from r/Roms Magethread

Download Web-10K data by querying Bing Image Search

Download Apple Music Cover Artwork in the best Quality by providing an Apple Music Link. It downloads the jpg, png and webp version since they often differ from another.

The PornHub Downloader is a powerfull script used to download and manage both videos and pictures

YoutubeDownloader - Download any public Playlist from Youtube

A python program to download one or multiple videos from YouTube.

Download all your URI Online Judge source codes and upload to GitHub with simple steps.