https://github.com/fromkeys
Dude is a very simple framework for writing web scrapers using Python decorators. The design, inspired by Flask, was to easily build a web scraper in just a few lines of code. Dude has an easy-to-lea
cloudscraper A simple Python module to bypass Cloudflare's anti-bot page (also known as "I'm Under Attack Mode", or IUAM), implemented with Requests.
BiliComment A spider for BiliBili comment. Spider Usage Put config.json into config directory, and then python . ./config/config.json. A example confi
Meme Videos Scrapes memes from reddit using praw and request and then converts t
Provider API Scraper Example Example of scraping a paginated API endpoint and dumping the data into a DB. Pre-requisits Python = 3.9 Pipenv Setup # i
Gerapy Distributed Crawler Management Framework Based on Scrapy, Scrapyd, Scrapyd-Client, Scrapyd-API, Django and Vue.js. Documentation Documentation
Coinbase Blog Scraper A Python web scraper to scrape latest posts from official Coinbase's Blog. IDEA It scrapes up latest blog posts from https://blo
Divar.ir Ads Scrapper Introduction This project first asynchronously grab Divar.ir Ads and then save to .csv and .xlsx files named data.csv and data.x
Movies-Scraper You are probably tired of navigating through a movie website to get the right movie you'd want to watch during the weekend. There may e
Unja Fetch Known Urls What's Unja? Unja is a fast & light tool for fetching known URLs from Wayback Machine, Common Crawl, Virus Total & AlienVault's
Parallel web scraping The project is a training task for web scraping using python multithreading and a real-time-updated list of available proxy serv
scrapy-folder-tree This is a scrapy pipeline that provides an easy way to store files and images using various folder structures. Supported folder str
poolbooru_gelscraper a simple python script for scraping images off gelbooru pools. modules required:requests_html, and os by default saves files with
仓库简介 微博热搜榜, 参数wb 百度热搜榜, 参数bd 360热点榜, 参数360 csdn热榜接口, 下方查看 其他热搜待加入 如何使用? 注册vercel fork到你的仓库, 右上角 点击这里完成部署(一键部署) 请求参数 vercel配置好的地址+api?tit=+参数(仓库简介有参数信息
Web Crawlers for Data Labelling of Malicious Domain Detection & IP Reputation Evaluation This repository provides two web crawlers to label domain nam
Parsel Parsel is a BSD-licensed Python library to extract and remove data from HTML and XML using XPath and CSS selectors, optionally combined with re
Open Bilibili Crawer 哔哩哔哩是一个信息非常丰富的社交平台，我们基于此构造社交网络。在该网络中，节点包括用户（up主），以及视频、专栏等创作产物；关系包括：用户之间，包括关注关系（following/follower），回复关系（评论区），转发关系（对视频or动态转发）；用户对创
torrent-scrapper Python scrapper scrapping torrent website and download new movies Automatically. If you like it Put a ⭐ on this repo 😇 Run this git
demiurge PyQuery-based scraping micro-framework. Supports Python 2.x and 3.x. Documentation: http://demiurge.readthedocs.org Installing demiurge $ pip
Sportyfin Description Stream sports events straight from your Jellyfin server. Sportyfin allows users to scrape for live streamed events and watch str
326 Dec 15, 2022
2.6k Dec 31, 2022
3 Jul 05, 2021
12 Oct 28, 2022
1 Oct 20, 2021
2.9k Jan 03, 2023
3 Feb 18, 2022
4 Aug 29, 2022
1 Jan 31, 2022
10 Aug 07, 2022
1 Feb 10, 2022
7 Oct 23, 2022
1 Jan 02, 2022
3 Jul 08, 2022
1 Nov 05, 2021
859 Dec 29, 2022
3 Oct 21, 2021
1 Jan 08, 2022
109 Jul 20, 2022
38 Nov 05, 2022