Scrape the Twitter Frontend API without authentication.

Overview

Twitter Scraper

GitHub GitHub contributors code size maintain status

🇰🇷 Read Korean Version

Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it's own API, which I reverse–engineered. No API rate limits. No restrictions. Extremely fast.

You can use this library to get the text of any user's Tweets trivially.

Prerequisites

Before you begin, ensure you have met the following requirements:

  • Internet Connection
  • Python 3.6+

Installing twitter-scraper

If you want to use latest version, install from source. To install twitter-scraper from source, follow these steps:

Linux and macOS:

git clone https://github.com/bisguzar/twitter-scraper.git
cd twitter-scraper
sudo python3 setup.py install 

Also, you can install with PyPI.

pip3 install twitter_scraper

Using twitter_scraper

Just import twitter_scraper and call functions!

→ function get_tweets(query: str [, pages: int]) -> dictionary

You can get tweets of profile or parse tweets from hashtag, get_tweets takes username or hashtag on first parameter as string and how much pages you want to scan on second parameter as integer.

Keep in mind:

  • First parameter need to start with #, number sign, if you want to get tweets from hashtag.
  • pages parameter is optional.
Python 3.7.3 (default, Mar 26 2019, 21:43:19) 
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twitter_scraper import get_tweets
>>> 
>>> for tweet in get_tweets('twitter', pages=1):
...     print(tweet['text'])
... 
spooky vibe check

It returns a dictionary for each tweet. Keys of the dictionary;

Key Type Description
tweetId string Tweet's identifier, visit twitter.com/USERNAME/ID to view tweet.
userId string Tweet's userId
username string Tweet's username
tweetUrl string Tweet's URL
isRetweet boolean True if it is a retweet, False otherwise
isPinned boolean True if it is a pinned tweet, False otherwise
time datetime Published date of tweet
text string Content of tweet
replies integer Replies count of tweet
retweets integer Retweet count of tweet
likes integer Like count of tweet
entries dictionary Has hashtags, videos, photos, urls keys. Each one's value is list

→ function get_trends() -> list

You can get the Trends of your area simply by calling get_trends(). It will return a list of strings.

Python 3.7.3 (default, Mar 26 2019, 21:43:19) 
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twitter_scraper import get_trends
>>> get_trends()
['#WHUTOT', '#ARSSOU', 'West Ham', '#AtalantaJuve', '#バビロニア', '#おっさんずラブinthasky', 'Southampton', 'Valverde', '#MMKGabAndMax', '#23NParoNacional']

→ class Profile(username: str) -> class instance

You can get personal information of a profile, like birthday and biography if exists and public. This class takes username parameter. And returns itself. Access informations with class variables.

Python 3.7.3 (default, Mar 26 2019, 21:43:19) 
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twitter_scraper import Profile
>>> profile = Profile('bugraisguzar')
>>> profile.location
'Istanbul'
>>> profile.name
'Buğra İşgüzar'
>>> profile.username
'bugraisguzar'

.to_dict() -> dict

to_dict is a method of Profile class. Returns profile datas as Python dictionary.

Python 3.7.3 (default, Mar 26 2019, 21:43:19) 
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twitter_scraper import Profile
>>> profile = Profile("bugraisguzar")
>>> profile.to_dict()
{'name': 'Buğra İşgüzar', 'username': 'bugraisguzar', 'birthday': None, 'biography': 'geliş[email protected]', 'website': 'bisguzar.com', 'profile_photo': 'https://pbs.twimg.com/profile_images/1199305322474745861/nByxOcDZ_400x400.jpg', 'banner_photo': 'https://pbs.twimg.com/profile_banners/1019138658/1555346657/1500x500', 'likes_count': 2512, 'tweets_count': 756, 'followers_count': 483, 'following_count': 255, 'is_verified': False, 'is_private': False, user_id: "1019138658"}

Contributing to twitter-scraper

To contribute to twitter-scraper, follow these steps:

  1. Fork this repository.
  2. Create a branch with clear name: git checkout -b <branch_name>.
  3. Make your changes and commit them: git commit -m '<commit_message>'
  4. Push to the original branch: git push origin <project_name>/<location>
  5. Create the pull request.

Alternatively see the GitHub documentation on creating a pull request.

Contributors

Thanks to the following people who have contributed to this project:

  • @kennethreitz (author)
  • @bisguzar (maintainer)
  • @lionking6792
  • @ozanbayram
  • @xeliot

Contact

If you want to contact me you can reach me at @bugraisguzar.

License

This project uses the following license: MIT.

Owner
Buğra İşgüzar
⚕️Paramedic Hobbyist Pythonista
Buğra İşgüzar
An automated bot for twitter using Tweepy!

Tweeby An automated bot for twitter using Tweepy! About This bot will look for tweets that contain certain hashtags, if found. It'll send them a messa

Ori 1 Dec 06, 2021
Debugging with Stack Overflow: Web Search Behavior in Novice and Expert Programmers

Debugging with Stack Overflow: ICSE SEET, 2022 This is the GitHub repository associated with the 2022 ICSE SEET paper, Debugging with Stack Overflow:

Madeline Endres 1 Jan 31, 2022
Baota-docker - Deploying baota panel via docker

baota-docker Deploying baota panel via docker. 通过docker一键部署宝塔面板。 一、前言 好像很多人对这个感兴

Mr. Cat 15 Dec 12, 2022
A Discord Bot coded using Python. Open to collaboration

DisPy-Bot A Discord Bot coded using Python. Open to collaboration La syntax pour intégrer le bot (imaginons la fonction lol_reponse dans le fichier au

BiMathAx 2 Mar 03, 2022
Fully automated YouTube Channel. Using Reddit and YouTube API.

Fully Automated YouTube Shorts Channel This code will show you how to setup and fully autmated YouTube Channel. Content is gathered from Reddit using

Jacob Lower 96 Dec 25, 2022
A powerful discord bot for forming team.

Discord_SquadBot A powerful discord bot for forming team. Pre-requirement Python 3.7 and latest Discord.py module is required. Installation guideline

Jacky Yu 2 Jan 29, 2022
Exchange indicators & Basic functions for Binance API.

binance-ema Exchange indicators & Basic functions for Binance API. This python library has been written to calculate SMA, EMA, MACD etc. functions wit

Emre MENTEŞE 24 Jan 06, 2023
Ever wanted a dashboard for making your antispam? This is it.

Ever wanted a dashboard for making your antispam? This is it.

Skelmis 1 Oct 27, 2021
An instagram bot developed in Python with Selenium that helps you get more Instagram followers.

instabot An instagram bot developed in Python with Selenium that helps you get more Instagram followers. Install You’ll need to have: Python Selenium

65 Nov 22, 2022
With this simple app you can customize your presence in Discord.

Discord Rich Presence This a simple console app which can customize your Discord Presence easily, with only one config file! Setup Go to Discord Devel

Mai 1 Jan 06, 2022
A tool to build scripts to toggle between minimal & default services in Windows based on user defined lists.

A tool to build scripts to toggle between minimal & default services in Windows based on user defined lists.

AMIT 29 Jan 01, 2023
A high level library for building Discord bots.

Qord A high level library for building Discord bots. 🚧 This library is currently in development. Questions that you are having What is this? This is

Izhar Ahmad 16 May 14, 2022
The official wrapper for spyse.com API, written in Python, aimed to help developers build their integrations with Spyse.

Python wrapper for Spyse API The official wrapper for spyse.com API, written in Python, aimed to help developers build their integrations with Spyse.

Spyse 15 Nov 22, 2022
A Telegram Bot for adding Footer caption beside main caption of Telegram Channel Messages.

Footer-Bot A Telegram Bot for adding Footer caption beside main caption of Telegram Channel Messages. Best for Telegram Movie Channels. Made by @AbirH

Abir Hasan 35 Jan 02, 2023
TFT Bot that automatically surrenders and allows finishing TFT Passes easily.

Image Based TFT Bot TFT Bot that automatically surrenders and allows finishing TFT Passes easily. Please read full file! You can check new releases he

1 Feb 06, 2022
Bearer API client for Python

Bearer Python Bearer Python client Installation pip install bearer Usage Get your Bearer Secret Key and integration id from the Dashboard and use the

Bearer 9 Oct 31, 2022
Bot Realm of the Mad God Exalt (ROTMG). (Auto_nexus, Auto_HP, Auto_Loot)

Bot_ROTMG Bot Realm of the Mad God Exalt (ROTMG). (Auto_nexus, Auto_HP, Auto_Loot) *Este projeto visa apenas o aprendizado, quem faz mal uso é o único

Guilherme Silva Uchoa 1 Oct 30, 2021
Multi Account Generator Minecraft/NordVPN/Hulu/Origin And ...

Multi Account Generator Minecraft/NordVPN/Hulu/Origin And ...

76 Jan 01, 2023
Throttle and debounce add-on for Pyrogram

pyrothrottle Throttle and debounce add-on for Pyrogram Quickstart implementation on decorators from pyrogram import Client, filters from pyrogram.type

7 Oct 01, 2022
This is a small Messnger with the cmd as an interface

Messenger This is a small messenger with the cmd as an interface. It started as a project to learn more about Python 3. If you want to run a version o

1 Feb 24, 2022