Scrape the Twitter Frontend API without authentication.

Overview

Twitter Scraper

GitHub GitHub contributors code size maintain status

🇰🇷 Read Korean Version

Twitter's API is annoying to work with, and has lots of limitations — luckily their frontend (JavaScript) has it's own API, which I reverse–engineered. No API rate limits. No restrictions. Extremely fast.

You can use this library to get the text of any user's Tweets trivially.

Prerequisites

Before you begin, ensure you have met the following requirements:

  • Internet Connection
  • Python 3.6+

Installing twitter-scraper

If you want to use latest version, install from source. To install twitter-scraper from source, follow these steps:

Linux and macOS:

git clone https://github.com/bisguzar/twitter-scraper.git
cd twitter-scraper
sudo python3 setup.py install 

Also, you can install with PyPI.

pip3 install twitter_scraper

Using twitter_scraper

Just import twitter_scraper and call functions!

→ function get_tweets(query: str [, pages: int]) -> dictionary

You can get tweets of profile or parse tweets from hashtag, get_tweets takes username or hashtag on first parameter as string and how much pages you want to scan on second parameter as integer.

Keep in mind:

  • First parameter need to start with #, number sign, if you want to get tweets from hashtag.
  • pages parameter is optional.
Python 3.7.3 (default, Mar 26 2019, 21:43:19) 
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twitter_scraper import get_tweets
>>> 
>>> for tweet in get_tweets('twitter', pages=1):
...     print(tweet['text'])
... 
spooky vibe check

It returns a dictionary for each tweet. Keys of the dictionary;

Key Type Description
tweetId string Tweet's identifier, visit twitter.com/USERNAME/ID to view tweet.
userId string Tweet's userId
username string Tweet's username
tweetUrl string Tweet's URL
isRetweet boolean True if it is a retweet, False otherwise
isPinned boolean True if it is a pinned tweet, False otherwise
time datetime Published date of tweet
text string Content of tweet
replies integer Replies count of tweet
retweets integer Retweet count of tweet
likes integer Like count of tweet
entries dictionary Has hashtags, videos, photos, urls keys. Each one's value is list

→ function get_trends() -> list

You can get the Trends of your area simply by calling get_trends(). It will return a list of strings.

Python 3.7.3 (default, Mar 26 2019, 21:43:19) 
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twitter_scraper import get_trends
>>> get_trends()
['#WHUTOT', '#ARSSOU', 'West Ham', '#AtalantaJuve', '#バビロニア', '#おっさんずラブinthasky', 'Southampton', 'Valverde', '#MMKGabAndMax', '#23NParoNacional']

→ class Profile(username: str) -> class instance

You can get personal information of a profile, like birthday and biography if exists and public. This class takes username parameter. And returns itself. Access informations with class variables.

Python 3.7.3 (default, Mar 26 2019, 21:43:19) 
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twitter_scraper import Profile
>>> profile = Profile('bugraisguzar')
>>> profile.location
'Istanbul'
>>> profile.name
'Buğra İşgüzar'
>>> profile.username
'bugraisguzar'

.to_dict() -> dict

to_dict is a method of Profile class. Returns profile datas as Python dictionary.

Python 3.7.3 (default, Mar 26 2019, 21:43:19) 
[GCC 8.2.1 20181127] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from twitter_scraper import Profile
>>> profile = Profile("bugraisguzar")
>>> profile.to_dict()
{'name': 'Buğra İşgüzar', 'username': 'bugraisguzar', 'birthday': None, 'biography': 'geliş[email protected]', 'website': 'bisguzar.com', 'profile_photo': 'https://pbs.twimg.com/profile_images/1199305322474745861/nByxOcDZ_400x400.jpg', 'banner_photo': 'https://pbs.twimg.com/profile_banners/1019138658/1555346657/1500x500', 'likes_count': 2512, 'tweets_count': 756, 'followers_count': 483, 'following_count': 255, 'is_verified': False, 'is_private': False, user_id: "1019138658"}

Contributing to twitter-scraper

To contribute to twitter-scraper, follow these steps:

  1. Fork this repository.
  2. Create a branch with clear name: git checkout -b <branch_name>.
  3. Make your changes and commit them: git commit -m '<commit_message>'
  4. Push to the original branch: git push origin <project_name>/<location>
  5. Create the pull request.

Alternatively see the GitHub documentation on creating a pull request.

Contributors

Thanks to the following people who have contributed to this project:

  • @kennethreitz (author)
  • @bisguzar (maintainer)
  • @lionking6792
  • @ozanbayram
  • @xeliot

Contact

If you want to contact me you can reach me at @bugraisguzar.

License

This project uses the following license: MIT.

Owner
Buğra İşgüzar
⚕️Paramedic Hobbyist Pythonista
Buğra İşgüzar
Discord bot that automatically fills out health screenings

Auto Covid Bot Automatically fill out the NYC DOE health screening form by registering with a discord bot School code can be found on https://schoolse

Cleo 2 Jul 29, 2022
Senexia - A powerful telegram bot to manage your groups as effectively as possible

⚡ Kenechi bot ⚡ A Powerful, Smart And Simple Group Manager ... Written with AioG

Akhi 2 Jan 11, 2022
Discord-shell - Reverse shell accessible via discord.py bot

Discord-shell reverse shell on discord bot. (STILL IN DEVELOPMENT) Installation

Skultz 6 Oct 21, 2022
A Telegram Calculator to calculate your maths sums

CalculatorBot A Telegram Calculator to calculate your maths sums! Made by /Team

TeamOctave 2 Dec 31, 2021
QR login for pyrogram client

Generate Pyrogram session via QRlogin

ポキ 18 Oct 21, 2022
Modified Version Of Media Search bot

Modified Version Of Media Search bot

1 Oct 09, 2021
A python script fetches all your starred repositories from your GitHub account and clones them to your server so you will never lose important resources

A python script fetches all your starred repositories from your GitHub account and clones them to your server so you will never lose important resources

Ringo Hoffmann 27 Oct 01, 2022
Send pm to Admin - Telegram

Send pm to Admin - Telegram

Ahoora 3 Nov 17, 2022
Source code from thenewboston Discord Bot with Python tutorial series.

Project Setup Follow the steps below to set up the project on your environment. Local Development Create a virtual environment with Python 3.7 or high

Bucky Roberts 24 Aug 19, 2022
Volt is yet another discord api wrapper for Python. It supports python 3.8 +

Volt Volt is yet another discord api wrapper for Python. It supports python 3.8 + How to install [Currently Not Supported.] pip install volt.py Speed

Minjun Kim (Lapis0875) 11 Nov 21, 2022
An hcaptcha-solving discord account generator; capable of randomizing names, profile pictures, and verifying phone numbers.

discord-account-generator An hcaptcha-solving discord account generator; capable of randomizing names, profile pictures, and verifying phone numbers.

Acier 61 Dec 10, 2022
A simpler way to make forms, surveys, and reaction input using discord.py.

discord-ext-forms An easier way to make forms and surveys in discord.py. This module is a very simple way to ask questions and create complete forms i

thrizzle 16 Nov 06, 2022
A simple bot discord in PY with moderation controls

Voila un bot discord en py avec les commandes simples de modération tout simplement faut changer les lignes 70 vous mettez votre token de votre bot 53

Ethan 1 Nov 20, 2021
A powerful bot to copy your google drive data to your team drive

⚛️ Clonebot - Heroku version ⚡ CloneBot is a telegram bot that allows you to copy folder/team drive to team drives. One of the main advantage of this

MsGsuite 269 Dec 23, 2022
A Discord bot written in Python to help with guild administration

forgotten-hydra A Discord bot written in Python to help with guild administration. External libraries Pycord-Development/pycord 1.7.3 djc/couchdb-pyth

1 May 14, 2022
Telegram Link Wayback Bot. This bot archives a web page thrown at itself with wayback Machine (Archive.org).

Telegram Link Wayback Bot. This bot archives a web page thrown at itself with wayback Machine (Archive.org).

Hüzünlü Artemis [HuzunluArtemis] 11 Feb 18, 2022
Discord-Token-Formatter - A simple script to convert discord tokens from email token to token only format

Discord-Token-Formatter A simple script to convert discord tokens from email:pas

2 Oct 23, 2022
this is a telegram bot repository, that can stream video on telegram group video chat.

VIDEO STREAM BOT telegram bot project for streaming video on telegram video chat, powered by tgcalls and pyrogram 🛠 Commands: /vstream (reply to vide

levina 319 Aug 15, 2022
PyFacebook

== PyFacebook == PyFacebook is a Python client library for the Facebook API. Samuel Cormier-Iijima ( Samuel Cormier-Iijima 573 Dec 20, 2022

THERE IS AN IMPOSTER AMONG US. VOTE HIM OUT BEFORE HE [ R E D A C T E D ].

🛡️ Guardian There is an impostor among us. Can you help us find out who it is? ⚙️ Installation and Usage Make sure to install Tesseract-OCR before ru

Catus Magnus 1 Jan 06, 2022