Twayback: Downloading deleted Tweets from the Wayback Machine, made easy

Overview

Twayback: Downloading deleted Tweets from the Wayback Machine, made easy

forthebadge made-with-python

windows python

screenshot

Finding and downloading deleted Tweets takes a lot of time. Thankfully, with this tool, it becomes a piece of cake! πŸŽ‚

Twayback is a portmanteau of Twitter and the Wayback Machine. Enter your desired Twitter username, and let Twayback do the rest!

Requirements

  • Python 3
  • waybackpack
    • Install: pip install waybackpack or pip3 install waybackpack
    • Link to repo

Features

  • Can download some or all of a user's archived deleted Tweets.
  • Allows custom time range to narrow search for deleted Tweets archived between two dates.
  • Differentiates between accounts that are active, suspended, or don't/no longer exist.
  • Lets you know if a target handle's archived Tweets have been excluded from the Wayback Machine.

Usage

twayback -u USERNAME [OPTIONS]
Example: twayback -u jack

-u, --username        Specify target user's Twitter handle
-from, --fromdate     Narrow search for deleted Tweets *archived* on and after this date
                      (can be combined with -to)
                      (format YYMMDD)
-to, --todate         Narrow search for deleted Tweets *archived* on and before this date
                      (can be combined with -from)
                      (format YYMMDD)

Installation

For Windows only

  1. Download the latest EXE file.
  2. Launch Command Prompt in the EXE file's directory.
  3. Run the command twayback -u USERNAME (Replace USERNAME with your target handle).

For Windows, Linux, and macOS

  1. Download the latest ZIP file.
  2. Extract ZIP file to a directory of your choice.
  3. Open terminal in that directory.
  4. Run the command pip install -r requirements.txt.
  5. Run the command twayback -u USERNAME (Replace USERNAME with your target handle).

For more information, check out the Usage section above.

Things to keep in mind

  • Quality of the HTML files depends on how the Wayback Machine saved them. Some are better than others.
  • This tool is best for text. You might have some luck with photos. You cannot download videos.
  • By definition, if an account is suspended or no longer exists, all their Tweets would be considered deleted.
  • Custom date range is not about when Tweets were made, but rather when they were archived. For example, a Tweet from 2011 may have been archived today.

Future plans

  • GUI. This is a biggie. I don't know shit about Python, let alone GUI. But I'm hoping I can design one using Tkinter Designer. But I don't know how I can link actions to buttons and shit like that, that stuff is super foreign to me, so any help is appreciated, it would mean so much.

Plenty of thanks to jsvine for his amazing work on waybackpack. Without it, this tool cannot work nearly as well.

I hope you enjoy my little script. Please use it for good. Whatever you are, be a good one.

Comments
  • AttributeError: 'NoneType' object has no attribute 'getText'

    AttributeError: 'NoneType' object has no attribute 'getText'

    I got two errors:

    1. The first error occured after grabbing links from wayback machine UnboundLocalError: local variable 'wayback_id' referenced before assignment

    2. Second, the error occured when I typed 'text' or 'both' AttributeError: 'NoneType' object has no attribute 'getText'

    please help, thanks.

    bug good first issue 
    opened by adrn-mm 9
  • Error in the process

    Error in the process

    Hello,

    Thanks for the app which helps save a lot of time. I have tried with several users and in various ways and it does not work. I pass the screenshots of the errors.

    Thank you very much.

    image image

    opened by barripdmx 8
  • Twayback Partial Re-Write

    Twayback Partial Re-Write

    Hello,

    We recently came across your project, and thought we could contribute by re-writing some parts of the code. An effort was made to keep the logic and structure of the code the same.

    By our metrics, we have achieved a speed-up of around 20-30% for accounts with over 1000 tweets. We don't have any actual tests written for the script however, so we have attached it here for review.

    A substantial change was moving from Selenium to Playwright. Users need to run 'playwright install' to install the playwright browsers before running the script.

    Another smaller, but notable change is that due to the nature of the as_completed function, it is not possible to show any indicator of progress during the gathering of statuses, and the order in which the tweets are returned is scrambled due to the script processing website information based on what web request finishes first.

    If the style & format of the re-write is acceptable, and the changes proposed are not considered to be critical, we will proceed with creating a pull request.

    twayback2.txt

    opened by AccentuSoft 5
  • Parsing fails when encountering text in cyrillic

    Parsing fails when encountering text in cyrillic

    Hi, when I try to search for deleted tweets from an account that uses cyrillic, the process fails with the following exception:

    Parsing text...: 100%|β–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆβ–ˆ| 4/4 [00:07<00:00, 2.00s/it] Traceback (most recent call last): File "twayback.py", line 184, in File "encodings\cp1250.py", line 19, in encode UnicodeEncodeError: 'charmap' codec can't encode character '\u0430' in position 0: character maps to [20476] Failed to execute script 'twayback' due to unhandled exception!

    Anything that could be done on my end? Thanks,

    opened by Traut89now 5
  • KeyError: 'closest' when parsing accounts with larger number of tweets.

    KeyError: 'closest' when parsing accounts with larger number of tweets.

    Hi,

    after the latest update, the tool seems to be working perfectly when scraping accounts with lower number of tweets archived, but when I tried accounts with 1000+ tweets archived, the process failed with: Traceback (most recent call last): File "C:\15tway\twayback B\twayback.py", line 112, in wayback_url = (jsonResponse['archived_snapshots']['closest']['url']) KeyError: 'closest'

    as also seen below on the screenshots.

    image

    opened by Traut89now 4
  • No deleted tweets have been found

    No deleted tweets have been found

    Hi! Does anyone else, after they pull one search of an user, get an "No deleted tweets have been found" when they try to look for another one? I even try with the same one I did first and it pulls the same thing. I even close and open again the terminal but nothing, got the same issue. Was anyone run into this?

    opened by sofiemmc 3
  • Cleaned up code a bit

    Cleaned up code a bit

    No changes to functionality. I went through and added some more comments, renamed variables to be more descriptive and combined some redundancy with list creation

    opened by humandecoded 2
  • Feature proxy

    Feature proxy

    to avoid rate limiting for large groups of tweets added the ability to use proxies with our GET requests. User will need to provide a list of proxy URLs for the script to randomly pull from

    opened by humandecoded 1
  • Twitter Rate Limiting

    Twitter Rate Limiting

    It looks like twitter is throwing up 429 (too many requests) after the first 900 or so hits. According to their API documentation they limit to around 900 hits per 15 minutes. Although we are not using the API, it seems they are putting similar limiting in place per IP.

    I've begun working on a branch that lets users plug in a list of proxies the script will rotate through to avoid 429. This does make the tool less approachable but at this point it's not working on large groups of tweets. Will keep everyone updated

    enhancement 
    opened by humandecoded 1
  • Adding more Detailed information

    Adding more Detailed information

    I am submitting this PR so that newbie cyber people & OSINT Analysts with no python experience can better understand installation and usage.

    • There is no information about using the command git clone in order to clone the repository
    • No information in regards to changing directory | cd
    • No information in regards to the command pip3 install -r requirements.txt
    • No information in regards to the command python3 twayback.py -u username (if we put ourselves in the shoes of anyone without any python knowledge, they will just write in their terminal twayback.py which won't be of any use, it's typically something I would have done in the past as when reading installation & usage instructions I would just copy paste without trying to understand. 🀦

    Hope this helps a little, awesome tool & thanks for putting it out there !

    opened by C3n7ral051nt4g3ncy 1
  • UPDATE requirements.txt

    UPDATE requirements.txt

    Module aiohttp is required

    Traceback (most recent call last):
      File "/home/gdhindii/twayback/twayback.py", line 18, in <module>
        from aiohttp import ClientSession, TCPConnector
    ModuleNotFoundError: No module named 'aiohttp'
    
    opened by sam5epi0l 1
  • Proxy File List

    Proxy File List

    I am not sure this is an issue or just lack of knowledge from my part, I rather suspect it is the latter, however, I am attempting to recover tweets from an account that has a couple of thousand deleted tweets. I eventually encounter a rate limit error as is to be expected. I saw the way to handle requests like these is to use a proxy file. Now I have a .txt file that I added inside the twayback folder with a list of proxies and formatted according to the guidelines:

    url:port url:port url:port

    and so on. I keep getting error for each proxy server so Im not sure if Im doing this right and what the issue could be. It will stay stuck at 0%, switch between proxies until it eventually just gives up with a "" error.

    My request looks like this:

    python3 twayback.py --proxy-file proxyfiles.txt -u USERNAME

    Am I doing this correctly?

    opened by DirkGaston 2
  • it doesn't work anymore

    it doesn't work anymore

    once I launch the app it closes itself, it was working fine before the last update I tried downloading a pervious version and the same thing happened I tried it on 2 devices and it didn't work on any of them

    opened by unaufhaltbar 4
Releases(10-16-22)
Owner
He/Him. Good-for-nothing GitHubber.
A scriptable music downloader for Qobuz, Tidal, and Deezer

streamrip A scriptable stream downloader for Qobuz, Tidal, and Deezer. Features Downloads tracks, albums, playlists, discographies, and labels from Qo

967 Jan 03, 2023
Download Apple Music Cover Artwork in the best Quality by providing an Apple Music Link. It downloads the jpg, png and webp version since they often differ from another.

amogus.py - Version 0.0.5 amogus - Apple Music Hi-Res Artwork Fetcher this is my first real python tool so sorry if its bad amogus is a Python script

reaper 46 Jan 09, 2023
Ripurei is a free-to-use osu! replay downloader, that can be configured to download from any osu! server.

Ripurei Ripurei is a fully functional osu! replay downloader, fully capable of downloading from almost any osu! server. Functionality Timeline βœ”οΈ Able

Thomas 0 Feb 11, 2022
A cli tool to download purchased products from the DLsite.

dlsite-downloader A cli tool to download purchased products from the DLsite. How can I use? This program runs with configurations defined at settings.

AcrylicShrimp 9 Dec 23, 2022
πŸ”₯ A Bot To Telegram For Download High Qulity Videos & Songs From Youtube

πŸ”₯ A Bot To Telegram For Download High Qulity Videos & Songs From Youtube πŸŽ— Fast And Free Bot No Need To Pay βœ… By SL-Alpha-X-Team ⚑

Official Alpha-X-Team Account 7 Aug 31, 2022
Simple Youtube Video Downloader

Simple Youtube Video Downloader Download Youtube video using link and Will output result in D:/ (You can change the path in main.py file) Installation

Hansen Gianto 1 Oct 28, 2021
Command-line program to download videos from YouTube.com and other video sites

youtube-dl - download videos from youtube.com or other video platforms

youtube-dl 116.4k Jan 07, 2023
A manga download script written in python.

manga-dlp python script to download mangas Description A manga download script written in python. It only supports mangadex.org for now. But support f

Ivan Schaller 15 Nov 28, 2022
Vinetrimmer-DRM-TOOL - Widevine DRM downloader and decrypter for AMZN|NF|STAN And all

πŸƒ βœ‚οΈ Vinetrimmer Widevine DRM downloader and decrypter. Thanks to wvleaks for t

Vlad Tănăsescu 20 Jan 13, 2022
A simple Python program which uses youtube-dl for downloading YouTube videos as mp3 files.

yt-mp3 converter This is a simple Python program which uses youtube-dl for downloading YouTube videos as mp3 files. This program is for you if you are

nostalgicnerdpenguin 1 Oct 24, 2021
Downloads files and folders

PyDownloader Downloads files and folders at high speed (based on your interent speed). This is very useful to transfer big files from one computer to

ArmenG 4 Feb 24, 2022
Easily download audio described movies and TV shows found on audiovault.net

AudioVault Downloader A convenient downloader for audio described movies and TV shows found on the Audio Vault. get latest binary release for Windows

Carter Temm 5 Feb 10, 2022
Terminal based YouTube player and downloader

termitube NOTE: THIS REPOSITORY IS A FORK OF mps-youtube as mps-youtube has been unmaintained for almost a year now. Features Search and play audio/vi

Otis/Jacob Root 27 Dec 23, 2022
QGIS plugin to dwonload DEMs from OpenTopography.org

OpenTopography-DEM-Downloader-qgis-plugin QGIS plugin to dwonload DEMs from OpenTopography.org This plug-in allows you to download DEMs from OpenTopgr

Kyaw Naing Win 7 Sep 20, 2022
Youtube Downloader by PyTube Γ© uma ferramenta simples com interface grΓ‘fica e escrito em python para baixar vΓ­deos e playlists do youtube...

YouTube Downloader by PyTube O que Γ© o YouTube Downloader by PyTube? YouTube Downloader by PyTube Γ© um software simples para baixar vΓ­deos no YouTube

Elizeu Barbosa Abreu 5 Jul 30, 2022
File Downloader

File Downloader Watches a file containing download links and runs a command to download them. The link file is in form of: # comment DOWNLOAD_LINK

Pouriya 1 Jan 08, 2022
Code for "Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions"

Adversarial Motion Priors Make Good Substitutes for Complex Reward Functions Codebase for the "Adversarial Motion Priors Make Good Substitutes for Com

Alejandro Escontrela 54 Dec 13, 2022
YoutubeDownloader - Download any public Playlist from Youtube

YoutubeDownloader Download any public Youtube Channel / Playlist Features Bulk d

17 Nov 12, 2022
New York Times Front Page Downloader.

TIMETRAVELER New York Times Front Page Downloader. Usage python3 timetraveler.py All data will be saved at ~/timetraveler/ Goals To keep a historica

Daeshon Jones 0 Oct 31, 2021
YouTube-Downloader - YouTube Video Downloader made using python

YouTube-Downloader YouTube Videos Downloder made using python.

Shivam 1 Jan 16, 2022