A tool written in Python to download all Snapmaps content from a specific location.

Overview

snapmap-archiver

A tool written in Python to download all Snapmaps content from a specific location.

snapmap-archiver splash

Setup

pip3 install snapmap-archiver

View on PyPI

Install dependencies with pip3.

pip3 install -r requirements.txt

Install aria2c

Usage

python3 -m snapmap_archiver -o [OUTPUT DIR] -l="[LATITUDE],[LONGITUDE]"

Unfortunately you have to use the arbitrary -l="lat,lon" rather than just -l "lat,lon" when parsing negative numbers as argsparse interprets said numbers as extra arguments.

Optional Arguments

Export JSON

You can export a JSON file with info about downloaded snaps with the --write-json argument, which will contain information like the time the Snap was posted, and the Snap location.

Snap Radius

The radius from the coordinates you provide that will be included for downloads. -r 20000 will download all Snaps within a 20km radius of your coordinates.

Comments
  • Added support for merging video and overlay file into one video file

    Added support for merging video and overlay file into one video file

    Added support for merging video and overlay file into one video file using ffmpeg. You can disable this using the --no-overlay argument. This solves #3 Also added overlayText and filetype fields to the --write-json argument.

    opened by Gertje823 1
  • Merge overlay.png with media.mp4

    Merge overlay.png with media.mp4

    Snaps with text and stickers don't include said graphics in the video file, instead they're stored in an image called overlay.png, and displayed over the top by the browser/app.

    We could have an option like --merge-overlay which would use something like ffmpeg or avconv to put the image over the top of the media.mp4 file and export it as a new file.

    opened by king-millez 1
  • [Question] Downloading the snaps

    [Question] Downloading the snaps

    I notice you're using Aria for downloading the videos. Is there a significant enough time difference using the external library over say a piece of code like

    import requests
    ...
    with open(filename + '.mp4', 'w+') as f:
       # req_headers from get_data.py
       f.write(requests.get(snap['media']['raw_url'], headers=req_headers))
    

    now that above code most likely won't run out of the box but I feel removing the Aria requirement would make it more attractive to people as they could get it running straight from pip

    And sorry for opening an issue to discuss this, I don't know where the best place to talk about the code is.

    opened by AlanTheBlank 0
  • Bump urllib3 from 1.26.3 to 1.26.4

    Bump urllib3 from 1.26.3 to 1.26.4

    Bumps urllib3 from 1.26.3 to 1.26.4.

    Release notes

    Sourced from urllib3's releases.

    1.26.4

    :warning: IMPORTANT: urllib3 v2.0 will drop support for Python 2: Read more in the v2.0 Roadmap

    • Changed behavior of the default SSLContext when connecting to HTTPS proxy during HTTPS requests. The default SSLContext now sets check_hostname=True.

    If you or your organization rely on urllib3 consider supporting us via GitHub Sponsors

    Changelog

    Sourced from urllib3's changelog.

    1.26.4 (2021-03-15)

    • Changed behavior of the default SSLContext when connecting to HTTPS proxy during HTTPS requests. The default SSLContext now sets check_hostname=True.
    Commits

    Dependabot compatibility score

    Dependabot will resolve any conflicts with this PR as long as you don't alter it yourself. You can also trigger a rebase manually by commenting @dependabot rebase.


    Dependabot commands and options

    You can trigger Dependabot actions by commenting on this PR:

    • @dependabot rebase will rebase this PR
    • @dependabot recreate will recreate this PR, overwriting any edits that have been made to it
    • @dependabot merge will merge this PR after your CI passes on it
    • @dependabot squash and merge will squash and merge this PR after your CI passes on it
    • @dependabot cancel merge will cancel a previously requested merge and block automerging
    • @dependabot reopen will reopen this PR if it is closed
    • @dependabot close will close this PR and stop Dependabot recreating it. You can achieve the same result by closing it manually
    • @dependabot ignore this major version will close this PR and stop Dependabot creating any more for this major version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this minor version will close this PR and stop Dependabot creating any more for this minor version (unless you reopen the PR or upgrade to it yourself)
    • @dependabot ignore this dependency will close this PR and stop Dependabot creating any more for this dependency (unless you reopen the PR or upgrade to it yourself)
    • @dependabot use these labels will set the current labels as the default for future PRs for this repo and language
    • @dependabot use these reviewers will set the current reviewers as the default for future PRs for this repo and language
    • @dependabot use these assignees will set the current assignees as the default for future PRs for this repo and language
    • @dependabot use this milestone will set the current milestone as the default for future PRs for this repo and language

    You can disable automated security fix PRs for this repo from the Security Alerts page.

    dependencies 
    opened by dependabot[bot] 0
  • File naming

    File naming

    A better way to name the downloaded Snaps would be nice, rather than [SNAP_ID].mp4, which isn't really searchable. The data returned by utils.organise_data() could be used, which includes the Snap location and the timestamp from when it was posted.

    The actual API also contains some extra data for certain snaps, like the raw text used for a video.

    At the moment I'm thinking a good naming convention could be [MINOR_LOCATION] - [TIMESTAMP] - [SNAP_ID].mp4...

    opened by king-millez 0
  • Dynamically change radius to get maximum amount of relevant content

    Dynamically change radius to get maximum amount of relevant content

    When querying the API, you can pass the parameter radiusMeters, which will give you more/less specific content the lower/higher the number you provide is. Starting from 95000 and working down to 0, we could iterate through the maximum amount of content in a specific area. I'm thinking iterating by -2500 for values > 10000 would be good, then -1000 until 1000, then -100 to 100, then -10 to 0, arbitrarily ending with 1.

    opened by king-millez 0
Releases(2.0)
  • 2.0(Dec 23, 2022)

    Version 2.0

    Install with pip install snapmap-archiver

    Updated Codebase

    • Usable as a package
    • The SnapmapArchiver class can be used across projects for custom integration with other packages
    • More efficient: no more blank excepts!
    • Better API integration
    • Uses dict.get instead of countless except KeyError checks

    Installable

    • Package is now (properly) installable through pip
    • Now buildable with setup.py
    Source code(tar.gz)
    Source code(zip)
  • 1.3.1(Jan 6, 2022)

Owner
Canberra based developer. 15yo
Storing, versioning, and downloading files from S3 made as easy as using open() in Python. Caching included.

open(LARGE) Storing, versioning, and downloading files from S3 made as easy as using open() in Python. Caching included. Motivation Oftentimes, especi

András Schmelczer 2 Jan 30, 2022
MMDL (Mega Music Downloader) - A tool to easily download music.

mmdl - Mega Music Downloader What is mmdl ❓ TLDR: MMDL is a cli app which allows you to quickly and efficiently download one or multiple songs from Yo

techboy-coder 30 Dec 13, 2022
Easily download audio described movies and TV shows found on audiovault.net

AudioVault Downloader A convenient downloader for audio described movies and TV shows found on the Audio Vault. get latest binary release for Windows

Carter Temm 5 Feb 10, 2022
YT-Downloader is a Tool to download youtube video.

YT-Downloader YT-Downloader is a Tool to download youtube video.If you are looking for a simple video downloader tool Than This YT-Downloader may be u

Pradip Thapa 7 May 11, 2022
DYA ( Ditch YouTube API ) is a package created to power the user with YouTube Data API functionality without any API Key

Ditch YouTubeAPI (BETA) DYA ( Ditch YouTube API ) is a package created to power the user with YouTube Data API functionality without any API Key Detai

Sougata Jana 23 Dec 22, 2022
Fetch papers and metadata.

Fetch PubMed Central for open-access papers as well as Sci-Hub

4 Oct 31, 2022
Using Youtube downloader is the fast and easy way to download and save any YouTube video.

Youtube video downloader using Django Using Django as a backend along with pytube module to create Youtbue Video Downloader. https://yt-videos-downloa

Suman Raj Khanal 10 Jun 18, 2022
Python module to download all media from a CyberDrop gallery.

CyberDrop Downloader Intro Let's suppose you found out the Eva G (bby_gee) leak on https://cyberdrop.me/a/aWAt4TWY. You wish you could download the en

Quatrecentquatre 1 Dec 12, 2021
A simple Python program which uses youtube-dl for downloading YouTube videos as mp3 files.

yt-mp3 converter This is a simple Python program which uses youtube-dl for downloading YouTube videos as mp3 files. This program is for you if you are

nostalgicnerdpenguin 1 Oct 24, 2021
YouTube Video publisher using youtube-dl & ROS2🐢

YouTube-publisher-ROS2 Publish sensor_msgs/Image by "YouTube" 🤗 🤗 🤗 ! You don't have to use webcamera or your video to check demos. Purpose Quick d

Ar-Ray 5 Dec 04, 2022
Python utility to download jobs at seek.com.au

Job Seeker job_seeker is an utility to download data of a job search from seek.com.au into a csv file for data analysis and exploration Install using

PyBites 3 May 14, 2022
Python module to download all media from a GoFile gallery.

GoFile Downloader Setup First of all, clone this repository : ~$ git clone https://github.com/quatrecentquatre-404/gofile-downloader Second, oh wait..

Quatrecentquatre 61 Jan 01, 2023
1Fichier Download Manager.

1fichier-dl 1Fichier Download Manager. Features ⭐ Manage your downloads ⭐ Bypass time limits Credits All icons, including the app icon, were provided

manuGMG 470 Oct 08, 2022
A Celery application to collect data, download media and extract information from social media APIs

Project IBEX A Celery application to collect data, download media and extract information from social media APIs. Requirements You must have a Redis D

ibex 4 Dec 15, 2022
This is a tool to allow downloading any links from r/Roms Magethread

ILYFPR This is a tool to allow downloading any links from r/Roms Magethread Limitations: It downloads ALL roms of the system you choose. This will be

Erase 4 Nov 04, 2021
Downloads photos you saved from a specific profile.

instagram-download-saved A python script that downloads photos you saved from a specific profile. The only dependency is instaloader, an open-source p

Aviv 1 Dec 19, 2021
A simple Python +3.x script to download videos from Facebook.

Facebook Video Downloader A simple Python +3.x script to download videos from Facebook posts

Kerolos Atef Saber 1 Dec 03, 2021
Libretrofuzz - Fuzzy Retroarch thumbnail downloader

Fuzzy Retroarch thumbnail downloader In Retroarch, when you use the manual scann

8 Nov 26, 2022
Web Downloader With Python

Web Downloader Introduction This module will provide API to download the webpage components : html file, image file, css fil, javascript file, href li

3 Dec 28, 2022
A user-friendly GUI for the ZSpotify music downloader.

ZSpotifyGUI A user-friendly desktop app for ZSpotify music downloader for Windows, MacOs, and Linux Discord Server - Matrix Server - Gitea Mirror - Ma

94 Dec 17, 2022