Amazon Scraper: A command-line tool for scraping Amazon product data

Related tags

Command-line Toolspythonamazonweb-scrapingamazon-scraperamazon-scraping-libraryamazon-scraping
Overview

Amazon Product Scraper: 2021

Description

A command-line tool for scraping Amazon product data to CSV or JSON format(s).

Requirements

  • Python 3
  • pip3

Installation

Using git clone (you'll need git installed for this):

git clone https://github.com/scrapewalrus/amazon-scraper-python-2021

Or download and extract the zip file of the project manually

You'll also need to install requirements for the project to run. Locate amazon-product-scraper folder via terminal and type pip install -r requirements.txt:

Usage

To launch the Amazon scraper locate the amazon-product-scraper folder via terminal and type python amazon_scraper.py -k "your keyword". This will start the program.

NOTE: you must declare either -k or --keyword before entering your keyword. It's a required argument.

Example:

amazon_scraper.py - the name of a scraper file.

-k or --keyword - required argument to pass before entering your keyword.

-p or --proxies - optional argument to enable proxies. To avoid getting blocked I highly recommend using proxies. I'm using Residential Proxies from Oxylabs. For highest success rate, I suggest Residential Proxies over Datacenter as they're almost impossible to detect and have the smallest footprint. If you decide to use different proxy provider services keep in mind that you'll have to make some minor adjustments in get-proxies.py file.

-j or --json - optional argument for storing extracted data in .json format. Default output format is .csv.

Example of product data #1: JSON

[
    {
        "SOURCE_URL": "https://www.amazon.com/s?k=funny+t+shirt+for+women&page=1",
        "PAGE": 1,
        "KEYWORD": "funny t shirt for women",
        "PRODUCT_LINK": "https://www.amazon.com/Mostly-T-Shirt-Womens-Letter-Printed/dp/B07QN2NQ59/ref=sr_1_3?dchild=1&keywords=funny+t+shirt+for+women&qid=1627833682&sr=8-3",
        "PRODUCT_NAME": "I'm Mostly Peace Love and Light Funny T-Shirt Womens Graphic Printed Short Sleeve Tops Tee",
        "PRICE": "$21.99",
        "PRODUCT_RATING": "4.6",
        "NUMBER_OF_RATINGS": "1,637"
    },
    {
        "SOURCE_URL": "https://www.amazon.com/s?k=funny+t+shirt+for+women&page=1",
        "PAGE": 1,
        "KEYWORD": "funny t shirt for women",
        "PRODUCT_LINK": "https://www.amazon.com/YITAN-Women-Graphic-Funny-X-Large/dp/B074QMG4D7/ref=sr_1_4?dchild=1&keywords=funny+t+shirt+for+women&qid=1627833682&sr=8-4",
        "PRODUCT_NAME": "YITAN Women's Cute Juniors Tops Teen Girl Tee Funny T Shirt",
        "PRICE": "$12.99",
        "PRODUCT_RATING": "4.6",
        "NUMBER_OF_RATINGS": "12,281"
    },
    {
        "SOURCE_URL": "https://www.amazon.com/s?k=funny+t+shirt+for+women&page=1",
        "PAGE": 1,
        "KEYWORD": "funny t shirt for women",
        "PRODUCT_LINK": "https://www.amazon.com/DANVOUY-Womens-V-Neck-Doesnt-Definitely/dp/B07V55ZXVS/ref=sr_1_5?dchild=1&keywords=funny+t+shirt+for+women&qid=1627833682&sr=8-5",
        "PRODUCT_NAME": "DANVOUY Womens If My Mouth Doesn't Say It My Face Definitely Will T Shirt",
        "PRICE": "$12.99",
        "PRODUCT_RATING": "4.6",
        "NUMBER_OF_RATINGS": "6,787"
    }]

Example of product data #2: CSV

amazon-csv-product-data-example

Owner
GitHub Repository
Bringing emacs' greatest feature to neovim - Tetris!

nvim-tetris Bringing emacs' greatest feature to neovim - Tetris! This plugin is written in Fennel using Olical's project Aniseed for creating the proj

129 Dec 26, 2022
Loading animation; a progress bar

Loading animation; a progress bar. When you know the remaining time or task completion percentage, then you’re able to show an animated progress bar:

Goldy 1 Jan 23, 2022
lfb (light file browser) is a terminal file browser

lfb (light file browser) is a terminal file browser. The whole program is a mess as of now. In the feature I will remove the need for external dependencies, tidy up the code, make an actual readme, a

2 Apr 09, 2022
A set of libraries and functions for simplifying automating Cisco devices through SecureCRT.

This is a set of libraries for automating Cisco devices (and to a lesser extent, bash prompts) over ssh/telnet in SecureCRT.

Matthew Spangler 7 Mar 30, 2022
A Python package for a basic CLI and GUI user interface

Organizer CLI Organizer CLI is a python command line tool that goes through a given directory and organizes all un-folder bound files into folders by

Caltech Library 12 Mar 25, 2022
YouCompleteMe: a code-completion engine for Vim

YouCompleteMe: a code-completion engine for Vim Help, Advice, Support Looking for help, advice or support? Having problems getting YCM to work? First

24.5k Jan 06, 2023
CLI program that allows you to change your Alacritty config with one command without editing the config file.

Pycritty Change your alacritty config on the fly! Installation: pip install pycritty By default, only the program itself will be installed, but you ca

Antonio Sarosi 184 Jan 07, 2023
command line tool for frequent nmigen tasks (generate sources, show design)

nmigen-tool command line tool for frequent nmigen tasks (generate sources, show design) Usage: generate verilog: nmigen generate verilog nmigen_librar

Hans Baier 8 Nov 27, 2022
A Python module and command-line utility for converting .ANS format ANSI art to HTML

ansipants A Python module and command-line utility for converting .ANS format ANSI art to HTML. Installation pip install ansipants Command-line usage

4 Oct 16, 2022
A python program to detect your emotion and suggest a playlist.

mood_music A python program to detect your emotion and suggest a playlist. This program is written in python using opencv,FER() and tensorflow This pr

Aditya_Sai 1 Jan 02, 2022
Yet another bash/zsh prompt script

Here we have yet another script for Git-aware customization of the command prompt in Bash and zsh. Unlike all the other scripts, I wrote this one, so

John T. Wodder II 5 Oct 13, 2021
A command line application, written in Python, for interacting with Spotify.

spotify-py-cli A command line application, written in Python, for interacting with Spotify. The primary purpose behind developing this app was to gain

Drew Loukusa 0 Oct 07, 2021
A simple note taker CLI program written in python

note-taker A simple note taker program written in python This allows you to snip your todo's, notes, and your tasks easily without extra charges Requi

marcusz 4 Nov 02, 2021
Command-line tool to use LNURL with your LND instance

Sprint planner Sprint planner is a Python script for planning your Jira tasks based on your calendar availability. Installation Use the package manage

Djuri Baars 6 Jan 14, 2022
A linux-like remote terminal for Micropython

A linux-like remote terminal for Micropython

Christian Köver - Draxl 2 Nov 14, 2021
A collection of command-line interface games written in python

Command Line Interface Python Games Collection of some starter python game projects for beginners How to play these games Clone this repository git cl

Paras Gupta 7 Jun 06, 2022
Centauro - a command line tool with some network management functionality

Centauro Ferramenta de rede O Centauro é uma ferramenta de linha de comando com

1 Jan 01, 2022
adds flavor of interactive filtering to the traditional pipe concept of UNIX shell

percol __ ____ ___ ______________ / / / __ \/ _ \/ ___/ ___/ __ \/ / / /_/ / __/ / / /__/ /_/ / / / .__

Masafumi Oyamada 3.2k Jan 07, 2023
⚙ A lightweight command line interface library for creating commands.

⚙ A lightweight command line interface library for creating cli commands. About | Installation | Usage | Features | Contributors | License About Next:

Serum 16 Sep 25, 2022
A CLI tools to get you started on any project in any language

Any Template A faster easier to Quick start any programming project. Installation pip3 install any-template Features No third party dependencies. Tem

Adwaith Rajesh 2 Jan 11, 2022