Amazon Product Scraper: 2021
Description
A command-line tool for scraping Amazon product data to CSV or JSON format(s).
Requirements
- Python 3
- pip3
Installation
Using git clone (you'll need git installed for this):
git clone https://github.com/scrapewalrus/amazon-scraper-python-2021
Or download and extract the zip file of the project manually
You'll also need to install requirements for the project to run. Locate amazon-product-scraper folder via terminal and type pip install -r requirements.txt:
Usage
To launch the Amazon scraper locate the amazon-product-scraper folder via terminal and type python amazon_scraper.py -k "your keyword". This will start the program.
NOTE: you must declare either -k or --keyword before entering your keyword. It's a required argument.
Example:
amazon_scraper.py - the name of a scraper file.
-k or --keyword - required argument to pass before entering your keyword.
-p or --proxies - optional argument to enable proxies. To avoid getting blocked I highly recommend using proxies. I'm using Residential Proxies from Oxylabs. For highest success rate, I suggest Residential Proxies over Datacenter as they're almost impossible to detect and have the smallest footprint. If you decide to use different proxy provider services keep in mind that you'll have to make some minor adjustments in get-proxies.py file.
-j or --json - optional argument for storing extracted data in .json format. Default output format is .csv.
Example of product data #1: JSON
[
{
"SOURCE_URL": "https://www.amazon.com/s?k=funny+t+shirt+for+women&page=1",
"PAGE": 1,
"KEYWORD": "funny t shirt for women",
"PRODUCT_LINK": "https://www.amazon.com/Mostly-T-Shirt-Womens-Letter-Printed/dp/B07QN2NQ59/ref=sr_1_3?dchild=1&keywords=funny+t+shirt+for+women&qid=1627833682&sr=8-3",
"PRODUCT_NAME": "I'm Mostly Peace Love and Light Funny T-Shirt Womens Graphic Printed Short Sleeve Tops Tee",
"PRICE": "$21.99",
"PRODUCT_RATING": "4.6",
"NUMBER_OF_RATINGS": "1,637"
},
{
"SOURCE_URL": "https://www.amazon.com/s?k=funny+t+shirt+for+women&page=1",
"PAGE": 1,
"KEYWORD": "funny t shirt for women",
"PRODUCT_LINK": "https://www.amazon.com/YITAN-Women-Graphic-Funny-X-Large/dp/B074QMG4D7/ref=sr_1_4?dchild=1&keywords=funny+t+shirt+for+women&qid=1627833682&sr=8-4",
"PRODUCT_NAME": "YITAN Women's Cute Juniors Tops Teen Girl Tee Funny T Shirt",
"PRICE": "$12.99",
"PRODUCT_RATING": "4.6",
"NUMBER_OF_RATINGS": "12,281"
},
{
"SOURCE_URL": "https://www.amazon.com/s?k=funny+t+shirt+for+women&page=1",
"PAGE": 1,
"KEYWORD": "funny t shirt for women",
"PRODUCT_LINK": "https://www.amazon.com/DANVOUY-Womens-V-Neck-Doesnt-Definitely/dp/B07V55ZXVS/ref=sr_1_5?dchild=1&keywords=funny+t+shirt+for+women&qid=1627833682&sr=8-5",
"PRODUCT_NAME": "DANVOUY Womens If My Mouth Doesn't Say It My Face Definitely Will T Shirt",
"PRICE": "$12.99",
"PRODUCT_RATING": "4.6",
"NUMBER_OF_RATINGS": "6,787"
}]
Example of product data #2: CSV
2 Dec 28, 2021
5.9k Dec 30, 2022
19 Jul 20, 2022
1.7k Jan 01, 2023
2 Sep 21, 2022
51 Nov 29, 2022
1 Dec 11, 2021
13 Jan 07, 2022
2.5k Dec 28, 2022
20 Dec 22, 2022
2 Nov 14, 2021
12.4k Jan 08, 2023
4 Nov 14, 2022
1 Dec 26, 2021
3.4k Dec 30, 2022
1 Dec 24, 2021
14 Dec 17, 2022
6 Mar 12, 2022
12 Jan 25, 2021
5 Feb 20, 2022