当前位置:网站首页>Crawl the product data of Xiaomi Youpin app
Crawl the product data of Xiaomi Youpin app
2022-04-23 18:00:00 【Round programmer】
This paper aims to exchange learning , Don't use it for other purposes , Otherwise, we will be responsible for the consequences
Environmental Science linux+pycharm+anaconda
import csv
import requests
from lxml import etree
import re
import random
import json
from usere_agent import UA
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)
url = 'https://youpin.mi.com/app/shopv3/pipe'
headers1 = {
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'zh-CN,zh;q=0.9',
'Connection': 'keep-alive',
'Content-Length': '130',
'Content-Type': 'application/x-www-form-urlencoded',
'Host': 'youpin.mi.com',
'Origin': 'https://youpin.mi.com',
'Referer': 'https://youpin.mi.com/',
'User-Agent': UA)
}
headers2 = {
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'zh-CN,zh;q=0.9',
'Connection': 'keep-alive',
'Content-Length': '145',
'Content-Type': 'application/x-www-form-urlencoded',
'Cookie': ( Yours cookie),
'Origin': 'https://youpin.mi.com',
'Referer': 'https://youpin.mi.com/',
'User-Agent': UA,
}
headers3 = {
'Accept': '*/*',
'Accept-Encoding': 'gzip, deflate, br',
'Accept-Language': 'zh-CN,zh;q=0.9',
'Connection': 'keep-alive',
'Content-Length': '364',
'Content-Type': 'application/x-www-form-urlencoded',
'Host': 'youpin.mi.com',
'Origin': 'https://youpin.mi.com',
'Referer': 'https://youpin.mi.com/detail?gid={}'.format(str(id)),
'User-Agent': UA,
}
data1 = {
'mkbl_data': '{"result": {"model": "Homepage", "action": "GetGroup2ClassInfo", "parameters": {}}}',
}
req = requests.post(url=url, headers=headers1, data=data1, verify=False).json()
groups = req['result']['result']['mkbl_data']['groups']
c_name=[]
c_id=[]
for i in groups:
for j in i:
class1_name = j['class']['name']
ucid1 = j['class']['ucid']
c_name.append(class1_name)
c_id.append(ucid1)
for k in j['sub_class']:
class2_name = k['name']
ucid2 = k['ucid']
for i,j in zip(c_name,c_id):
s = requests.session()
s.headers.update(headers2)
data2 = {
'mkbl_data': '{"uClassList": {"model": "Homepage", "action": "BuildHome", "parameters": {"id": "' + str(
j) + '"}}}'
}
respon = s.post(url=url, data=data2, verify=False).json()
print(respon)
itemdata = respon['result']['uClassList']['mkbl_data']
for j in itemdata:
if 'content' in j:
content_name = j['content']['name']
ucid = j['content']['ucid']
for k in j['mkbl_data']:
try:
gid = k['gid'] ## goods ID
name = k['name'] ## Name of commodity
summary = k['summary'] ## Product introduction
pic_url = k['pic_url'] ## Commodity images
price_min = int(k['price_min']) / 100 ## Price
itemurl = k['url'] ## Product links
print(i,name,summary,pic_url,price_min,itemurl)
with open('/media/liu/_dde_data/project/spider/ supplier /xmyp/' + i + '.csv', 'a+') as f:
f_csv = csv.writer(f)
f_csv.writerows([(i,name,summary,pic_url,price_min,itemurl)])
except:
continue
版权声明
本文为[Round programmer]所创,转载请带上原文链接,感谢
https://yzsam.com/2022/04/202204230545316057.html
边栏推荐
- Generate verification code
- Auto. JS custom dialog box
- C language input and output (printf and scanf functions, putchar and getchar functions)
- 消费者灰度实现思路
- Halo open source project learning (II): entity classes and data tables
- 读取excel,int 数字时间转时间
- Click Cancel to return to the previous page and modify the parameter value of the previous page, let pages = getcurrentpages() let prevpage = pages [pages. Length - 2] / / the data of the previous pag
- Oil monkey website address
- JS get link? The following parameter name or value, according to the URL? Judge the parameters after
- Submit local warehouse and synchronize code cloud warehouse
猜你喜欢
Nat commun | current progress and open challenges of applied deep learning in Bioscience
Logic regression principle and code implementation
QTableWidget使用讲解
Element calculation distance and event object
JS forms the items with the same name in the array object into the same array according to the name
[UDS unified diagnostic service] (Supplement) v. detailed explanation of ECU bootloader development points (1)
An example of linear regression based on tensorflow
C# 网络相关操作
2022 judgment questions and answers for operation of refrigeration and air conditioning equipment
[UDS unified diagnostic service] v. diagnostic application example: Flash bootloader
随机推荐
纳米技术+AI赋能蛋白质组学|珞米生命科技完成近千万美元融资
undefined reference to `Nabo::NearestNeighbourSearch
How to read literature
On the method of outputting the complete name of typeID from GCC
positioner
ROS package NMEA_ navsat_ Driver reads GPS and Beidou Positioning Information Notes
油猴网站地址
Leak detection and vacancy filling (6)
C language array processing batch data
Flash operates on multiple databases
.105Location
re正則錶達式
Go语言JSON包使用
Go的Gin框架学习
Anchor location - how to set the distance between the anchor and the top of the page. The anchor is located and offset from the top
Laser slam theory and practice of dark blue College Chapter 3 laser radar distortion removal exercise
Go对文件操作
ES6
.104History
Operation of 2022 mobile crane driver national question bank simulation examination platform