Hi, I try to use pyspider, but I can't see the web interface when I click the web button after crawling a page, so I can't use CSS selector on pyspider. when I click the html button, it can show the page source. but on the demo.pyspider.org. everything is ok do you know what's wrong?

安装完成后，在command line输入pyspider，结果有：ImportError: dlopen(/usr/local/lib/python2.7/site-packages/pycurl.so, 2): Library not loaded: libssl.1.0.0.dylib Referenced from: /usr/local/lib/python2.7/site-packages/pycurl.so Reason: image not found

大神这是怎么回事啊？

另，我用的是mac os x，谢谢！

When adding a batch job, why do you want to wait until after the completion of the task to grab? For example, I added a URL 5W, I read the log is to wait until the completion of the 5W to start?

the command will show up only after I install pyspider of python2, can't work in python3

$ python3 /usr/local/bin/pyspider
[I 150114 16:17:57 result_worker:44] result_worker starting...
Exception in thread Thread-1:
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 2195, in _find_and_load_unlocked
AttributeError: '_MovedItems' object has no attribute '__path__'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/pyspider/scheduler/scheduler.py", line 418, in xmlrpc_run
    from six.moves.xmlrpc_server import SimpleXMLRPCServer
ImportError: No module named 'six.moves.xmlrpc_server'; 'six.moves' is not a package

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.4/threading.py", line 920, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.4/threading.py", line 868, in run
    self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.4/dist-packages/pyspider/scheduler/scheduler.py", line 420, in xmlrpc_run
    from SimpleXMLRPCServer import SimpleXMLRPCServer
ImportError: No module named 'SimpleXMLRPCServer'

[I 150114 16:17:57 scheduler:388] loading projects
[I 150114 16:17:57 processor:157] processor starting...
[I 150114 16:17:57 tornado_fetcher:387] fetcher starting...
Traceback (most recent call last):
  File "/usr/local/bin/pyspider", line 9, in <module>
    load_entry_point('pyspider==0.3.0', 'console_scripts', 'pyspider')()
  File "/usr/local/lib/python3.4/dist-packages/pyspider/run.py", line 532, in main
    cli()
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 610, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 590, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 916, in invoke
    return Command.invoke(self, ctx)
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 782, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/pyspider/run.py", line 144, in cli
    ctx.invoke(all)
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/pyspider/run.py", line 409, in all
    ctx.invoke(webui, **webui_config)
  File "/usr/local/lib/python3.4/dist-packages/click/core.py", line 416, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.4/dist-packages/pyspider/run.py", line 277, in webui
    app = load_cls(None, None, webui_instance)
  File "/usr/local/lib/python3.4/dist-packages/pyspider/run.py", line 47, in load_cls
    return utils.load_object(value)
  File "/usr/local/lib/python3.4/dist-packages/pyspider/libs/utils.py", line 348, in load_object
    module = __import__(module_name, globals(), locals(), [object_name])
  File "/usr/local/lib/python3.4/dist-packages/pyspider/webui/__init__.py", line 8, in <module>
    from . import app, index, debug, task, result, login
  File "/usr/local/lib/python3.4/dist-packages/pyspider/webui/app.py", line 79, in <module>
    template_folder=os.path.join(os.path.dirname(__file__), 'templates'))
  File "/usr/local/lib/python3.4/dist-packages/flask/app.py", line 319, in __init__
    template_folder=template_folder)
  File "/usr/local/lib/python3.4/dist-packages/flask/helpers.py", line 741, in __init__
    self.root_path = get_root_path(self.import_name)
  File "/usr/local/lib/python3.4/dist-packages/flask/helpers.py", line 631, in get_root_path
    loader = pkgutil.get_loader(import_name)
  File "/usr/lib/python3.4/pkgutil.py", line 467, in get_loader
    return find_loader(fullname)
  File "/usr/lib/python3.4/pkgutil.py", line 488, in find_loader
    return spec.loader
AttributeError: 'NoneType' object has no attribute 'loader'

Could you please add sample config.json file for list of valid parameters? e.g

{
  "webui": {
"host": "127.0.0.1",
 "port": "5501"
 },
...
}

首先，我改了改wiki，因为不加 :latest 会自动下载所有tags 然后，觉得我们这个项目可以用fig操作docker跑起来，效果更好

首先，新建一个目录pyspider 下载这个命名为fig.yml http://p.esd.cc/paste/wp5ELQ2M (手机打的代码 没有测试过抱歉) 然后fig up即可！

安装fig pip install -U fig

代码中使用了本地的一个python文件，将抓取的结果写入mysql中，把此文件放在\database\mysql文件夹下运行代码时提示导入出错，找不到文件，小问题不知道怎么解决。看过issues中有说导入project当做模块的feature，但是不知道怎么用，请指导。 附带的一个问题是，如果要替换数据库，需要重写on_result(self,result)函数，有没有示例参考下。

class Handler(BaseHandler): configuration = {'a' : 'b', 'c' : 'd'}

@every(minutes=12 * 60)
def on_start(self):
    self.configuration = {'a' : 'a', 'c' : 'c'}

@config(age= 12 * 60 * 60)
def index_page(self, response):
    print(self.configuration)

I changed configuration in on_start. But in index_page, it still print {'a' : 'b', 'c' : 'd'}. How can I define a global variable?

Hi，我安装后，启动run.py，可以正常进入web页面，但是一旦报错task任务。就会出错：

Traceback (most recent call last): File "/home/zfz/spider/stack/python-2.7.8/lib/python2.7/logging/init.py", line 859, in emit msg = self.format(record) File "/home/zfz/spider/stack/python-2.7.8/lib/python2.7/logging/init.py", line 732, in format return fmt.format(record) File "/home/zfz/spider/src/pyspider/pyspider/libs/log.py", line 121, in format formatted = formatted.rstrip() + "\n" + _unicode(record.exc_text) File "/home/zfz/spider/src/pyspider/pyspider/libs/log.py", line 27, in _unicode raise e UnicodeDecodeError: 'gb18030' codec can't decode bytes in position 670-671: illegal multibyte sequence Logged from file scheduler.py, line 354 Traceback (most recent call last): File "/home/zfz/spider/stack/python-2.7.8/lib/python2.7/logging/init.py", line 859, in emit msg = self.format(record) File "/home/zfz/spider/stack/python-2.7.8/lib/python2.7/logging/init.py", line 732, in format return fmt.format(record) File "/home/zfz/spider/src/pyspider/pyspider/libs/log.py", line 121, in format formatted = formatted.rstrip() + "\n" + _unicode(record.exc_text) File "/home/zfz/spider/src/pyspider/pyspider/libs/log.py", line 27, in _unicode raise e UnicodeDecodeError: 'gb18030' codec can't decode bytes in position 670-671: illegal multibyte sequence Logged from file scheduler.py, line 354 Traceback (most recent call last): File "/home/zfz/spider/stack/python-2.7.8/lib/python2.7/logging/init.py", line 859, in emit msg = self.format(record) File "/home/zfz/spider/stack/python-2.7.8/lib/python2.7/logging/init.py", line 732, in format return fmt.format(record) File "/home/zfz/spider/src/pyspider/pyspider/libs/log.py", line 121, in format formatted = formatted.rstrip() + "\n" + _unicode(record.exc_text) File "/home/zfz/spider/src/pyspider/pyspider/libs/log.py", line 27, in _unicode raise e UnicodeDecodeError: 'gb18030' codec can't decode bytes in position 670-671: illegal multibyte sequence Logged from file scheduler.py, line 354 Traceback (most recent call last): File "/home/zfz/spider/stack/python-2.7.8/lib/python2.7/logging/init.py", line 859, in emit msg = self.format(record) File "/home/zfz/spider/stack/python-2.7.8/lib/python2.7/logging/init.py", line 732, in format return fmt.format(record) File "/home/zfz/spider/src/pyspider/pyspider/libs/log.py", line 121, in format formatted = formatted.rstrip() + "\n" + _unicode(record.exc_text) File "/home/zfz/spider/src/pyspider/pyspider/libs/log.py", line 27, in _unicode raise e UnicodeDecodeError: 'gb18030' codec can't decode bytes in position 670-671: illegal multibyte sequence Logged from file scheduler.py, line 354

此后所有web请求全是500错误。操作系统版本centos 6.5，python环境2.7.8。

另外发现默认的是sqlite数据库，我想换用mysql，没找到在哪里配置？谢谢。

hi, pyspider is very nice to manage many crawler projects. but i have problem of how to store results into database. I see your tutorial on doc.pyspider.org. here is some tutorial on how to do with results

from pyspider.result import ResultWorker

Class MyResultWorker(ResultWorker): def on_result(self, task, result): assert task['taskid'] assert task['project'] assert task['url'] assert result #save result to database can i use this way in the script to save crawler results into database automatically?

code as follow: from pyspider.libs.base_handler import *

class Handler(BaseHandler): crawl_config = {'headers': { 'Content-Type':'application/x-www-form-urlencoded', 'Accept':'/', 'Accept-Encoding':'gzip, deflate', 'Accept-Language':'zh-CN,zh;q=0.8', 'Cache-Control':'max-age=0', 'Connection':'keep-alive', 'Content-Length':'295', 'X-Requested-With': 'XMLHttpRequest', 'Cookie':'__utmt=1; __utma=89664858.1557390068.1472454301.1472628080.1472628080.6; __utmb=89664858.3.10.1472628080; __utmc=89664858; __utmz=89664858.1472628080.5.5.utmcsr=sogou|utmccn=(organic)|utmcmd=organic|utmctr=phytozome', 'Host':'phytozome.jgi.doe.gov', 'Origin':'https://phytozome.jgi.doe.gov', 'Referer':'https://phytozome.jgi.doe.gov/pz/portal.html', 'User-Agent':'Mozilla/5.0 (X11; Linux i686) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/48.0.2564.116 Safari/537.36', 'X-GWT-Module-Base':'https://phytozome.jgi.doe.gov/pz/phytoweb/', 'X-GWT-Permutation':'80DA602CF8FBCB99E9D79278AD2DA616', } }

@every(minutes=24 * 60)
def on_start(self):
    self.crawl('https://phytozome.jgi.doe.gov/pz/portal.html#!results?search=0&crown=1&star=1&method=2296&searchText=AUX/IAA&offset=0',callback=self.detail_page,fetch_type='js')

def index_page(self, response):
    for each in response.doc('*').items():
        self.crawl(each.attr.href, callback=self.detail_page,fetch_type='js')

@config(priority=2)
def detail_page(self, response):
    self.index_page(response)
    for each in response.doc('*').items():
        self.crawl(each.attr.href, callback=self.detail_page,fetch_type='js')
    return {
        "url": response.url,
        "content":response.doc("*").text()
    }

只能抓取到css

What happened？

There are 1 security vulnerabilities found in lxml 4.3.3

• MPS-2022-14974

What did I do？

Upgrade lxml from 4.3.3 to 4.9.1 for vulnerability fix

What did you expect to happen？

Ideally, no insecure libs should be used.

The specification of the pull request

PR Specification from OSCS Signed-off-by:pen4[email protected]

What happened？

There are 1 security vulnerabilities found in tornado 4.5.3

• CVE-2018-1000518

What did I do？

Upgrade tornado from 4.5.3 to 5.1 for vulnerability fix

What did you expect to happen？

Ideally, no insecure libs should be used.

The specification of the pull request

PR Specification from OSCS

Problem

On aarch64, ‘pip install pyspider’ is giving the below error -

ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-w68_cof2/pycurl/setup.py'"'"'; __file__='"'"'/tmp/pip-install-w68_cof2/pycurl/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-w68_cof2/pycurl/pip-egg-info
         cwd: /tmp/pip-install-w68_cof2/pycurl/
    Complete output (22 lines):
    Traceback (most recent call last):
      File "/tmp/pip-install-w68_cof2/pycurl/setup.py", line 235, in configure_unix
        p = subprocess.Popen((self.curl_config(), '--version'),
      File "/usr/lib/python3.8/subprocess.py", line 858, in __init__
        self._execute_child(args, executable, preexec_fn, close_fds,
      File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
        raise child_exception_type(errno_num, err_msg, err_filename)
    FileNotFoundError: [Errno 2] No such file or directory: 'curl-config'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-w68_cof2/pycurl/setup.py", line 1017, in <module>
        ext = get_extension(sys.argv, split_extension_source=split_extension_source)
      File "/tmp/pip-install-w68_cof2/pycurl/setup.py", line 673, in get_extension
        ext_config = ExtensionConfiguration(argv)
      File "/tmp/pip-install-w68_cof2/pycurl/setup.py", line 99, in __init__
        self.configure()
      File "/tmp/pip-install-w68_cof2/pycurl/setup.py", line 240, in configure_unix
        raise ConfigurationError(msg)
    __main__.ConfigurationError: Could not run curl-config: [Errno 2] No such file or directory: 'curl-config'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Resolution

On aarch64, ‘pip install pyspider’ should download the wheels from PyPI.

@binux and Team, Please let me know your interest in releasing aarch64 wheels. I can help in this.

[W 220404 15:02:18 run:413] phantomjs not found, continue running without it. [I 220404 15:02:20 result_worker:49] result_worker starting... [I 220404 15:02:20 processor:211] processor starting... [I 220404 15:02:20 tornado_fetcher:638] fetcher starting... [I 220404 15:02:20 scheduler:647] scheduler starting... [I 220404 15:02:20 scheduler:782] scheduler.xmlrpc listening on 127.0.0.1:23333 [I 220404 15:02:20 scheduler:586] in 5m: new:0,success:0,retry:0,failed:0 [I 220404 15:02:20 app:84] webui exiting... Traceback (most recent call last): File "/usr/local/Caskroom/miniconda/base/envs/web/bin/pyspider", line 8, in sys.exit(main()) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 754, in main cli() File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1128, in call return self.main(*args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1053, in main rv = self.invoke(ctx) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1637, in invoke super().invoke(ctx) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 1395, in invoke return ctx.invoke(self.callback, **ctx.params) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 165, in cli ctx.invoke(all) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 497, in all ctx.invoke(webui, **webui_config) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/core.py", line 754, in invoke return __callback(*args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/click/decorators.py", line 26, in new_func return f(get_current_context(), *args, **kwargs) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/run.py", line 384, in webui app.run(host=host, port=port) File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/webui/app.py", line 59, in run from .webdav import dav_app File "/usr/local/Caskroom/miniconda/base/envs/web/lib/python3.6/site-packages/pyspider/webui/webdav.py", line 207, in '/': ScriptProvider(app) TypeError: Can't instantiate abstract class ScriptProvider with abstract methods get_resource_inst