如何调试 scrapy 管道？

Question

我正在关注 this tutorial 以学习如何一起使用 scrapy 和 mongodb。但是，我不断收到这些错误消息：

[Anaconda2] C:\Users\Segovia\Dropbox\stack>scrapy crawl stack
Traceback (most recent call last):
  File "c:\users\segovia\anaconda2\lib\runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "c:\users\segovia\anaconda2\lib\runpy.py", line 72, in _run_code
    exec code in run_globals
  File "C:\Users\Segovia\Anaconda2\Scripts\scrapy.exe\__main__.py", line 9, in <module>
  File "c:\users\segovia\anaconda2\lib\site-packages\scrapy\cmdline.py", line 108, in execute
    settings = get_project_settings()
  File "c:\users\segovia\anaconda2\lib\site-packages\scrapy\utils\project.py", line 60, in get_project_settings
    settings.setmodule(settings_module_path, priority='project')
  File "c:\users\segovia\anaconda2\lib\site-packages\scrapy\settings\__init__.py", line 285, in setmodule
    self.set(key, getattr(module, key), priority)
  File "c:\users\segovia\anaconda2\lib\site-packages\scrapy\settings\__init__.py", line 260, in set
    self.attributes[name].set(value, priority)
  File "c:\users\segovia\anaconda2\lib\site-packages\scrapy\settings\__init__.py", line 55, in set
    value = BaseSettings(value, priority=priority)
  File "c:\users\segovia\anaconda2\lib\site-packages\scrapy\settings\__init__.py", line 91, in __init__
    self.update(values, priority)
  File "c:\users\segovia\anaconda2\lib\site-packages\scrapy\settings\__init__.py", line 317, in update
    for name, value in six.iteritems(values):
  File "c:\users\segovia\anaconda2\lib\site-packages\six.py", line 599, in iteritems
    return d.iteritems(**kw)
AttributeError: 'list' object has no attribute 'iteritems'

谁能告诉我可能出了什么问题？或者也许有人可以给我提示如何调试它？我已经尝试了 scrapy 官方文档中提供的 'parse' 方法，但它对我不起作用。为了调试它，我希望我可以使用 IDE 和 'step-in' 这些代码并详细检查发生了什么。谢谢！

settings.py 文件中有这些行：

ITEM_PIPELINES = ['stack.pipelines.MongoDBPipeline', ]

MONGODB_SERVER = "localhost"
MONGODB_PORT = 27017
MONGODB_DB = "Whosebug"
MONGODB_COLLECTION = "questions"

而且我确定 'mongod' 在另一个命令 window 中是运行。

Answer 1

我们来看错误：

AttributeError: 'list' object has no attribute 'iteritems'

在你的这部分项目设置：

ITEM_PIPELINES = ['stack.pipelines.MongoDBPipeline', ]

而在这个 documentation page.

Scrapy 期望 ITEM_PIPELINES 是一个字典，而你给它一个列表。修复它：

ITEM_PIPELINES = {'stack.pipelines.MongoDBPipeline': 300}

如何调试 scrapy 管道？

How to debug a scrapy pipeline?

python

debugging

pipeline

mongodb

scrapy