Scrapy和MySQLdb

Question

我在Mac OS X Lion 10.7.5上使用Python 2.7。

我原來在安裝MySQLdb時同時使用了pip-2.7 install MySQL-python以及下載並運行了python2.7 setup.py build和python2.7 setup.py install。我嘗試了使用32位和64位安裝的MySQL和相應體系結構的這些不同的方法，但無濟於事。我的解決方案是安裝Macports。然後我使用Macports安裝了MySQL和MySQL-python（MySQLdb）。

我使用Wing IDE進行開發代碼，所以我切換到Macintosh版本的Python - 導入MySQLdb的作品。我還將Python的默認終端版本切換到此Macports版本，並通過命令行調用python來驗證它是默認版本 - 啓動了正確的版本。

所以，現在的問題：我正在使用scrapy來抓取電影網頁的信息。我的管道將抓取的數據導入數據庫，該數據庫使用前面提到的MySQLdb模塊。當我去命令行，cd到我的項目，並運行scrapy crawl MySpider，我得到以下錯誤：

raise ImportError, "Error loading object '%s': %s" % (path, e) 
ImportError: Error loading object 'BoxOfficeMojo.pipelines.BoxofficemojoPipeline': No module named MySQLdb.cursors 

我檢查，並確保我可以從python2.7外殼導入MySQLdb.cursors，所以我覺得這是與Python的scrapy的版本使用的問題...

::::: UPDATE :::::

以下是完整的回溯：

Traceback (most recent call last): 
    File "/usr/local/bin/scrapy", line 4, in <module> 
execute() 
    File "/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 131, in execute 
_run_print_help(parser, _run_command, cmd, args, opts) 
    File "/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 76, in _run_print_help 
func(*a, **kw) 
    File "/Library/Python/2.7/site-packages/scrapy/cmdline.py", line 138, in _run_command 
cmd.run(args, opts) 
    File "/Library/Python/2.7/site-packages/scrapy/commands/crawl.py", line 43, in run 
spider = self.crawler.spiders.create(spname, **opts.spargs) 
    File "/Library/Python/2.7/site-packages/scrapy/command.py", line 33, in crawler 
self._crawler.configure() 
    File "/Library/Python/2.7/site-packages/scrapy/crawler.py", line 41, in configure 
self.engine = ExecutionEngine(self, self._spider_closed) 
    File "/Library/Python/2.7/site-packages/scrapy/core/engine.py", line 63, in __init__ 
self.scraper = Scraper(crawler) 
    File "/Library/Python/2.7/site-packages/scrapy/core/scraper.py", line 66, in __init__ 
self.itemproc = itemproc_cls.from_crawler(crawler) 
    File "/Library/Python/2.7/site-packages/scrapy/middleware.py", line 50, in from_crawler 
return cls.from_settings(crawler.settings, crawler) 
    File "/Library/Python/2.7/site-packages/scrapy/middleware.py", line 29, in from_settings 
mwcls = load_object(clspath) 
    File "/Library/Python/2.7/site-packages/scrapy/utils/misc.py", line 39, in load_object 
    raise ImportError, "Error loading object '%s': %s" % (path, e) 
    ImportError: Error loading object 'BoxOfficeMojo.pipelines.BoxofficemojoPipeline': No module named MySQLdb.cursors 

:::::更新2 :::::

這裏是我當前的路徑：

$PATH 
    -bash:  /opt/local/bin:/opt/local/sbin:/usr/local/bin:/usr/local/sbin:~/bin:/Library/Frameworks/Python .framework/Versions/3.3/bin:/Library/Frameworks/Python.framework/Versions/3.3/bin:/Library/Frameworks/Python.framework/Versions/3.3/bin:/Library/Frameworks/Python.framework/Versions/3.3/bin:/Library/Frameworks/Python.framework/Versions/2.7/bin:/usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin: No such file or directory 

:: ALSO ::

我在固定的東西希望加入這個代碼 - 這是py27-mysql位置（MySQLdb），而是返回了同樣的錯誤：

import sys; sys.path.append("/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages") 

:: ALSO＃2 ::

這裏是我的pipeline-代碼我不知道它是否工作，因爲我不斷收到關於import錯誤，但認爲它可能會有所幫助：

from scrapy import log 
from twisted.enterprise import adbapi 
import time 
import MySQLdb.cursors 
import sys; sys.path.append("/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages") 
class BoxofficemojoPipeline(object): 

    def __init__(self): 
     print ('init') 
     self.dbpool = adbapi.ConnectionPool('MySQLdb', db = 'testdb', user='testuser', passwd='test', cursorclass=MySQLdb.cursors.DictCursor, charset='utf8', use_unicode=True) 



     def process_item(self, item, spider): 
     print('process') 
     query = self.dbpool.runInteraction(self._conditional_insert, item) #("""INSERT INTO Example_Movie (title, url, gross, release) VALUES (%s, %s, %s, %s)""", (item['title'].endcode('utf-8'), item['url'].encode('utf-8'), item['gross'].encode('utf-8'), item['release'].encode('utf-8'))) 
      query.addErrback(self.handle_error)#self.conn.commit() 

     return item 

     def _conditional_insert(self, tx, item): 
     print ('conditional insert') 
      #Create record if doesn't exist 
      #all this block run on it's own thread 

     tx.execute("select * from example_movie where url = %s", (item['url'],)) 
     result = tx.fetchone() 
     if result: 
       log.msg("Item already stored in db: %s" % item, level = log.DEBUG) 
     else: 
       tx.execute("insert into example_movie (title, url, gross, release) values (%s, %s, %s, %s)", (item['title'].encode('utf-8'), item['url'].encode('utf-8'), item['gross'].encode('utf-8'), item['release'].encode('utf-8'))) 
       log.msg("Item stored in db: %s" % item, level=log.DEBUG) 

     def handle_error(self, e): 
     print ('handle_error') 
     log.err(e) 

Answer 1

謝謝大家誰了閱讀和看看這個時間。我最後一次查看代碼後發現問題。我意識到，在我的管道代碼中，問題import MySQLdb.cursors出現在import sys; sys.path.append("/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages")（這是MySQLdb module的位置）之前。在import MySQLdb之前插入import sys; sys.path.append語句修復了import error問題。