將Scrapy數據保存到MySQL中相應的URL

Question

當前正在使用Scrapy。

我有一個存儲在MySQL數據庫中的URL列表。蜘蛛訪問這些URL，捕獲兩個目標信息（得分和計數）。我的目標是當Scrapy完成抓取時，它會在移動到下一個URL之前自動填充相應的列。

我是一個新手，我似乎無法讓保存部分正常工作。 得分和計數成功傳遞到數據庫。但它被保存爲新行，而不是關聯到源URL。

這裏是我的代碼： amazon_spider.py

import scrapy 
from whatoplaybot.items import crawledScore 
import MySQLdb 

class amazonSpider(scrapy.Spider): 
    name = "amazon" 
    allowed_domains = ["amazon.com"] 
    start_urls = [] 

    def parse(self, response): 
     print self.start_urls 

    def start_requests(self): 
     conn = MySQLdb.connect(
       user='root', 
       passwd='', 
       db='scraper', 
       host='127.0.0.1', 
       charset="utf8", 
       use_unicode=True 
       ) 
     cursor = conn.cursor() 
     cursor.execute(
      'SELECT url FROM scraped;' 
      ) 

     rows = cursor.fetchall() 

     for row in rows: 
      yield self.make_requests_from_url(row[0]) 
     conn.close() 

    def parse(self, response): 
     item = crawledScore() 
     item['reviewScore'] = response.xpath('//*[@id="avgRating"]/span/a/span/text()').re("[0-9,.]+")[0] 
     item['reviewCount'] = response.xpath('//*[@id="summaryStars"]/a/text()').re("[0-9,]+") 
     yield item 

pipelines.py

import sys 
import MySQLdb 

class storeScore(object): 
    def __init__(self): 
     self.conn = MySQLdb.connect(
      user='root', 
      passwd='', 
      db='scraper', 
      host='127.0.0.1', 
      charset="utf8", 
      use_unicode=True 
     ) 
     self.cursor = self.conn.cursor() 

    def process_item(self, item, spider): 
     try: 
      self.cursor.execute("""INSERT INTO scraped(score, count) VALUES (%s, %s)""", (item['reviewScore'], item['reviewCount'])) 
      self.conn.commit() 

     except MySQLdb.Error, e: 
      print "Error %d: %s" % (e.args[0], e.args[1]) 

      return item 

任何幫助和指導將非常感謝。

謝謝，夥計們。

Answer 1

按照這個步驟：

添加reviewURL場到您的crawledScore項目：

class crawledScore(scrapy.Item): 
    reviewScore = scrapy.Field() 
    reviewCount = scrapy.Field() 
    reviewURL = scrapy.Field() 

保存響應URL到項目[ 'reviewURL']：

def parse(self, response): 
    item = crawledScore() 
    item['reviewScore'] = response.xpath('//*[@id="avgRating"]/span/a/span/text()').re("[0-9,.]+")[0] 
    item['reviewCount'] = response.xpath('//*[@id="summaryStars"]/a/text()').re("[0-9,]+") 
    item['reviewURL'] = response.url 
    yield item 

最後，在您的管道文件中，在SERT或更新取決於你的邏輯：

INSERT：

def process_item(self, item, spider): 
    try: 
     self.cursor.execute("""INSERT INTO scraped(score, count, url) VALUES (%s, %s, %s)""", (item['reviewScore'], item['reviewCount'], item['reviewURL'])) 
     self.conn.commit() 
    except MySQLdb.Error, e: 
     print "Error %d: %s" % (e.args[0], e.args[1]) 

     return item 

UPDATE：

def process_item(self, item, spider): 
     try: 
      self.cursor.execute("""UPDATE scraped SET score=%s, count=%s WHERE url=%s""", (item['reviewScore'], item['reviewCount'], item['reviewURL'])) 
      self.conn.commit() 
     except MySQLdb.Error, e: 
      print "Error %d: %s" % (e.args[0], e.args[1]) 

      return item