調用外部API的最佳實踐是什麼？

Question

因此，假設我想寫一個使用Facebook API計算網站每頁上喜歡的蜘蛛。如果我導入請求庫，我可以按照以下方式調用Facebook圖形API。

import scrapy 
import json 
import requests 

API_KEY="KEY_GOES_HERE" 

class WebSite(scrapy.Spider): 
    name = "website_page" 
    allowed_domains = ["website.com"] 
    start_urls = ['https://website.com/'] 
    def get_likes(self,url): 
     base='https://graph.facebook.com/{}?access_token={}'.format(url,API_KEY) 
     data=requests.get(base) 
     return self.parse_likes(data) 
    def parse_likes(self, data): 
     data = json.loads(data.text) 
     return data['id'],data['share']['comment_count'],data['share']['share_count'] 

    def parse(self, response): 
     item= {} 
     item['url'] = response.url 
     links = response.css('a::attr(href)').extract() 
     item['fb_url'],item['shares'],item['comments'] = self.get_likes(response.url) 
     for link in links: 
      link = response.urljoin(link) 
      item['link'] = link 
      yield scrapy.Request(link, callback=self.parse) 
     yield item 

不過，我似乎無法獲得此代碼工作，如果，而不是使用的要求，我用的是scrapy.Request通話。像這樣的東西。

import scrapy 
import json 
import requests 

API_KEY="KEY_GOES_HERE" 

class WebSite(scrapy.Spider): 
    name = "website_page" 
    allowed_domains = ["website.com"] 
    start_urls = ['https://website.com/'] 
    def get_likes(self,url): 
     base='https://graph.facebook.com/{}?access_token={}'.format(url,API_KEY) 
     return scrapy.Request(base,callback=self.parse_likes) 
    def parse_likes(self, data): 
     data = json.loads(data.text) 
     return data['id'],data['share']['comment_count'],data['share']['share_count'] 

    def parse(self, response): 
     item= {} 
     links = response.css('a::attr(href)').extract() 
     item['url'] = response.url 
     item['fb_data']=self.get_likes(response.url).body 
     for link in links: 
      link = response.urljoin(link) 
      item['link'] = link 
      yield scrapy.Request(link, callback=self.parse) 
     yield item 

在這種情況下，我只是得到了Facebook數據的空白回覆。我想我缺少一些關於scrapy.Request方法如何相對於標準請求庫的理解。有任何想法嗎？

Answer 1

這是一個非常常見的情況：如何從多個url的項目中產生？
而最常見的解決方案是通過在request.meta參數中攜帶您的物品來鏈接請求。

對於這種邏輯的示例實現可能看起來像：

class WebSite(scrapy.Spider): 
    base='https://graph.facebook.com/{}?access_token={}'.format 
    api_key = '1234' 

    def parse(self, response): 
     links = response.css('a::attr(href)').extract() 
     for link in links: 
      item= {} 
      item['url'] = response.url 
      item['fb_data']=self.get_likes(response.url).body 
      item['link'] = response.urljoin(link) 
      api_url = self.base(self.api_key, link) 
      yield scrapy.Request(api_url, 
           callback=self.parse_likes, 
           meta={'item': item}) 

    def parse_likes(self, response): 
     item = response.meta['item'] 
     data = json.loads(data.text) 
     share_count = data['id'],data['share']['comment_count'],data['share']['share_count'] 
     item['share_count'] = share_count 
     yield item