如何檢索數據......該頁面使用AJAX

Question

我想從這個網站 http://www.univercell.in/buy/SMART

我試圖測試獲得手機的成本，所以我用裝： scarpy殼http://www.univercell.in/control/AjaxCategoryDetail?productCategoryId=PRO-SMART&category_id=PRO-SMART&attrName=&min=&max=&sortSearchPrice=&VIEW_INDEX=2&VIEW_SIZE=15&serachupload=&sortupload=

但我無法連接到此網站。當頁面使用ajax加載時，我使用螢火蟲發現了start_url。任何人都可以向我建議我錯誤的地方

Answer 1

如何編寫JavaScript腳本來執行單擊頁碼時已執行的操作，然後轉儲從服務器返回的XML。我的意思是嘗試撥打服務器的電話，就好像該網站託管在桌面上一樣！

當您點擊某個號碼時調用的JavaScript函數是paginateList('numberOfPage')，其中numberOfPage是您想要訪問的頁面。

功能的主體是

function paginateList(viewIndex) { 
     var productCategoryId = document.pageSelect.category_id.value; 
     var viewSize = document.pageSelect.VIEW_SIZE.value; 
     var min = ""; 
     if(document.pageSelect.min!=null) 
      min = document.pageSelect.min.value; 
     var max = ""; 
     if(document.pageSelect.max!=null) 
      max = document.pageSelect.max.value; 
     var attrName = ""; 
     if(document.pageSelect.attrName!=null) 
     attrName = document.pageSelect.attrName.value; 
     if(attrName==""){ 
     var commaAttr=document.getElementById('commaAttr'); 
      attrName=commaAttr.value; 
      } 
     var limitView = 'true'; 
     var sortSearchPrice = ""; 
     if(document.pageSelect.sortSearchPrice!=null) 
     sortSearchPrice = document.pageSelect.sortSearchPrice.value; 
      var url2="/control/AjaxCategoryDetail?productCategoryId="+productCategoryId+"&category_id="+productCategoryId+"&attrName="+attrName+"&min="+min+"&max="+max+"&sortSearchPrice="+sortSearchPrice+"&VIEW_INDEX="+viewIndex+"&VIEW_SIZE="+viewSize+"&serachupload=&sortupload="; 
      pleaseWait('Y'); 
      jQuery.ajax({url: url2, 
      data: null, 
      type: 'post', 
      async: false, 
      success: function(data) { 
       $('#searchResult').html(data); 
       pleaseWait('N'); 
      }, 
      error: function(data) { 
       alert("Error during product searching"); 
      } 
     }); 

使用這些從每個頁面獲取數據遞歸。

希望它有幫助！

Answer 2

這是你的蜘蛛：

from scrapy.item import Item, Field 
from scrapy.spider import BaseSpider 
from scrapy.selector import HtmlXPathSelector 


class UnivercellItem(Item): 
    vendor = Field() 
    model = Field() 
    price = Field() 

BASE_URL = "http://www.univercell.in/control/AjaxCategoryDetail?productCategoryId=PRO-SMART&category_id=PRO-SMART&attrName=&min=&max=&sortSearchPrice=&VIEW_INDEX=%s&VIEW_SIZE=15&serachupload=&sortupload=" 

class UnivercellSpider(BaseSpider): 
    name = "univercell_spider" 
    allowed_domains = ["www.univercell.in"] 
    start_urls = [BASE_URL % index for index in range(1, 21)] 

    def parse(self, response): 
     hxs = HtmlXPathSelector(response) 
     mobiles = hxs.select("//div[@class='productsummary']") 
     print mobiles 
     for mobile in mobiles: 
      item = UnivercellItem() 
      item['vendor'] = mobile.select('.//div[1]/div/text()').extract()[0].strip() 
      item['model'] = mobile.select('.//div[3]/div[1]/a/text()').extract()[0].strip() 
      item['price'] = mobile.select('.//span[@class="regularPrice"]/span/text()').extract()[0].strip() 
      yield item 

保存到spider.py，並通過scrapy runspider spider.py -o output.json運行。然後在output.json你會看到：

{"model": "T375", "vendor": "LG", "price": "Special Price Click Here"} 
{"model": "P725 Optimus 3D Max", "vendor": "LG", "price": "Special Price Click Here"} 
{"model": "P705 Optimus L7", "vendor": "LG", "price": "Special Price Click Here"} 
{"model": "9320 Curve", "vendor": "Blackberry", "price": "Special Price Click Here"} 
{"model": "Xperia Sola", "vendor": "Sony", "price": "Rs.14,500.00"} 
{"model": "Xperia U", "vendor": "Sony", "price": "Special Price Click Here"} 
{"model": "Lumia 610", "vendor": "Nokia", "price": "Special Price Click Here"} 
... 

希望有所幫助。