Casperjs使用casper.each

Question

遍歷鏈接列表我試圖使用Casperjs從頁面獲取鏈接列表，然後打開每個鏈接，並向這些數組中添加特定類型的數據頁面。

我遇到的問題是在每個列表項上執行的循環。

首先，我從原始頁面獲得listOfLinks。這部分工作和使用長度，我可以檢查這個列表是否填充。

但是，如下所示使用循環語句this.each，沒有任何控制檯語句顯示出來，並且casperjs似乎跳過該塊。

用標準for循環替換this.each，執行只是部分通過第一個鏈接，因爲語句「在x.html中創建對象的新數組」會出現一次，然後代碼停止執行。使用IIFE不會改變這一點。

編輯：在詳細調試模式下會發生以下情況：

Creating new array object for https://example.com 
[debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=true 

所以出於某種原因傳遞到thenOpen功能的URL被改爲空白......

我覺得有一些關於Casperjs的異步性質，我不是在這裏掌握的，希望能夠指出一個實際的例子。

casper.then(function() { 

    var date = Date.now(); 
    console.log(date); 

    var object = {}; 
    object[date] = {}; // new object for date 

    var listOfLinks = this.evaluate(function(){ 
    console.log("getting links"); 
    return document.getElementsByClassName('importantLink'); 
    }); 

    console.log(listOfLinks.length); 

    this.each(listOfLinks, function(self, link) { 

    var eachPageHref = link.href; 

    console.log("Creating new array in object for " + eachPageHref); 

    object[date][eachPageHref] = []; // array for page to store names 

    self.thenOpen(eachPageHref, function() { 

     var listOfItems = this.evaluate(function() { 
     var items = []; 
     // Perform DOM manipulation to get items 
     return items; 
     }); 
    }); 

    object[date][eachPageHref] = items; 

    }); 
    console.log(JSON.stringify(object)); 

}); 

Answer 1

我決定使用我們自己的Stackoverflow.com作爲演示站點來運行您的腳本。我在代碼中修正了一些小問題，其結果就是從PhantomJS賞金問題中獲得評論。

var casper = require('casper').create(); 

casper 
.start() 
.open('http://stackoverflow.com/questions/tagged/phantomjs?sort=featured&pageSize=30') 
.then(function() { 

    var date = Date.now(), object = {}; 
    object[date] = {}; 

    var listOfLinks = this.evaluate(function(){ 

     // Getting links to other pages to scrape, this will be 
     // a primitive array that will be easily returned from page.evaluate 
     var links = [].map.call(document.querySelectorAll("#questions .question-hyperlink"), function(link) { 
      return link.href; 
     });  
     return links; 
    }); 

    // Now to iterate over that array of links 
    this.each(listOfLinks, function(self, eachPageHref) { 

     object[date][eachPageHref] = []; // array for page to store names 

     self.thenOpen(eachPageHref, function() { 

      // Getting comments from each page, also as an array 
      var listOfItems = this.evaluate(function() { 
       var items = [].map.call(document.getElementsByClassName("comment-text"), function(comment) { 
        return comment.innerText; 
       });  
       return items; 
      }); 
      object[date][eachPageHref] = listOfItems; 
     }); 
    }); 

    // After each links has been scraped, output the resulting object 
    this.then(function(){ 
     console.log(JSON.stringify(object)); 
    }); 
}) 

casper.run(); 

什麼改變：page.evaluate現在返回簡單數組，這是需要casper.each（）來正確地迭代。 href屬性在page.evaluate中立即提取。另外這個修正：

object[date][eachPageHref] = listOfItems; // previously assigned items which were undefined in this scope 

腳本運行的結果是

{"1478596579898":{"http://stackoverflow.com/questions/40410927/phantomjs-from-node-on-windows":["en.wikipedia.org/wiki/File_URI_scheme – Igor 2 days ago\n","@Igor is there something in particular you see wrong, or are you suggesting the phantom module has an incorrect URI? – Danny Buonocore 2 days ago\n","Probably windows security issue not allowing to run an unsigned program. – Vaviloff yesterday\n"],"http://stackoverflow.com/questions/40412726/casperjs-iterating-over-a-list-of-links-using-casper-each":["Thanks, this looked really promising. I made the changes but it didn't solve the problem. And I just realised that in debug mode the following happens: Creating new array object for https://example.com [debug] [phantom] Navigation requested: url=about:blank, type=Other, willNavigate=true, isMainFrame=true and then Casperjs silently fails. It seems that the correct link that gets passed into thenOpen gets changed to about:blank... – cyc665 yesterday\n"]}} 

Answer 2

您在evaluate()功能，這是不允許的返回DOM節點。您可以改爲返回實際的網址。

注意：參數和評估函數的返回值必須是簡單的原始對象。經驗法則：如果它可以通過JSON序列化，那麼它很好。

閉包，函數，DOM節點等不起作用！

參考：PhantomJS#evaluate

Answer 3

如果我正確理解你的問題，要解決，給項目[]全球範圍。在你的代碼中，我會做到以下幾點：

var items = []; 
this.each(listOfLinks, function(self, link) { 

    var eachPageHref = link.href; 

    console.log("Creating new array in object for " + eachPageHref); 

    object[date][eachPageHref] = []; // array for page to store names 

    self.thenOpen(eachPageHref, function() { 

     this.evaluate(function() { 
     // Perform DOM manipulation to get items 
     items.push(whateverThisItemIs); 
     }); 
    }); 

希望這會有所幫助。