使用curl登錄後得到動態生成的內容

Question

我知道curl不執行javascript，它只抓取靜態html，所以這就是爲什麼簡單的curl不適合我的原因。 我對PHP知之甚少，我對此很陌生，但我的理解是，如果我不必先登錄以獲取內容，我可以簡單地使用file_get_contents女巫將首先執行動態內容，然後抓住html內容，女巫回報給我我需要的，但我首先必須登錄，然後獲得頁面。 我試着用捲曲

$user = "myuser"; 
$pass = "mypassword"; 

//create cookie file 
$random = rand(0,9999999); 
$cookie = $random."cookie.txt"; 
$fp = fopen("$cookie","w") or die("<BR><B>Unable to open cookie file $cookie_file_path for write!<BR>"); 
fclose($fp); 

//do login using curl 
$LOGINURL = "https://controlpanel.example.com/index.html"; 
$agent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0"; 
$v2 = array('userName'=>$user, 'password'=>$pass); 
$reffer = "https://www.google.com"; 
//this first call is to set the cookie 
$ch = curl_init(); 
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie); 
    curl_setopt($ch, CURLOPT_URL,$LOGINURL); 
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); 
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); 
    curl_setopt($ch, CURLOPT_USERAGENT, $agent); 
ob_start();  // Prevent output 
curl_exec ($ch); 
ob_end_clean(); // End preventing output 
curl_close ($ch); 
unset($ch); 
//now that the cookie is set, do login 
$ch = curl_init(); 
    curl_setopt($ch, CURLOPT_POST, true); 
    curl_setopt($ch, CURLOPT_POSTFIELDS,$v2); 
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie); 
    curl_setopt($ch, CURLOPT_URL,$LOGINURL); 
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); 
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); 
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); 
    curl_setopt($ch, CURLOPT_REFERER, $reffer); 
    curl_setopt($ch, CURLOPT_USERAGENT, $agent); 

$result = curl_exec($ch); 

//now we are logged-in 
//now grab the page you need 

$profileurl = 'https://controlpanel.example.com/information.html'; 
curl_setopt($ch, CURLOPT_URL, $profileurl); 
curl_setopt($ch, CURLOPT_POST, 0); 
curl_setopt($ch, CURLOPT_TIMEOUT, 5); 

$result = curl_exec ($ch); 

登錄但這隻會得到靜態html，而不是動態的內容了。 讓我更好地解釋。 代碼我得到的，在這一點上使用上述捲曲的方法，在$結果是：

..... 
<div id="DisplayAccountInfo"><span class="loading">Loading info</span></div> 
..... 

如果我做到這一點使用Firefox和Firebug的檢查元素手工來源是：

..... 
<div id="DisplayAccountInfo"> 
    <div class="formModule" id="formContainer"> 
    ...... 
     <legend>Your code for this hour is 8T5D9LO</legend> 
    ..... 
    </div> 
</div> 
..... 

什麼我在Firebug控制檯注意的是：

GET https://controlpanel.example.com/async/information.html 

200 OK 
     669ms 
jquery-....min.js (line 19) 

我，作爲一個菜鳥，從這個不解的是，內容是使用jQuery dinamicly加載，和捲曲不知道該怎麼做。

我試圖把代替

$profileurl = 'https://controlpanel.example.com/information.html'; 
curl_setopt($ch, CURLOPT_URL, $profileurl); 
curl_setopt($ch, CURLOPT_POST, 0); 
curl_setopt($ch, CURLOPT_TIMEOUT, 5); 

$result = curl_exec ($ch); 

//replaced the above with this 
$result = file_get_contents($profileurl); 

，但我從登錄頁面的HTML，因爲我覺得現在不承認，我登錄。

所以，我怎麼能解決這個問題？你能幫我麼？

Answer 1

哈哈，這麼容易就沒有過我的腦海。 對我來說很簡單，我沒有打電話給

https://controlpanel.example.com/information.html

但

https://controlpanel.example.com/async/information.html

得到我想要的:)

對我來說幸運的股利我注意到在螢火蟲中獲取功能:)

所以鱈魚現在是：

$user = "myuser"; 
$pass = "mypassword"; 

//create cookie file 
$random = rand(0,9999999); 
$cookie = $random."cookie.txt"; 
$fp = fopen("$cookie","w") or die("<BR><B>Unable to open cookie file $cookie for write!<BR>"); 
fclose($fp); 

//do login using curl 
$LOGINURL = "https://controlpanel.example.com/index.html"; 
$agent = "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20120101 Firefox/29.0"; 
$v2 = array('userName'=>$user, 'password'=>$pass); 
$reffer = "https://www.google.com"; 
//this first call is to set the cookie 

$ch = curl_init(); 
    curl_setopt($ch, CURLOPT_COOKIEJAR, $cookie); 
    curl_setopt($ch, CURLOPT_URL,$LOGINURL); 
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); 
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); 
    curl_setopt($ch, CURLOPT_USERAGENT, $agent); 
ob_start();  // Prevent output 
curl_exec ($ch); 
ob_end_clean(); // End preventing output 
curl_close ($ch); 
unset($ch); 

//now that the cookie is set, do login 
$ch = curl_init(); 
    curl_setopt($ch, CURLOPT_POST, true); 
    curl_setopt($ch, CURLOPT_POSTFIELDS,$v2); 
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, true); 
    curl_setopt($ch, CURLOPT_COOKIEFILE, $cookie); 
    curl_setopt($ch, CURLOPT_URL,$LOGINURL); 
    curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false); 
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false); 
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); 
    curl_setopt($ch, CURLOPT_REFERER, $reffer); 
    curl_setopt($ch, CURLOPT_USERAGENT, $agent); 

$result = curl_exec($ch); 

//now we are logged-in 
//now grab the page you need 

$profileurl = 'https://controlpanel.example.com/async/information.html'; 
curl_setopt($ch, CURLOPT_URL, $profileurl); 
curl_setopt($ch, CURLOPT_POST, 0); 

$result = curl_exec ($ch); 

Answer 2

我想我明白了你在做什麼。

這裏的關鍵是，大多數網站使用cookie處理登錄。在https://controlpanel.example.com/information.html中，如果網站在您的瀏覽器中登錄後設置了cookie，那麼好消息是您可以解決此問題。

你的代碼中的問題是，PHP不會爲你設置cookie。

您需要2步：您需要獲得cookie時你的PHP捲曲登錄

這裏是你如何從登錄頁面返回的cookie頭

第1步。

$ch = curl_init('https://controlpanel.example.com/index.html'); 

.... 

$result = curl_exec($ch); 
preg_match('/^Set-Cookie:\s*([^;]*)/mi', $result, $m); 
parse_str($m[1], $cookies); 
echo $cookies;//See if you've successfully obtained the return cookie 

第2步：您訪問https://controlpanel.example.com/information.html與您在步驟1中獲得的餅乾（像你已經在自己的代碼一樣）