同時HTTP請求與捲曲

Question

我試圖採取域的一個相當大的列表查詢每個軍銜使用compete.com API這裏看到的 - >https://www.compete.com/developer/documentation

我寫的劇本需要一個數據庫我填充的域名，併發起cURL請求，爭奪網站的排名。我很快意識到這是非常緩慢的，因爲每次請求都是一次發送一次。我做了一些搜索，並遇到了這個post->http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/，它解釋瞭如何用cURL在PHP中執行同時的HTTP請求。

不幸的是，腳本將採用25,000個域的數組並嘗試一次處理它們。我發現一批1000個工作很好。

任何想法如何發送1,000個查詢到conte.com，然後等待完成併發送下一個1000，直到數組爲空？這裏就是我，因此到目前爲止，幹活：

<?php 

//includes 
include('includes/mysql.php'); 
include('includes/config.php'); 

//get domains 
$result = mysql_query("SELECT * FROM $tableName"); 
while($row = mysql_fetch_array($result)) { 
    $competeRequests[] = "http://apps.compete.com/sites/" . $row['Domain'] . "/trended/rank/?apikey=xxx&start_date=201207&end_date=201208&jsonp="; 
} 

//first batch 
$curlRequest = multiRequest($competeRequests); 
$j = 0; 
foreach ($curlRequest as $json){ 
    $j++; 
    $json_output = json_decode($json, TRUE); 
    $rank = $json_output[data][trends][rank][0][value]; 

    if($rank) { 
     //Create mysql query 
     $query = "Update $tableName SET Rank = '$rank' WHERE ID = '$j'"; 

     //Execute the query 
     mysql_query($query); 
     echo $query . "<br/>"; 
    } 
} 


function multiRequest($data) { 
    // array of curl handles 
    $curly = array(); 
    // data to be returned 
    $result = array(); 

    // multi handle 
    $mh = curl_multi_init(); 

    // loop through $data and create curl handles 
    // then add them to the multi-handle 
    foreach ($data as $id => $d) { 

    $curly[$id] = curl_init(); 

    $url = (is_array($d) && !empty($d['url'])) ? $d['url'] : $d; 
    curl_setopt($curly[$id], CURLOPT_URL,   $url); 
    curl_setopt($curly[$id], CURLOPT_HEADER,   0); 
    curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1); 

    // post? 
    if (is_array($d)) { 
     if (!empty($d['post'])) { 
     curl_setopt($curly[$id], CURLOPT_POST,  1); 
     curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']); 
     } 
    } 

    curl_multi_add_handle($mh, $curly[$id]); 
    } 

    // execute the handles 
    $running = null; 
    do { 
    curl_multi_exec($mh, $running); 
    } while($running > 0); 

    // get content and remove handles 
    foreach($curly as $id => $c) { 
    $result[$id] = curl_multi_getcontent($c); 
    curl_multi_remove_handle($mh, $c); 
    } 

    // all done 
    curl_multi_close($mh); 

    return $result; 

} 
?> 

Answer 1

而不是

//first batch 
$curlRequest = multiRequest($competeRequests); 

$j = 0; 
foreach ($curlRequest as $json){ 

你可以這樣做：

$curlRequest = array(); 

foreach (array_chunk($competeRequests, 1000) as $requests) { 
    $results = multiRequest($requests); 

    $curlRequest = array_merge($curlRequest, $results); 
} 

$j = 0; 
foreach ($curlRequest as $json){ 
    $j++; 
    // ... 

這大陣分成1000塊，將這1000個值傳遞給您的multiRequest函數，該函數使用cURL執行這些請求。