2015-11-04 20 views
6

我遇到大卷曲調用問題。捲曲:傳輸關閉,剩餘的未完成的讀取數據

我得到

  • NREAD < = 0,服務器關閉連接,汲水
  • 傳遞以優異的讀出的數據剩餘

和內容被局部遞送

關閉
GET /stats/?stats_breakdown=track__track&campaign=&search_criteria=2&period=0&date_month=11&date_day=03&date_year=2015&start_date_month=11&start_date_day=03&start_date_year=2015&end_date_month=12&end_date_day=31&end_date_year=2014 HTTP/1.1 
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13 
Host: domain.com 
Accept: */* 
Cookie: sessionid=xxg4gglsm7o3b224wihqz8od19wl31h1; csrftoken=JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C 
Cache-Control: no-cache 
Connection: Keep-Alive 
Keep-Alive: 600 
Accept-Language: en-us 
X-CSRFToken: SeN9bHryRK8FWLTLJIs5c6u9AZ47a8pR 
Content-Type: application/x-www-form-urlencoded 
Origin: https://domain.com 
Referer: https://domain.com 

* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* additional stuff not fine transfer.c:1037: 0 0 
* HTTP 1.1 or later with persistent connection, pipelining supported 
< HTTP/1.1 200 OK 
< Server: nginx/1.8.0 
< Date: Wed, 04 Nov 2015 12:54:05 GMT 
< Content-Type: text/html; charset=utf-8 
< Transfer-Encoding: chunked 
< Connection: keep-alive 
< Vary: Accept-Encoding 
< Vary: Cookie, Accept-Language 
< P3P: CP="ALL IND DSP COR ADM CONo CUR CUSo IVAo IVDo PSA PSD TAI TELo OUR SAMo CNT COM INT NAV ONL PHY PRE PUR UNI" 
< Content-Language: en 
* Replaced cookie csrftoken="JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C" for domain domain.com, path /, expire 1478091245 
< Set-Cookie: csrftoken=JBpLxNtgAVvDEw2wNqvBnRmzDJIjxL6C; expires=Wed, 02-Nov-2016 12:54:05 GMT; Max-Age=31449600; Path=/; secure 
< 
* nread <= 0, server closed connection, bailing 
* transfer closed with outstanding read data remaining 
* Closing connection #0 

這是PHP配置我用

function getHeaders() 
{ 
    $headers = array(); 
    $headers[] = 'Cache-Control: no-cache'; 
    $headers[] = 'Connection: Keep-Alive'; 
    $headers[] = 'Keep-Alive: 600'; 
    $headers[] = 'Accept-Language: en-us'; 
    $headers[] = 'X-CSRFToken: SeN9bHryRK8FWLTLJIs5c6u9AZ47a8pR'; 

    $headers[] = 'Content-Type: application/x-www-form-urlencoded'; 
    $headers[] = 'Origin: https://domain.com'; 
    $headers[] = 'Referer: https://domain.com'; 

    return $headers; 

} 

curl_setopt($connection, CURLOPT_URL, $url); 
    curl_setopt($connection, CURLOPT_RETURNTRANSFER, 1); 
    curl_setopt($connection, CURLOPT_FOLLOWLOCATION, true); 
    curl_setopt($connection, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13'); 
    curl_setopt($connection, CURLOPT_SSL_VERIFYPEER, false); 
    curl_setopt($connection, CURLOPT_SSL_VERIFYHOST, false); 
    curl_setopt($connection, CURLOPT_COOKIEFILE, 'cookie.txt'); 
    curl_setopt($connection, CURLOPT_COOKIEJAR, 'cookie.txt'); 
    curl_setopt($connection, CURLOPT_CONNECTTIMEOUT ,550000000); 
    curl_setopt($connection, CURLOPT_TIMEOUT, 5500000000); //timeout in seconds 
    curl_setopt($connection, CURLOPT_HTTPHEADER, getHeaders()); 
    curl_setopt($connection, CURLOPT_VERBOSE, 1); 
+0

你是否碰到負載平衡器或者像miracloud這樣的DDOS保護的服務器?他們可能會出於安全原因關閉長連接,即使您有數據流 – Bolovsky

+0

我無法控制遠程服務器。但它加載完美鉻.. –

回答

5

好的,經過一些搜索和IRC聊天之後,我發現瞭解決方案,但並非100%確定原因是什麼。看起來保持活力不足以保持連接。將在這裏發佈解決方案,希望我可以幫助別人。

什麼幫助對我來說是增加

--keepalive-time 2 

捲曲選項的說明

--keepalive-time <seconds> 

此選項設置的連接需要發送存活探測 之前保持空閒的時間,個人保活 探針之間的時間。它目前在提供TCP_KEEPIDLE和TCP_KEEPINTVL套接字選項(即Linux,最近的 AIX,HP-UX和更多)的操作系統上有效。如果使用--no-keepalive爲 ,則此選項無效。 (在7.18.0中增加)

如果多次使用此選項,將使用最後一個。如果未指定 ,則該選項默認爲60秒。

看起來像默認值太高,不能保持我的連接打開。

下面是完整的命令我用我的電話

curl URL -H 'Accept-Encoding: gzip, deflate, sdch' -H 'Accept-Language: en-US,en;q=0.8,et;q=0.6,nl;q=0.4' -H 'Upgrade-Insecure-Requests: 1' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/45.0.2454.101 Safari/537.36' -H 'Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8' -H 'Connection: keep-alive' --compressed -v --keepalive-time 2 

而且我在OSX

curl 7.43.0 (x86_64-apple-darwin15.0) libcurl/7.43.0 SecureTransport zlib/1.2.5 
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS IPv6 Largefile GSS-API Kerberos SPNEGO NTLM NTLM_WB SSL libz UnixSockets 

運行此版本捲曲如果有人想在PHP中使用此選項curl,--keepalive-time選項自PHP 5.5起可用。你可以使用它如下:

curl_setopt($connection, CURLOPT_TCP_KEEPALIVE, 1); 
curl_setopt($connection, CURLOPT_TCP_KEEPIDLE, 2); 

希望這可以幫助有人在同一個問題掙扎!

+0

如何添加'CURLOPT_TCP_KEEPALIVE'&'CURLOPT_TCP_KEEPIDLE'在PHP'Guzzle/HttpClient'? – 2016-04-29 18:15:49

+0

@ user6067756您只需將它們作爲捲髮設置傳遞:https://justrocketscience.com/post/guzzle-transfer-closed-n-bytes-remaining-read – NaN

1

的libcurl只告訴你,連接被服務器在不潔淨的方式它並沒有提供它曾答應做數據削減。看起來分塊編碼並不表示傳輸結束。

瀏覽器因爲在接收到的內容時非常自由而臭名昭着,所以他們忽略了這些違反協議的行爲,並且違反了libcurl的規定。

1

遇到類似的問題,我的服務器在nginx後面。捲曲能夠,如果它直接連接到服務器,以接收響應,但是如果捲曲連接到經由nginx的服務器,捲曲投擲下面誤差

「會話*傳遞以優異的讀出的數據的剩餘*截止連接0捲曲關閉:(18 )轉讓封閉以優異的讀取數據剩餘

當我連接到使用瀏覽器相同的nginx的URL,響應是顯示了罰款,這是很奇怪的,當我試圖使用curl連接到同一個nginx的URL,它被扔以上錯誤

比較瀏覽器和curl發送的報頭後發現,瀏覽器能夠接收因爲低於報頭的響應,其捲曲不發送

'Accept-Encoding: gzip' 

使用捲曲以上標頭中發送,它工作正常。所以,上面的頭文件正在壓縮對gzip的響應,從而減少響應大小。

經過多次挖掘,發現nginx無法發送> 80kb的任何有效載荷。浪費大量的時間後,發現問題是與nginx的緩衝和nginx的在nginx.conf添加以下proxy_buffering財產

location/{ 
proxy_buffering off; 
} 

接受的答案並沒有解決我的問題後,工作就像魅力。寫下這個答案,以便任何人如果面臨與我一樣的問題,就不必浪費時間。