net/protocol.rb分配的大量內存：153

運行memory_profiler時，我注意到大量的內存被Ruby的net/protocol.rb組件分配。我在執行HTTP請求到服務器以下載文件時調用它。該文件大小爲43.67MB，net/protocol.rb單獨分配262,011,476字節來下載它。net/protocol.rb分配的大量內存：153

查看下面的分析器報告中的「按位置分配的內存」部分，我可以看到net/protocol.rb:172和http/response.rb:334每個分配50-60MB的內存，這大約與文件的大小有關，因此看起來合理。但是，最上面的條目（net/protocol.rb:153）讓我擔心：這是200MB的內存，至少是文件大小的4倍。

我有兩個問題：

爲什麼網/協議需要分配5X文件的大小，以下載呢？
有什麼我可以做的，以減少淨/協議使用的內存量？

memory_profiler輸出：

Total allocated: 314461424 bytes (82260 objects) 
Total retained: 0 bytes (0 objects) 

allocated memory by gem 
----------------------------------- 
314461304 ruby-2.1.2/lib 
     120 client/lib 

allocated memory by file 
----------------------------------- 
262011476 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb 
    52435727 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb 
     7971 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb 
     2178 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb 
     1663 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb 
     1260 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb 
     949 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb 
     120 /Users/andre.debrito/git/techserv-cache/client/lib/connections/cache_server_connection.rb 
     80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/http.rb 

allocated memory by location 
----------------------------------- 
200483909 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 
    60548199 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:172 
    52428839 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:334 
    978800 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:155 
     2537 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:61 
     2365 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:172 
     2190 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:54 
     1280 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:56 
     960 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:62 
     836 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:165 
     792 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:13 
     738 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:125 
     698 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:263 
     576 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:214 
     489 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:40 
     480 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:127 
     360 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:40 
     328 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:610 
     320 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:71 
     320 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:30 
     320 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:59 
     308 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:322 
     256 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:879 
     240 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1615 
     239 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:211 
     232 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:38 
     224 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:181 
     200 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:17 
     192 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:42 
     179 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:877 
     169 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1459 
     160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1029 
     160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:434 
     160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:435 
     160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:445 
     160 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1617 
     149 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1445 
     147 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1529 
     129 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:98 
     128 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1475 
     120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:444 
     120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:446 
     120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:447 
     120 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:29 
     120 /Users/andre.debrito/git/techserv-cache/client/lib/connections/cache_server_connection.rb:45 
     96 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:899 
     80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:39 
     80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:45 
     80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:46 
     80 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:145 

allocated memory by class 
----------------------------------- 
309678360 String 
    3445304 Thread::Backtrace 
    981096 Array 
    352376 IO::EAGAINWaitReadable 
     1960 MatchData 
     1024 Hash 
     328 Net::HTTP 
     256 TCPSocket 
     256 URI::HTTP 
     128 Time 
     120 Net::HTTP::Get 
     120 Net::HTTPOK 
     96 Net::BufferedIO 

allocated objects by gem 
----------------------------------- 
    82259 ruby-2.1.2/lib 
     1 client/lib 

allocated objects by file 
----------------------------------- 
    81908 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb 
     129 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb 
     127 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb 
     28 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb 
     23 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb 
     23 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb 
     19 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/http.rb 
     1 /Users/andre.debrito/git/techserv-cache/client/lib/connections/cache_server_connection.rb 

allocated objects by location 
----------------------------------- 
    36373 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 
    24470 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:155 
    21057 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:172 
     48 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:61 
     38 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:54 
     32 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:56 
     31 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:172 
     24 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:62 
     12 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:127 
     9 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:40 
     8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:71 
     8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:165 
     8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:30 
     8 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:59 
     6 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:214 
     6 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1615 
     5 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:263 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1029 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:322 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:17 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:434 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:435 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:445 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:42 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:125 
     4 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1617 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:1529 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:444 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:446 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:447 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:40 
     3 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1445 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http.rb:877 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:39 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:45 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/generic_request.rb:46 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:13 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:145 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/header.rb:31 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:111 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:144 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:98 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:179 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:181 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:213 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1640 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:1642 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:343 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:530 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/generic.rb:557 

allocated objects by class 
----------------------------------- 
    47935 String 
    24519 Array 
     4894 IO::EAGAINWaitReadable 
     4894 Thread::Backtrace 
     7 MatchData 
     3 Hash 
     2 URI::HTTP 
     1 Net::BufferedIO 
     1 Net::HTTP 
     1 Net::HTTP::Get 
     1 Net::HTTPOK 
     1 TCPSocket 
     1 Time 

retained memory by gem 
----------------------------------- 
NO DATA 

retained memory by file 
----------------------------------- 
NO DATA 

retained memory by location 
----------------------------------- 
NO DATA 

retained memory by class 
----------------------------------- 
NO DATA 

retained objects by gem 
----------------------------------- 
NO DATA 

retained objects by file 
----------------------------------- 
NO DATA 

retained objects by location 
----------------------------------- 
NO DATA 

retained objects by class 
----------------------------------- 
NO DATA 


Allocated String Report 
----------------------------------- 
    11926 "" 
     7019 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:172 
     4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 
     10 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/http/response.rb:54 
     2 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/uri/common.rb:179 
     1 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:67 

     4894 "Resource temporarily unavailable - read would block" 
     4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 

     4894 "UTF-8" 
     4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 

     4894 "read would block" 
     4894 /Users/andre.debrito/.rvm/rubies/ruby-2.1.2/lib/ruby/2.1.0/net/protocol.rb:153 
...

相關的代碼：

report = MemoryProfiler.report do 
    begin 
     response = nil 
     Net::HTTP.new(uri.host, uri.port).start { |http| 
     request = Net::HTTP::Get.new uri.request_uri 
     response = http.request request 
     } 
    rescue Net::ReadTimeout => e 
     raise RequestTimeoutError.new(e.message) 
    rescue Exception => e 
     raise ServerConnectionError.new(e.message) 
    end 
    end 
    report.pretty_print

自Charles代理

網絡流量數據：

請求報頭：168個字節
響應頭：288個字節
請求： -
響應：43.67 MB（45792735個字節）
總計：43.67 MB（45793191個字節）

來源

2017-08-29 André de Brito

幾乎所有net/protocol.rb#L153分配這些字符串是短期住下來並在下一次GC運行中回收。這些分配的對象因此非常無害，並且不會導致顯着更大的進程大小。

您會得到很多異常（用於此處的控制流以讀取套接字）以及附加到緩衝區的實際讀取數據。所有這些操作都會創建臨時（內部使用）的對象。

因此，您可能正在測量錯誤的東西。可能更有意義的是：

測量進程的最大RSS（即「已用」內存）;
並測量讀取後仍分配的額外內存量。

您會注意到（取決於計算機上的內存壓力），RSS不會顯着高於實際讀取的數據量，並且讀取後的參考內存與讀取的大小大致相同大約沒有內部中間對象的數據仍被引用。

來源

2017-08-29 16:47:12

感謝您的回答。你是正確的：在代碼運行後測量正在使用的內存，我看它穩定在77。5MB，所以Net /協議分配的所有內存都會非常快速地得到GC。 –

net/protocol.rb分配的大量內存：153

回答

相關問題