PowerShell的FTP發送大文件的System.OutOfMemoryException

Question

我有一個腳本部分地基於一個位置：Upload files with FTP using PowerShell

它所有的工作與小文件，絕對沒問題，但我想用它來做我們用於導出過程訪問mdb文件到只有ftp更健壯的客戶端。

我的第一個測試包括一個10MB的文件，我碰到了在PowerShell ISE是在獲取嘗試的過程中運行到近2GIG使用的獲取內容階段

一個System.OutOfMemoryException。

這是一個完整的腳本示例（溫柔我是相當新的吧。）：

##### 
# User variables to control the script 
##### 

# How many times connection will be re-tried 
$connectionTries = 5 
#time between tries in seconds 
$connectionTryInterval = 300 
#Where to log the output 
$logFile = "D:\MyPath\ftplog.txt" 
#maximum log file size in KB before it is archived 
$logFileMaxSize = 500 

#formatted date part for the specific file to transfer 
#This is appended to the filename base. Leave as "" for none 
$datePart = "" 
#base part of the file name 
$fileNameBase = "Myfile" 
#file extension 
$fileExtension = ".mdb" 
#location of the source file (please include trailing backslash) 
$sourceLocation = "D:\MyPath\" 

#location and credentials of the target ftp server  
$userName = "iamafish" 
$password = "ihavenofingers" 
$ftpServer = "10.0.1.100" 

###### 
# Main Script 
##### 

#If there is a log file and it is longer than the declared limit then archive it with the current timestamp 
if (test-path $logfile) 
{ 
    if($((get-item $logFile).Length/1kb) -gt $logFileMaxSize) 
    { 
     write-host $("archiving log to ftplog_" + (get-date -format yyyyMMddhhmmss) +  ".txt") 
     rename-item $logFile $("ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt") 
    } 
} 

#start new log entry 
#Add-Content $logFile "___________________________________________________________" 
#write-host $logEntry 

#contruct source file and destination uri 
$fileName = $fileNameBase + $datePart + $fileExtension 
$sourceFile = $sourceLocation + $fileName 
$sourceuri = "ftp://" + $ftpServer + "/" + $fileName 


# Create a FTPWebRequest object to handle the connection to the ftp server 
$ftprequest = [System.Net.FtpWebRequest]::create($sourceuri) 

# set the request's network credentials for an authenticated connection 
$ftprequest.Credentials = New-Object System.Net.NetworkCredential($username,$password) 

$ftprequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile 
$ftprequest.UseBinary = $true 
$ftprequest.KeepAlive = $false 

$succeeded = $true 
$errorMessage = "" 

# read in the file to upload as a byte array 
trap [exception]{ 
    $script:succeeded = $false 
    $script:errorMessage = $_.Exception.Message 
    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|" + $_.Exception.Message) 
    #write-host $logEntry 
    #write-host $("TRAPPED: " + $_.Exception.GetType().FullName) 
    #write-host $("TRAPPED: " + $_.Exception.Message) 
    exit 
} 
#The -ea 1 forces the error to be trappable 
$content = gc -en byte $sourceFile -ea 1 


$try = 0 

do{ 
    trap [System.Net.WebException]{ 
     $script:succeeded = $false 
     $script:errorMessage = $_.Exception.Message 
     Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|" + $_.Exception.Message) 
     #write-host $logEntry 
     #write-host $("TRAPPED: " + $_.Exception.GetType().FullName) 
     $script:try++ 
     start-sleep -s $connectionTryInterval 
     continue 
     } 
     $ftpresponse = $ftprequest.GetResponse() 

} while(($try -le $connectionTries) -and (-not $succeeded)) 

if ($succeeded) { 

    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|0|" + "Starting file transfer.") 
    # get the request stream, and write the bytes into it 
    $rs = $ftprequest.GetRequestStream() 
    $rs.Write($content, 0, $content.Length) 
    # be sure to clean up after ourselves 
    $rs.Close() 
    $rs.Dispose() 
    $content.Close() 
    $content.Dispose() 
    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|0|" + "Transfer complete.") 
    #write-host $logEntry 
} 

我不能把代碼中的註釋是這樣，由於從基思指針我已經移動的文件存取權限位降到谷底將其與其他像這樣鏈接..

trap [Exception]{ 
    $script:succeeded = $false 
    $script:errorMessage = $_.Exception.Message 
    Add-Content $logFile $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|Check File Connection|" + $_.Exception.Message) 
    $sourceStream.Close() 
    $sourceStream.Dispose() 
    #write-host $((get-Date -format "yyyy-MM-dd hh:mm:ss") + "|1|Attempt to open file|" + $_.Exception.Message) 
    #write-host $("TRAPPED: " + $_.Exception.GetType().FullName) 
    exit 
} 
$sourceStream = New-Object IO.FileStream ($(New-Object System.IO.FileInfo $sourceFile),[IO.FileMode]::Open) 
[byte[]]$readbuffer = New-Object byte[] 1024 

# get the request stream, and write the bytes into it 
$rs = $ftprequest.GetRequestStream() 
do{ 
    $readlength = $sourceStream.Read($readbuffer,0,1024) 
    $rs.Write($readbuffer,0,$readlength) 
} while ($readlength -ne 0) 

我只需要明白爲什麼我得到：異常調用「的GetResponse」和「0」的說法（S）：「無法訪問一個處置對象。 每隔一個當我運行它。這是在ISE中運行它的一個怪癖嗎，還是我在做初始聲明或最終處置時做了一些嚴重錯誤？

完成後我會發布完整的最終腳本，因爲我認爲它會使錯誤捕獲和日誌記錄成爲一個非常堅固的ftp導出示例。

---

好的，這裏是完整的腳本。 Dispose被編輯出來，但有或沒有它在5分鐘內運行該腳本將要麼給我一個消息，我不能使用一個處理對象或告訴我，getResponse（）產生了一個錯誤（226）文件傳輸（在ISE中運行） 。雖然在正常的操作中這不會成爲問題，但我想正確地登錄FTP會話並清理腳本末尾的資源，並確保根據需要正確聲明它們。

的試探末轉讓OK響應

##### 
# User variables to control the script 
##### 

# How many times connection will be re-tried 
$connectionTries = 5 
#time between tries in seconds 
$connectionTryInterval = 1 
#Where to log the output 
$logFile = "D:\MyPath\ftplog.txt" 
#maximum log file size in KB before it is archived 
$logFileMaxSize = 500 
#log to file or console - #true=log to file, #false = log to console 
$logToFile=$false 

#formatted date part for the specific file to transfer 
#This is appended to the filename base. Leave as "" for none 
$datePart = "" 
#base part of the file name 
$fileNameBase = "MyFile" 
#file extension 
$fileExtension = ".mdb" 
#location of the source file (please include trailing backslash) 
$sourceLocation = "D:\MyPath\" 

#location and credentials of the target ftp server 
$userName = "iamafish" 
$password = "ihavenofingers" 
$ftpServer = "10.0.1.100" 

###### 
# Main Script 
##### 

function logEntry($entryType, $section, $message) 
{ 
    #just to make a one point switch for logging to console for testing 
    # $entryType: 0 = success, 1 = Error 
    # $section: The section of the script the log entry was generated from 
    # $message: the log message 

    #This is pipe separated to fit in with my standard MSSQL linked flat file schema for easy querying 
    $logString = "$(get-Date -format "yyyy-MM-dd hh:mm:ss")|$entryType|$section|$message" 

    if($script:logtoFile) 
    { 
     Add-Content $logFile $logString 
    } 
    else 
    { 
     write-host $logString 
    } 
} 

#If there is a log file and it is longer than the declared limit then archive it with the current timestamp 
if (test-path $logfile) 
{ 
    if($((get-item $logFile).Length/1kb) -gt $logFileMaxSize) 
    { 
     write-host $("archiving log to ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt") 
     rename-item $logFile $("ftplog_" + (get-date -format yyyyMMddhhmmss) + ".txt") 
     New-Item $logFile -type file 
    } 
} 
else 
{ 
    New-Item $logFile -type file 
} 


#contruct source file and destination uri 
$fileName = $fileNameBase + $datePart + $fileExtension 
$sourceFile = $sourceLocation + $fileName 
$destination = "ftp://" + $ftpServer + "/" + $fileName 


#Check if the source file exists 
if ((test-path $sourceFile) -eq $false) 
{ 
    logEntry 1 "Check Source File" $("File not found: " + $sourceFile) 
    Exit 
} 


# Create a FTPWebRequest object to handle the connection to the ftp server 
$ftpRequest = [System.Net.FtpWebRequest]::create($destination) 

# set the request's network credentials for an authenticated connection 
$ftpRequest.Credentials = New-Object System.Net.NetworkCredential($username,$password) 
$ftpRequest.Method = [System.Net.WebRequestMethods+Ftp]::UploadFile 
$ftpRequest.UseBinary = $true 
$ftpRequest.KeepAlive = $false 

$succeeded = $true 
$try = 1 

do{ 
    trap [Exception]{ 
     $script:succeeded = $false 
     logEntry 1 "Check FTP Connection" $_.Exception.Message 
     $script:try++ 
     start-sleep -s $connectionTryInterval 
     continue 
     } 
     $ftpResponse = $ftpRequest.GetResponse() 

} while(($try -le $connectionTries) -and (-not $succeeded)) 

if ($succeeded) { 
    logEntry 0 "Connection to FTP" "Success" 


    # Open a filestream to the source file 
    trap [Exception]{ 
     logEntry 1 "Check File Connection" $_.Exception.Message 
     $sourceStream.Close() 
     $ftpResponse.Close() 
     exit 
    } 
    $sourceStream = New-Object IO.FileStream ($(New-Object System.IO.FileInfo $sourceFile),[IO.FileMode]::Open) 
    [byte[]]$readbuffer = New-Object byte[] 1024 

    logEntry 0 "Starting file transfer" "Success" 
    # get the request stream, and write the bytes into it 
    $rs = $ftpRequest.GetRequestStream() 
    do{ 
     $readlength = $sourceStream.Read($readbuffer,0,1024) 
     $rs.Write($readbuffer,0,$readlength) 
    } while ($readlength -ne 0) 

    logEntry 0 "Transfer complete" "Success" 
    # be sure to clean up after ourselves 
    $rs.Close() 
    #$rs.Dispose() 
    $sourceStream.Close() 
    #$sourceStream.Dispose() 

} 
$ftpResponse.Close() 

---

例子：

logEntry 0 "Starting file transfer" "Success" 
# get the request stream, and write the bytes into it 
$rs = $ftpRequest.GetRequestStream() 
do{ 
    $readlength = $sourceStream.Read($readbuffer,0,1024) 
    $rs.Write($readbuffer,0,$readlength) 
} while ($readlength -ne 0) 
$rs.Close() 
#start-sleep -s 2 

trap [Exception]{ 
    $script:succeeded = $false 
    logEntry 1 "Check FTP Connection" $_.Exception.Message 
    continue 
} 
$ftpResponse = $ftpRequest.GetResponse() 

Answer 1

而不是讀取整個文件到內存使用獲取內容，嘗試閱讀它一大塊一段時間並將其寫入FTP請求流。我會使用其中一個較低級別的.NET文件流API來讀取數據。無可否認，你不會認爲10MB會造成內存問題。

另外，確保在創建請求流並寫入之後獲得響應。獲取響應流是上傳數據。從文檔：

When using an FtpWebRequest object to upload a file to a server, you must write the file content to the request stream obtained by calling the GetRequestStream method or its asynchronous counterparts, the BeginGetRequestStream and EndGetRequestStream methods. You must write to the stream and close the stream before sending the request.

Requests are sent to the server by calling the GetResponse method or its asynchronous counterparts, the BeginGetResponse and EndGetResponse methods. When the requested operation completes, an FtpWebResponse object is returned. The FtpWebResponse object provides the status of the operation and any data downloaded from the server.

Answer 2

已經打了一個類似的問題我自己與RAM使用擊中GB的上傳3MB的文件，我發現更換：

$content = gc -en byte $sourceFile 

有了：

$content = [System.IO.File]::ReadAllBytes($sourceFile) 

給出性能更好。正如其他地方所提到的，分塊對於大文件來說是更好的解決方案，因爲這樣你就不會一次將整個文件保存在內存中，但上面的代碼至少只消耗〜（文件大小）字節的RAM，這意味着它應該好到MB類型的〜10s。