通過HTTP上傳BIG文件

Question

我試圖使用PowerShell將真正大的VM鏡像（5-15 Gb大小）上傳到HTTP服務器。

我試圖用爲一些方法（這裏鏈接到script with net.WebClient.UploadFile和script with Invoke-webRequest）

它非常適用的文件小於2GB，但不大於該文件。

我試圖直接使用httpWebRequest，但我無法將FileStream放入其中。

所以我的問題是：如何把文件流到webrequest？

或更一般地說：如何通過http上傳PowerShell的大文件？

$Timeout=10000000; 
$fileName = "0.iso"; 
$data = "C:\\$fileName"; 
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName"; 
#$buffer = [System.IO.File]::Open("$data",[System.IO.Filemode]::Open, [System.IO.FileAccess]::Read) #Err Cannot convert argument "buffer", with value: "System.IO.FileStream", for "Write" to type "System.Byte[]": 
#$buffer = gc -en byte $data # too much space in memory 
$buffer = [System.IO.File]::ReadAllBytes($data) #Limit 2gb 
[System.Net.HttpWebRequest] $webRequest = [System.Net.WebRequest]::Create($url) 
$webRequest.Timeout = $timeout 
$webRequest.Method = "POST" 
$webRequest.ContentType = "application/data" 
#$webRequest.ContentLength = $buffer.Length; 
$webRequest.Credentials = New-Object System.Net.NetworkCredential("admin", "admin123"); 

$requestStream = $webRequest.GetRequestStream() 
$requestStream.Write($buffer, 0, $buffer.Length) 
$requestStream.Flush() 
$requestStream.Close() 

[System.Net.HttpWebResponse] $webResponse = $webRequest.GetResponse() 
$streamReader = New-Object System.IO.StreamReader($webResponse.GetResponseStream()) 
$result = $streamReader.ReadToEnd() 
return $result 
$stream.Close() 

Answer 1

默認情況下，HttpWebRequest緩衝內存中的數據。 只需將HttpWebRequest.AllowWriteStreamBuffering屬性設置爲false，即可上傳幾乎任何大小的文件。 在msdn

Answer 2

查看更多詳細信息謝謝@Stoune，這是幫助最終解決問題的最後一件事。

另外，還需要組織流文件讀寫webRequest緩衝區。它可能做的是一段代碼：

$requestStream = $webRequest.GetRequestStream() 
$fileStream = [System.IO.File]::OpenRead($file) 
$chunk = New-Object byte[] $bufSize 
    while($bytesRead = $fileStream.Read($chunk,0,$bufsize)) 
    { 
    $requestStream.write($chunk, 0, $bytesRead) 
    $requestStream.Flush() 
    } 

和最終的腳本是這樣的：

$user = "admin" 
$pass = "admin123" 
$dir = "C:\Virtual Hard Disks" 
$fileName = "win2012r2std.vhdx" 
$file = "$dir/$fileName" 
$url = "http://nexus.lab.local:8081/nexus/content/sites/myproj/$fileName" 
$Timeout=10000000 
$bufSize=10000 

$cred = New-Object System.Net.NetworkCredential($user, $pass) 

$webRequest = [System.Net.HttpWebRequest]::Create($url) 
$webRequest.Timeout = $timeout 
$webRequest.Method = "POST" 
$webRequest.ContentType = "application/data" 
$webRequest.AllowWriteStreamBuffering=$false 
$webRequest.SendChunked=$true # needed by previous line 
$webRequest.Credentials = $cred 

$requestStream = $webRequest.GetRequestStream() 
$fileStream = [System.IO.File]::OpenRead($file) 
$chunk = New-Object byte[] $bufSize 
    while($bytesRead = $fileStream.Read($chunk,0,$bufsize)) 
    { 
    $requestStream.write($chunk, 0, $bytesRead) 
    $requestStream.Flush() 
    } 

$responceStream = $webRequest.getresponse() 
#$status = $webRequest.statuscode 

$FileStream.Close() 
$requestStream.Close() 
$responceStream.Close() 

$responceStream 
$responceStream.GetResponseHeader("Content-Length") 
$responceStream.StatusCode 
#$status 

Answer 3

上傳到Sonatype的Nexus3我用下面的代碼。花了我一些時間弄清楚，使用Nexus3的上傳和響應以及上傳和下載大文件（大於2GB）。我們在Nexus3前面安裝了Apache，負責管理https連接。 使用基本身份驗證確保Nexus在Apache之前正確響應。 通過HEAD發送預認證並對大文件使用分塊上載修復上傳大文件不會提前結束。

通過Invoke-WebRequest下載大文件也會因不同的錯誤而失敗。現在我通過.Net使用WebRequest，它的工作原理見Download-File。

而當代碼通過自動化進程（從System Center Orchestrator）運行時，https會失敗。因此，當檢測到https方案時，我們強制使用TLS 1.2。

function New-HttpWebRequest 
{ 
    <# 
    .SYNOPSIS 
    Creates a new [System.Net.HttpWebRequest] ready for file transmission. 

    .DESCRIPTION 
    Creates a new [System.Net.HttpWebRequest] ready for file transmission. 
    The method will be Put. If the filesize is larger than the buffersize, 
    the HttpWebRequest will be configured for chunked transfer. 

    .PARAMETER Url 
    Url to connect to. 

    .PARAMETER Credential 
    Credential for authentication at the Url resource. 

    .EXAMPLE 
    An example 
    #> 
    param(
     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$Url, 

     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [pscredential]$Credential, 

     [Parameter(Mandatory=$true)] 
     [long]$FileSize, 

     [Parameter(Mandatory=$true)] 
     [long]$BufferSize 
    ) 

    $webRequest = [System.Net.HttpWebRequest]::Create($Url) 
    $webRequest.Timeout = 600 * 1000; 
    $webRequest.ReadWriteTimeout = 600 * 1000; 
    $webRequest.ProtocolVersion = [System.Net.HttpVersion]::Version11; 
    $webRequest.Method = "PUT"; 
    $webRequest.ContentType = "application/octet-stream"; 
    $webRequest.KeepAlive = $true; 
    $webRequest.UserAgent = "<I use a specific UserAgent>"; 
    #$webRequest.UserAgent = 'Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1)'; 
    $webRequest.PreAuthenticate = $true; 
    $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password)); 
    $webRequest.Headers["Authorization"] = "Basic $auth" 

    if (Get-UseChunkedUpload -FileSize $FileSize -BufferSize $BufferSize) 
    { 
     Write-Verbose "FileSize is greater than BufferSize, using chunked transfer."; 
     $webRequest.AllowWriteStreamBuffering = $false; 
     $webRequest.SendChunked = $true; 
    } 
    else 
    { 
     # Filesize is equal to or smaller than the BufferSize. The file will be transferred in one write. 
     # Chunked cannot be used in this case. 
     $webRequest.AllowWriteStreamBuffering = $true; 
     $webRequest.SendChunked = $false; 
     $webRequest.ContentLength = $FileSize; 
    } 

    return $webRequest; 
} 

function Get-BufferSize 
{ 
    <# 
    .SYNOPSIS 
    Returns a buffer size that is 1% of ByteLength, rounded in whole MB's or at least AtLeast size. 

    .DESCRIPTION 
    Returns a buffer size that is 1% of ByteLength, rounded to whole MB's or if 1% is smaller than AtLeast, then AtLeast size is returned which is 1MB by default. 

    .PARAMETER ByteLength 
    Length of the bytes for which to calculate a valid buffer size. 

    .PARAMETER AtLeast 
    The minimum required buffer size, default 1MB. 

    .EXAMPLE 
    Get-BufferSize 4283304773 

    Returns 42991616 which is 41MB. 

    .EXAMPLE 
    Get-BufferSize 4283304 

    Returns 1048576 which is 1MB. 

    .EXAMPLE 
    Get-BufferSize 4283304 5MB 

    Returns 5242880 which is 5MB. 
    #> 
    param(
     [Parameter(Mandatory=$true)] 
     [long]$ByteLength, 

     [long]$AtLeast = 1MB 
    ) 

    [long]$size = $ByteLength/100; 
    if ($size -lt $AtLeast) 
    { 
     $size = $AtLeast; 
    } 
    else 
    { 
     $size = [Math]::Round($size/1MB) * 1MB; 
    } 

    return $size; 
} 

function Get-UseChunkedUpload 
{ 
    param(
     [Parameter(Mandatory=$true)] 
     [long]$FileSize, 

     [Parameter(Mandatory=$true)] 
     [long]$BufferSize 
    ) 

    return $FileSize -gt $BufferSize; 
} 

function Configure-Tls 
{ 
    param(
     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$Url 
    ) 

    [System.Uri]$uri = $Url; 
    if ($uri.Scheme -eq "https") 
    { 
     Write-Verbose "Using TLS 1.2"; 
     [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12; 
    } 
} 

function Send-PreAuthenticate 
{ 
    param(
     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$Url, 

     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [pscredential]$Credential 
    ) 

    $response = $null; 
    try 
    { 
     [System.Uri]$uri = $Url; 
     $repositoryAuthority = (($uri.GetLeftPart([System.UriPartial]::Authority)).TrimEnd('/') + '/'); 
     Write-Verbose "Send-PreAuthenticate - Sending HEAD to $repositoryAuthority"; 
     $wr = [System.Net.WebRequest]::Create($repositoryAuthority); 
     $wr.Method = "HEAD"; 
     $wr.PreAuthenticate = $true; 
     $auth = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes($Credential.UserName + ":" + $Credential.GetNetworkCredential().Password)); 
     $wr.Headers["Authorization"] = "Basic $auth" 
     $response = $wr.GetResponse(); 
    } 
    finally 
    { 
     if ($response) 
     { 
      $response.Close(); 
      $response.Dispose(); 
      $response = $null; 
     } 
    } 
} 

function Upload-File 
{ 
    <# 
    .SYNOPSIS 
    Uploads a file to the Nexus repository. 

    .DESCRIPTION 
    Uploads a file to the Nexus repository. 
    If the file was uploaded successfully, the url via which the resource can be downloaded is returned. 

    .PARAMETER Url 
    The Url where the resource should be created. 
    Please note that underscores and dots should be encoded, otherwise the Nexus repository does not accept the upload. 

    .PARAMETER File 
    The file that should be uploaded. 

    .PARAMETER Credential 
    Credential used for authentication at the Nexus repository. 

    .EXAMPLE 
    Upload-File -Url https://nexusrepo.domain.com/repository/repo-name/myfolder/myfile%2Eexe -File (Get-ChildItem .\myfile.exe) -Credential (Get-Credential) 

    .OUTPUTS 
    If the file was uploaded successfully, the url via which the resource can be downloaded. 
    #> 
    param(
     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$Url, 

     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [System.IO.FileInfo]$File, 

     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [pscredential]$Credential 
    ) 

    Write-Verbose "Upload-File Url:$Url" 

    Configure-Tls -Url $Url; 

    $fileSizeBytes = $File.Length; 
    #$bufSize = Get-BufferSize $fileSizeBytes; 
    $bufSize = 4 * 1MB; 
    Write-Verbose ("FileSize is {0} bytes ({1:N0}MB). BufferSize is {2} bytes ({3:N0}MB)" -f $fileSizeBytes,($fileSizeBytes/1MB),$bufSize,($bufSize/1MB)); 
    if (Get-UseChunkedUpload -FileSize $fileSizeBytes -BufferSize $bufSize) 
    { 
     Write-Verbose "Using chunked upload. Send pre-auth first."; 
     Send-PreAuthenticate -Url $Url -Credential $Credential; 
    } 

    $progressActivityMessage = ("Sending file {0} - {1} bytes" -f $File.Name, $File.Length); 
    $webRequest = New-HttpWebRequest -Url $Url -Credential $Credential -FileSize $fileSizeBytes -BufferSize $bufSize; 
    $chunk = New-Object byte[] $bufSize; 
    $bytesWritten = 0; 
    $fileStream = [System.IO.File]::OpenRead($File.FullName); 
    $requestStream = $WebRequest.GetRequestStream(); 
    try 
    { 
     while($bytesRead = $fileStream.Read($chunk,0,$bufSize)) 
     { 
      $requestStream.Write($chunk, 0, $bytesRead); 
      $requestStream.Flush(); 
      $bytesWritten += $bytesRead; 
      $progressStatusMessage = ("Sent {0} bytes - {1:N0}MB" -f $bytesWritten, ($bytesWritten/1MB)); 
      Write-Progress -Activity $progressActivityMessage -Status $progressStatusMessage -PercentComplete ($bytesWritten/$fileSizeBytes*100); 
     } 
    } 
    catch 
    { 
     throw; 
    } 
    finally 
    { 
     if ($fileStream) 
     { 
      $fileStream.Close(); 
     } 
     if ($requestStream) 
     { 
      $requestStream.Close(); 
      $requestStream.Dispose(); 
      $requestStream = $null; 
     } 
     Write-Progress -Activity $progressActivityMessage -Completed; 
    } 

    # Read the response. 
    $response = $null; 
    try 
    { 
     $response = $webRequest.GetResponse(); 
     Write-Verbose ("{0} responded with {1} at {2}" -f $response.Server,$response.StatusCode,$response.ResponseUri); 
     return $response.ResponseUri; 
    } 
    catch 
    { 
     if ($_.Exception.InnerException -and ($_.Exception.InnerException -like "*bad request*")) 
     { 
      throw ("ERROR: " + $_.Exception.InnerException.Message + " Possibly the file already exists or the content type of the file does not match the file extension. In that case, disable MIME type validation on the server.") 
     } 

     throw; 
    } 
    finally 
    { 
     if ($response) 
     { 
      $response.Close(); 
      $response.Dispose(); 
      $response = $null; 
     } 
     if ($webRequest) 
     { 
      $webRequest = $null; 
     } 
    } 
} 

function Download-File 
{ 
    param(
     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$Url, 

     [Parameter(Mandatory=$true)] 
     [ValidateNotNullOrEmpty()] 
     [string]$FileName 
    ) 

    $SDXDownloadType = @" 
    using System.IO; 
    using System.Net; 

    public class SDXDownload 
    { 
     static public void DownloadFile(string Uri, string Filename) 
     { 
      HttpWebRequest webRequest = (HttpWebRequest)WebRequest.Create(Uri); 
      webRequest.Method = "GET"; 
      using (HttpWebResponse myHttpWebResponse = (HttpWebResponse)webRequest.GetResponse()) 
      using (Stream fileStream = File.OpenWrite(Filename)) 
      using (Stream streamResponse = myHttpWebResponse.GetResponseStream()) 
      { 
       int bufSize = 64 * 1024; 
       byte[] readBuff = new byte[bufSize]; 
       int bytesRead = streamResponse.Read(readBuff, 0, bufSize); 
       while (bytesRead > 0) 
       { 
        fileStream.Write(readBuff, 0, bytesRead); 
        bytesRead = streamResponse.Read(readBuff, 0, 256); 
       } 
      } 
     } 
    } 
"@ 

    Configure-Tls -Url $Url; 

    Add-Type -TypeDefinition $SDXDownloadType; 
    [SDXDownload]::DownloadFile($Url, $FileName); 
}