AWS CPP TransferManager vs GetObjectRequest流到文件fstream OOM

我正在使用AWS CPP SDK（https://github.com/aws/aws-iot-device-sdk-cpp）在小型Linux系統（僅32 MB RAM）上從S3下載文件。我正在使用GetObjectRequest類，如下所示。它運行良好，並將文件下載到我的系統上的FStream，以便它不會消耗太多RAM。AWS CPP TransferManager vs GetObjectRequest流到文件fstream OOM

現在，我想將下載代碼轉換爲TransferManager方法以獲取進度回調。我已經重寫了那部分代碼，它也顯示在下面。它開始很好，打印下載的百分比，但是當它達到〜14 MB RAM（大概是下載時在Linux中可用的數量）時，由於使用太多RAM而被內核殺死。

我創建了一個文件流，就像我爲GetObjectRequest所做的一樣。我究竟做錯了什麼？我怎樣才能解決這個問題？謝謝。

未使用所有的RAM舊方式：

// New way 
Aws::Transfer::TransferManagerConfiguration transferConfig; 
transferConfig.s3Client = SessionClient; 

std::shared_ptr<Aws::Transfer::TransferHandle> requestPtr(nullptr); 

transferConfig.downloadProgressCallback = 
     [](const Aws::Transfer::TransferManager*, const Aws::Transfer::TransferHandle& handle) 
{ 
    std::cout << "\r" << "<AWS DOWNLOAD> Download Progress: " << static_cast<int>(handle.GetBytesTransferred() * 100.0/handle.GetBytesTotalSize()) << " Percent " << handle.GetBytesTransferred() << " bytes\n"; 
}; 

Aws::Transfer::TransferManager transferManager(transferConfig); 

requestPtr = transferManager.DownloadFile(bucket.c_str(), keyName.c_str(), [&destination](){ 

    Aws::FStream *stream = Aws::New<Aws::FStream>("s3file", destination, std::ios_base::out); 
    stream->rdbuf()->pubsetbuf(NULL, 0); 

    return stream; }); 

requestPtr->WaitUntilFinished(); 

size_t retries = 0; 
//just make sure we don't fail because a download part failed. (e.g. network problems or interuptions) 
while (requestPtr->GetStatus() == Aws::Transfer::TransferStatus::FAILED && retries++ < 5) 
{ 
    std::cout << "<AWS DOWNLOAD> FW Download trying download again!" << std::endl; 
    transferManager.RetryDownload(requestPtr); 
    requestPtr->WaitUntilFinished(); 
} 

// Check status 
if (requestPtr->GetStatus() == Aws::Transfer::TransferStatus::COMPLETED) { 
    if (requestPtr->GetBytesTotalSize() == requestPtr->GetBytesTransferred()) { 
     std::cout << "<AWS DOWNLOAD> Get FW success!" << std::endl; 
     exit(0); 
    } 
    else { 
     std::cout << "<AWS DOWNLOAD> Get FW failed - Bytes downloaded did not equal requested number of bytes: " << requestPtr->GetBytesTotalSize() << requestPtr->GetBytesTransferred() << std::endl; 
     exit(1); 
    } 
} 
else { 
    std::cout << "<AWS DOWNLOAD> Get FW failed - download was never completed even after retries" << std::endl; 
    exit(1); 
}

來源

2017-05-19 PhilBot

TransferManager只有真正讓事情變得更容易，一旦：

// Old way 
    GetObjectRequest getObjectRequest; 
    getObjectRequest.SetBucket(bucket.c_str()); 
    getObjectRequest.SetKey(keyName.c_str()); 
    getObjectRequest.SetResponseStreamFactory([&destination](){ 
    return Aws::New<Aws::FStream>(
    "s3file", destination, std::ios_base::out); }); 

    GetObjectOutcome getObjectOutcome = SessionClient->GetObject(getObjectRequest); 
    if(getObjectOutcome.IsSuccess()) 
    { 
     std::cout << "<AWS DOWNLOAD> Get FW success!" << std::endl; 
    } 
    else 
    { 
     std::cout << "<AWS DOWNLOAD> Get FW failed: " << getObjectOutcome.GetError().GetMessage() << std::endl; 
     exit(1); 
    }

，最終佔用太多內存，是由內核殺死新途徑你在10mb或更大的地方，你想利用並行化。它將預先分配最大堆大小，而不會增加比這更大的堆。鑑於你的RAM限制，我不會使用TransferManager。您仍然可以收到進度通知。檢查AmazonWebServiceRequest類中的回調機制。

來源

2017-05-20 04:28:07

AWS CPP TransferManager vs GetObjectRequest流到文件fstream OOM

回答

相關問題