使用Java將多個文件並行下載到S3中的zip文件

Question

我想從S3等外部源下載多個文件，創建一個包含所有這些文件的zip文件，並向用戶提供一個鏈接以下載該文件zip文件。

很顯然，我可以按順序處理文件，讀取每個文件的輸入流並將其寫入ZipOutputStream。

如何並行讀取所有輸入文件流並寫入單個輸出流，以便我可以向用戶展示下載鏈接，而不必等到zip文件完全寫入？

我當前的代碼：

String realpath = getServletContext().getRealPath("/"); 
response.setContentType("application/zip"); 

response.setHeader("Content-Disposition","attachment; filename="+fi.replace('/', '-')+"_"+ff.replace('/', '-')+".zip"); 

ServletOutputStream out = null; 
ZipOutputStream zipfile = null; 

try 
{ 
    List<Object[]> cfdis = /*my hibernate criteria source, your Database?*/ 
    out = response.getOutputStream(); 
    zipfile = new ZipOutputStream(out); 
    ZipEntry zipentry = null; 
    for(Object[] cfdi:cfdis) 
    { 
     zipentry = new ZipEntry(cfdi[1].toString()+".xml"); 
     zipfile.putNextEntry(zipentry); 
     InputStream in = new FileInputStream(new File(realpath+cfdi[0].toString())); 
     byte[] bytes = new byte[FILEBUFFERSIZE]; 
     int bytesRead; 
     while ((bytesRead = in.read(bytes)) != -1) 
     { 
      zipfile.write(bytes, 0, bytesRead); 
     } 
    } 

Answer 1

的第一個文件&開始寫它讀取流的OutputStream

protected void doGet(HttpServletRequest request, HttpServletResponse response) 
     throws ServletException, IOException { 

    ServletOutputStream sos = response.getOutputStream(); 
    ZipOutputStream zos = new ZipOutputStream(sos); 

    try { 
     S3ServiceWrapper s3Service = new S3ServiceWrapper(); 
     ZipEntry zipentry = null; 
     byte bytes[] = new byte[4096]; 
     response.setContentType("application/zip"); 
     response.setHeader("Content-Disposition", "attachment; filename=java-s3-download.ZIP"); 

     for (String objectKey : objectKeys) { 
      String name = objectKey.substring(objectKey.lastIndexOf("/"), objectKey.length()); 
      log.info("Start Writing File::" + name); 

      zipentry = new ZipEntry(name); 
      zos.putNextEntry(zipentry); 
      InputStream in = s3Service.downloadFileAsStream(bucketName, objectKey); 
      int bytesRead = -1; 

      while ((bytesRead = in.read(bytes)) != -1) { 
       zos.write(bytes, 0, bytesRead); 
      } 

      log.info("Finsih Writing File::" + name); 
      in.close(); 
     } 
    } catch (Exception e) 

    { 
     e.printStackTrace(); 
    } finally { 
     zos.flush(); 
     zos.closeEntry(); 
     zos.close(); 
     sos.close(); 
    } 

} 

public InputStream downloadFileAsStream(String bucketName, String objectKey) { 
    if (s3Service == null) { 
     return null; 
    } 

    try { 
     GetObjectRequest s3ObjectReq = new GetObjectRequest(bucketName, objectKey); 
     log.info("Downloading file having key = " + objectKey); 
     long startTime=System.currentTimeMillis(); 
     S3Object downlodedObjectMD = s3Service.getObject(s3ObjectReq); 
     log.info("Time to load Stream is "+(System.currentTimeMillis()-startTime)+" ms"); 
     return downlodedObjectMD.getObjectContent(); 

    } catch (Exception e) { 
     log.error("EXCEPTION = " + e.getMessage(), e); 
    } 
    return null; 
} 

Answer 2

有一個在並行讀取文件沒有優勢，因爲只有一個線程可以一次寫入zip文件。

您可以輕鬆地在壓縮文件寫入時進行壓縮。檢查此博客的完整代碼： https://purushramrajblog.wordpress.com/2015/11/22/dynamically-construct-a-zip-file-and-stream-it-with-jersey/