更快創建CSV文件

Question

我通過獲取數據表然後遍歷該數據表並寫入CSV文件的每一行來創建一個CSV文件。我的數據源通常有大約65,000行。這個過程需要幾分鐘才能從瀏覽器下載。問題是本地和開發它不會花太長時間，但在客戶端，他們通常會超時。

有沒有更快的方式來生成這個文件？

Function GenerateCSVFile() As String 
    Dim stuPro As New studentProvider.StudentProvider 
    Dim emailCenter As New EmailCenter 
    Dim strFileName As String = System.IO.Path.GetRandomFileName().Replace(".", "") 
    Dim strResult As String = "" 

    Dim dtStudent As Data.DataTable 
    Dim paymentYear As String = "" 
    dtStudent = stuPro.generateDataFile() 

    If dtStudent.Rows.Count > 0 Then 

     Using sw As New System.IO.StreamWriter(Server.MapPath("Temp/" + strFileName + ".csv")) 
      Try 
       Dim lineValue As String = "" 

       lineValue += "Academic Year, StudentID, SSN, First, Middle, Last" 

       sw.WriteLine(lineValue) 

       For i As Integer = 0 To dtStudent.Rows.Count - 1 

        lineValue = dtStudent.Rows(i)("fy").ToString 
        lineValue += "," & dtStudent.Rows(i)("uniq_stu_id").ToString 
        lineValue += "," & dtStudent.Rows(i)("ssn").ToString 
        lineValue += "," & dtStudent.Rows(i)("fname").ToString 
        lineValue += "," & dtStudent.Rows(i)("mname").ToString 
        lineValue += "," & dtStudent.Rows(i)("lname").ToString 
        sw.WriteLine(lineValue) 

       Next 
      Catch ex As Exception 
       strResult += ex.ToString 
      Finally 
       sw.Close() 
      End Try 

     End Using 

     Dim strFriendlyName As String = Date.Now.ToString("MM-dd-yyyy") & ".csv" 

     If String.IsNullOrEmpty(strResult) Then 

      Dim fs As System.IO.FileStream = Nothing 

      fs = System.IO.File.Open(Server.MapPath("Temp/" + strFileName + ".csv"), System.IO.FileMode.Open) 
      Dim btFile(fs.Length) As Byte 
      fs.Read(btFile, 0, fs.Length) 
      fs.Close() 

      With Response 
       .AddHeader("Content-disposition", "attachment;filename=" & strFriendlyName) 
       .ContentType = "application/octet-stream" 
       .BinaryWrite(btFile) 
       .End() 
      End With 
     End If 
    Else 
     strResult = "No records found for specified academic year" 
    End If 

    Return strResult 
End Function 

更新的代碼

Function GenerateCSVFile() As String 
    Dim startDate As Date = Date.Now 
    Dim enddate As Date = Nothing 
    Dim stuPro As New studentProvider.StudentProvider 
    Dim emailCenter As New EmailCenter 
    Dim strFileName As String = System.IO.Path.GetRandomFileName().Replace(".", "") 
    Dim strResult As String = "" 

    Dim dtStudent As Data.DataTable 
    Dim paymentYear As String = "" 
    dtStudent = stuPro.generateDataFile(Session("VendorID"), txtAcademicYear.Text.Trim) 

    If dtStudent.Rows.Count > 0 Then 

     With Response 

      Dim strFriendlyName As String = Date.Now.ToString("MM-dd-yyyy") & ".csv" 
      .AddHeader("Content-disposition", "attachment;filename=" & strFriendlyName) 
      .ContentType = "application/octet-stream" 

      Dim lineValue As StringBuilder = New StringBuilder 

      lineValue.Append("Academic Year, StudentID, SSN, First, Middle, Last") 

      .Write(lineValue.ToString) 

      For i As Integer = 0 To dtStudent.Rows.Count - 1 

       lineValue = New StringBuilder 
       lineValue.Append(dtStudent.Rows(i)("fy").ToString) 
       lineValue.Append("," & dtStudent.Rows(i)("uniq_stu_id").ToString) 
       lineValue.Append("," & dtStudent.Rows(i)("ssn").ToString) 
       lineValue.Append("," & dtStudent.Rows(i)("fname").ToString) 
       lineValue.Append("," & dtStudent.Rows(i)("mname").ToString) 
       lineValue.Append("," & dtStudent.Rows(i)("lname").ToString) 

       .Write(lineValue.ToString) 

      Next 
      enddate = Date.Now 

      MsgBox(DateDiff(DateInterval.Second, startDate, enddate)) 

      .End() 
     End With 
    Else 
     strResult = "No records found for specified academic year" 
    End If 
    Return strResult 
End Function 

Answer 1

有一些選項，以加快這：

• 不要的StreamWriter寫入文件來響應，寫入頁面直接回應。
• 尋找一種使用數據讀取器而不是數據表來循環訪問數據的解決方案，使用這些數據可能會更快。它也會降低內存使用量（不會在內存中加載整個表）。
• 如果連接很多字符串使用StringBuilder，或者在這種情況下使用String.Join來輕鬆創建整行。

的string.join例如：

For Each row in dtStudent.Rows 
    Dim line as new List(of String) 
    line.Add(row("fy").ToString) 
    line.Add(row("uniq_stu_id").ToString) 
    line.Add(-etc-) 

    Response.Write(String.Join(",", line.ToArray) & vbcrlf) 
Next 

Answer 2

您正在編寫到一個臨時文件，讀取該文件中，他們編寫文件的內容到響應。跳過臨時文件並一次直接寫入響應一行。這將阻止瀏覽器認爲你的服務器超時，讓事情更快，並減少你的應用程序消耗的內存量。

之後，請查看如何在此Web請求上啓用緩存，以便ASP.NET在短時間內如果有多個用戶要求時不必重新創建CSV。

Answer 3

你應該使用StringBuilder而不是串聯。

Answer 4

除了@Robert利維的建議，您如何使用字符串變量要小心。你會更好地使用這些線上使用stringbuilder服務：

 dim sbTemp as new StringBuilder() 
     For i As Integer = 0 To dtStudent.Rows.Count - 1 

      sbTemp.Append(dtStudent.Rows(i)("fy").ToString) 
      sbTemp.Append(",") 
      sbTemp.Append(dtStudent.Rows(i)("uniq_stu_id").ToString) 

      'etc 
      sw.WriteLine(lineValue) 

     Next 

Answer 5

有一件事，你可以看看是生產者/消費者的設計模式。這可以讓你做的是有一個（或多個）線程提供一個包含需要寫入csv文件的數據的隊列和另一個執行實際寫入的線程（或多個）。