使用Epplus導入Excel電子表格時的代碼優化

Question

我需要從Excel電子表格中爲廣告系列導入35,000個以上的代碼。我有下面的代碼（它以我需要的方式工作），但是當我運行該過程時，可能需要20-30分鐘才能完成。

如果有什麼可以做的，以幫助加快過程，將不勝感激。我不會稱自己爲高級程序員，我知道這可能是通過先進的編碼技術完成的。如果有人能指引我，我將不勝感激。

廣告系列和主碼錶的型號。

public class Campaign 
{ 
    public int CampaignId { get; set; } 
    public string Name { get; set; } 
    public virtual List<MasterCode> MasterCodes { get; set; } 
} 

public class MasterCode 
{ 
    public int MasterCodeId { get; set; } 

    public int CampaignId { get; set; } 
    public virtual Campaign Campaign { get; set; } 

    public string Code { get; set; } 
    public bool Used { get; set; } 
    public DateTime SubmittedOn { get; set; } 
} 

以下是視圖中的代碼。表單字段不是模型綁定的，因爲此代碼位於已經模型綁定到另一個模型的視圖上的彈出窗口中。

@using (Html.BeginForm("UploadMasterCodes", "Campaigns", FormMethod.Post, new { enctype = "multipart/form-data" })) 
{ 
    @Html.AntiForgeryToken() 
    @Html.Hidden("CampaignId", Model.CampaignId) 

    <div class="form-group"> 
     @Html.Label("Master Code File") 
     @Html.TextBox("File", null, new { type = "file", @class = "form-control" }) 
    </div> 
} 

控制器代碼

[HttpPost] 
[ValidateAntiForgeryToken] 
public ActionResult UploadMasterCodes(CampaignMasterCodeUploadViewModel model) 
{ 
    if (ModelState.IsValid) 
    { 
     var result = CampaignMethods.uploadMasterCodes(model.File, model.CampaignId); 
     TempData["SuccessMessage"] = result; 
     return RedirectToAction("Details", new { id = model.CampaignId }); 
    } 

    return RedirectToAction("Details", new { id = model.CampaignId }); 
} 

最後這裏是執行上傳到數據庫的代碼。我正在構建一個字符串，用於跟蹤所有發生的錯誤，因爲我需要了解它們。

public static string uploadMasterCodes(HttpPostedFileBase file, int campaignId) 
    { 
     using (ApplicationDbContext _context = new ApplicationDbContext()) 
     { 
      string response = string.Empty; 

      var campaign = _context.Campaigns.Find(campaignId); 
      if (campaign == null) 
      { 
       return String.Format("Campaign {0} not found", campaignId.ToString()); 
      } 

      var submitDate = DateTime.Now; 
      int successCount = 0; 
      int errorCount = 0; 

      if ((file != null) && (file.ContentLength > 0) && !string.IsNullOrEmpty(file.FileName)) 
      { 
       byte[] fileBytes = new byte[file.ContentLength]; 
       var data = file.InputStream.Read(fileBytes, 0, Convert.ToInt32(file.ContentLength)); 

       using (var package = new ExcelPackage(file.InputStream)) 
       { 
        var currentSheet = package.Workbook.Worksheets; 
        var workSheet = currentSheet.First(); 
        var noOfRow = workSheet.Dimension.End.Row; 

        for (int i = 1; i <= noOfRow; i++) 
        { 
         var masterCode = new MasterCode(); 
         masterCode.Code = workSheet.Cells[i, 1].Value.ToString(); 
         masterCode.CampaignId = campaignId; 
         masterCode.Campaign = campaign; 
         masterCode.SubmittedOn = submitDate; 

         // Check to make sure that the Code does not already exist. 
         if (!campaign.MasterCodes.Any(m => m.Code == masterCode.Code)) 
         { 
          try 
          { 
           _context.MasterCodes.Add(masterCode); 
           _context.SaveChanges(); 
           successCount++; 
          } 
          catch (Exception ex) 
          { 
           response += String.Format("Code: {0} failed with error: {1} <br />", masterCode.Code, ex.Message); 
           errorCount++; 
          } 
         } 
         else 
         { 
          response += String.Format("Code: {0} already exists <br />", masterCode.Code); 
          errorCount++; 
         } 
        } 

        response += string.Format("Number of codes:{0}/Success: {1}/Errors {2}", noOfRow.ToString(), successCount.ToString(), errorCount.ToString()); 
       } 
      } 

      return response; 
     } 
    } 

Answer 1

關於@grmbl的建議和大量閱讀，我能夠使用SQLBulkCopy解決速度問題。我沒有使用OLEDB提供程序，而是將文件複製到服務器並創建了一個數據表。然後我用它來批量複製。下面的代碼將運行時間降低到大約10秒鐘，以獲得335,000條記錄。

我已經刪除了所有的錯誤檢查代碼，所以它不是一堵牆代碼。 控制器代碼。

[HttpPost] 
    [ValidateAntiForgeryToken] 
    public ActionResult UploadMasterCodes(CampaignMasterCodeUploadViewModel model) 
    { 
     if (ModelState.IsValid) 
     { 
      var filename = Path.GetFileName(model.File.FileName); 
      var path = FileMethods.UploadFile(model.File, Server.MapPath("~/App_Data/Bsa4/"), filename); 
      var dt = CampaignMethods.ProcessMasterCodeCsvToDatatable(path, model.CampaignId); 
      TempData["SuccessMessage"] = CampaignMethods.ProcessMastercodeSqlBulkCopy(dt); 
      return RedirectToAction("Details", new { id = model.CampaignId });     
     } 

     TempData["ErrorMessage"] = "Master code upload form error. Please refresh the page and try again."; 
     return RedirectToAction("Details", new { id = model.CampaignId }); 
    } 

處理代碼。

public static DataTable ProcessMasterCodeCsvToDatatable(string file, int campaignId) 
    { 
     using (ApplicationDbContext _context = new ApplicationDbContext()) { 

      var campaign = _context.Campaigns.Find(campaignId); 

      DataTable dt = new DataTable(); 
      dt.Columns.Add("CampaignId"); 
      dt.Columns.Add("Code"); 
      dt.Columns.Add("Used"); 
      dt.Columns.Add("SubmittedOn"); 

      string line = null; 
      var submitDate = DateTime.Now; 

      using (StreamReader sr = File.OpenText(file)) 
      { 
       while ((line = sr.ReadLine()) != null) 
       { 
        string[] data = line.Split(','); 
        if (data.Length > 0) 
        { 
         if (!string.IsNullOrEmpty(data[0])) 
         { 
          DataRow row = dt.NewRow(); 
          row[0] = campaign.CampaignId; 
          row[1] = data[0]; 
          row[2] = false; 
          row[3] = submitDate; 

          dt.Rows.Add(row); 
         } 
        } 
       } 
      } 

      return dt; 
     } 
    } 

    public static String ProcessMastercodeSqlBulkCopy(DataTable dt) 
    { 
     string Feedback = string.Empty; 

     using (SqlConnection cn = new SqlConnection(ConfigurationManager.ConnectionStrings["DefaultConnection"].ConnectionString)) 
     { 
      cn.Open(); 
      using (SqlBulkCopy copy = new SqlBulkCopy(cn)) 
      { 
       copy.ColumnMappings.Add(0, "CampaignId"); 
       copy.ColumnMappings.Add(2, "Code"); 
       copy.ColumnMappings.Add(3, "Used"); 
       copy.ColumnMappings.Add(4, "SubmittedOn"); 
       copy.DestinationTableName = "MasterCodes"; 
       try 
       { 
        //Send it to the server 
        copy.WriteToServer(dt); 
        Feedback = "Master Code Upload completed successfully"; 
       } 
       catch (Exception ex) 
       { 
        Feedback = ex.Message; 
       } 
      } 
     } 

     return Feedback; 
    } 

Answer 2

每當我需要閱讀我用這個OLEDB提供一個Excel文件：

Provider=Microsoft.ACE.OLEDB.12.0;Data Source={FilePath};Extended Properties='Excel 12.0 Xml;HDR=Yes;IMEX=1'; 

（關於這個主題的更多信息，可以發現here和here）

，那麼你可以爲最快的插入物做一個bulk insert。 請參閱this answer如何在內存中執行此操作。在你的情況下，你首先需要將文件保存在服務器的某個地方，處理並向用戶報告進度。