實體框架核心中的內存管理

Question

我試圖請求大量的數據，然後將其解析爲報告。問題是我請求的數據有2700萬行數據，每行有6個連接，當通過實體框架加載時，使用所有的服務器RAM。我實現了一個分頁系統，將處理緩存到更小的塊中，就像使用IO操作一樣。我要求10,000條記錄，將它們寫入文件流（到磁盤），我試圖從內存中清除10000條記錄，因爲它們不再需要。

我無法收集數據庫上下文的垃圾。我嘗試過處理對象，將引用歸零，然後在下一批10,000條記錄上創建一個新的上下文。這似乎並不奏效。 （這是由ef核心上的開發者之一推薦的：https://github.com/aspnet/EntityFramework/issues/5473）

我看到的唯一的另一種選擇是使用原始SQL查詢來實現我想要的。我試圖構建系統來處理任何請求大小，唯一的可變因素將是生成報告所需的時間。有什麼我可以做與EF上下文擺脫加載實體？

private void ProcessReport(ZipArchive zip, int page, int pageSize) 
     { 
      using (var context = new DBContext(_contextOptions)) 
      { 
       var batch = GetDataFromIndex(page, pageSize, context).ToArray(); 
       if (!batch.Any()) 
       { 
        return; 
       } 

       var file = zip.CreateEntry("file_" + page + ".csv"); 
       using (var entryStream = file.Open()) 
       using (var streamWriter = new StreamWriter(entryStream)) 
       { 
        foreach (var reading in batch) 
        { 
         try 
         { 
          streamWriter.WriteLine("write data from record here.") 
         } 
         catch (Exception e) 
         { 
          //handle error 
         } 
        } 
       } 
       batch = null; 
      } 
      ProcessReport(zip, page + 1, pageSize); 
     } 

private IEnumerable<Reading> GetDataFromIndex(int page, int pageSize, DBContext context) 
     { 

      var batches = (from rb in context.Reading.AsNoTracking() 
       //Some joins 
       select rb) 
       .Skip((page - 1) * pageSize) 
       .Take(pageSize); 

       return batches 
        .Includes(x => x.Something) 

     } 

Answer 1

這是EF內核中的內存泄漏。關於這個問題，GitHub問題跟蹤器上的一張票已經打開，目前預計將在EF核心2.1.0中修復。

https://github.com/aspnet/EntityFrameworkCore/issues/9367

Answer 2

除了你的內存管理的問題，你將不得不使用分頁這個不好的時候。運行分頁查詢將在服務器上變得昂貴。你不需要頁面。只需迭代查詢結果（即不要調用ToList（）或ToArray（））。

另外，當分頁時，您必須向查詢添加排序，否則SQL可能會返回重疊的行，或者存在間隙。請參閱SQL Server，例如：https://docs.microsoft.com/en-us/sql/t-sql/queries/select-order-by-clause-transact-sql EF Core不強制執行此操作，因爲某些提供程序可能會保證分頁查詢始終以相同順序讀取行。

這裏的EF核心的（1.1 .NET核心）通過一個巨大的結果集犁地不增加內存使用情況的示例：

using Microsoft.EntityFrameworkCore; 
using System.Linq; 
using System; 
using System.ComponentModel.DataAnnotations.Schema; 

namespace efCoreTest 
{ 
    [Table("SomeEntity")] 
    class SomeEntity 
    { 

     public int Id { get; set; } 
     public string Name { get; set; } 
     public string Description { get; set; } 

     public DateTime CreatedOn { get; set; } 
     public int A { get; set; } 
     public int B { get; set; } 
     public int C { get; set; } 
     public int D { get; set; } 

     virtual public Address Address { get; set; } 
     public int AddressId { get; set; } 

    } 

    [Table("Address")] 
    class Address 
    { 
     [DatabaseGenerated(DatabaseGeneratedOption.None)] 
     public int Id { get; set; } 
     public string Line1 { get; set; } 
     public string Line2 { get; set; } 
     public string Line3 { get; set; } 

    } 
    class Db : DbContext 
    { 
     public DbSet<SomeEntity> SomeEntities { get; set; } 

     protected override void OnConfiguring(DbContextOptionsBuilder optionsBuilder) 
     { 
      optionsBuilder.UseSqlServer("Server=.;Database=efCoreTest;Integrated Security=true"); 
     } 

    } 
    class Program 
    { 
     static void Main(string[] args) 
     { 
      using (var db = new Db()) 
      { 
       db.Database.EnsureDeleted(); 
       db.Database.EnsureCreated(); 

       db.Database.ExecuteSqlCommand("alter database EfCoreTest set recovery simple;"); 

       var LoadAddressesSql = @" 

with N as 
(
    select top (10) cast(row_number() over (order by (select null)) as int) i 
    from sys.objects o, sys.columns c, sys.columns c2 
) 
insert into Address(Id, Line1, Line2, Line3) 
select i Id, 'AddressLine1' Line1,'AddressLine2' Line2,'AddressLine3' Line3 
from N; 
"; 

       var LoadEntitySql = @" 

with N as 
(
    select top (1000000) cast(row_number() over (order by (select null)) as int) i 
    from sys.objects o, sys.columns c, sys.columns c2 
) 
insert into SomeEntity (Name, Description, CreatedOn, A,B,C,D, AddressId) 
select concat('EntityName',i) Name, 
     concat('Entity Description which is really rather long for Entity whose ID happens to be ',i) Description, 
     getdate() CreatedOn, 
     i A, i B, i C, i D, 1+i%10 AddressId 
from N 

"; 
       Console.WriteLine("Generating Data ..."); 
       db.Database.ExecuteSqlCommand(LoadAddressesSql); 
       Console.WriteLine("Loaded Addresses"); 

       for (int i = 0; i < 10; i++) 
       { 
        var rows = db.Database.ExecuteSqlCommand(LoadEntitySql); 
        Console.WriteLine($"Loaded Entity Batch {rows} rows"); 
       } 


       Console.WriteLine("Finished Generating Data"); 

       var results = db.SomeEntities.AsNoTracking().Include(e => e.Address).AsEnumerable(); 

       int batchSize = 10 * 1000; 
       int ix = 0; 
       foreach (var r in results) 
       { 
        ix++; 

        if (ix % batchSize == 0) 
        { 
         Console.WriteLine($"Read Entity {ix} with name {r.Name}. Current Memory: {GC.GetTotalMemory(false)/1024}kb GC's Gen0:{GC.CollectionCount(0)} Gen1:{GC.CollectionCount(1)} Gen2:{GC.CollectionCount(2)}"); 

        } 

       } 

       Console.WriteLine($"Done. Current Memory: {GC.GetTotalMemory(false)/1024}kb"); 

       Console.ReadKey(); 
      } 
     } 
    } 
} 

輸出

Generating Data ... 
Loaded Addresses 
Loaded Entity Batch 1000000 rows 
Loaded Entity Batch 1000000 rows 
. . . 
Loaded Entity Batch 1000000 rows 
Finished Generating Data 
Read Entity 10000 with name EntityName10000. Current Memory: 2854kb GC's Gen0:7 Gen1:1 Gen2:0 
Read Entity 20000 with name EntityName20000. Current Memory: 4158kb GC's Gen0:14 Gen1:1 Gen2:0 
Read Entity 30000 with name EntityName30000. Current Memory: 2446kb GC's Gen0:22 Gen1:1 Gen2:0 
. . . 
Read Entity 9990000 with name EntityName990000. Current Memory: 2595kb GC's Gen0:7429 Gen1:9 Gen2:1 
Read Entity 10000000 with name EntityName1000000. Current Memory: 3908kb GC's Gen0:7436 Gen1:9 Gen2:1 
Done. Current Memory: 3916kb 

注意，過多的存儲器的另一常見原因EF Core中的消耗是查詢的「Mixed client/server evaluation」。有關詳細信息，請參閱文檔以及如何禁用自動客戶端查詢評估。