使用JSON.NET解析1GB json文件的問題

Question

我已經得到了一個應用程序，其中輸入已經從50K位置記錄放大到1.1百萬位置記錄。 由於整個文件先前已反序列化爲單個對象，因此造成嚴重問題。 該對象的大小爲〜1GB，用於生成具有110萬條記錄的文件。 由於大對象GC問題，我想將反序列化的對象保持在85K以下。

我試圖一次解析出一個位置對象，並將其反序列化，以便我可以控制對象數量 ，這些對象被反序列化並反過來控制對象的大小。我正在使用Json.Net庫來執行此操作。

下面是我作爲流接收到我的應用程序中的JSON文件的示例。

{ 
    "Locations": [{ 
     "LocationId": "", 
     "ParentLocationId": "", 
     "DisplayFlag": "Y", 
     "DisplayOptions": "", 
     "DisplayName": "", 
     "Address": "", 
     "SecondaryAddress": "", 
     "City": "", 
     "State": "", 
     "PostalCode": "", 
     "Country": "", 
     "Latitude": 40.59485, 
     "Longitude": -73.96174, 
     "LatLonQuality": 99, 
     "BusinessLogoUrl": "", 
     "BusinessUrl": "", 
     "DisplayText": "", 
     "PhoneNumber": "", 
     "VenueGroup": 7, 
     "VenueType": 0, 
     "SubVenue": 0, 
     "IndoorFlag": "", 
     "OperatorDefined": "", 
     "AccessPoints": [{ 
      "AccessPointId": "", 
      "MACAddress": "", 
      "DisplayFlag": "", 
      "DisplayOptions": "", 
      "Latitude": 40.59485, 
      "Longitude": -73.96174, 
      "Status": "Up", 
      "OperatorDefined": "", 
      "RoamingGroups": [{ 
       "GroupName": "" 
      }, 
      { 
       "GroupName": "" 
      }], 
      "Radios": [{ 
       "RadioId": "", 
       "RadioFrequency": "", 
       "RadioProtocols": [{ 
        "Protocol": "" 
       }], 
       "WifiConnections": [{ 
        "BSSID": "", 
        "ServiceSets": [{ 
         "SSID": "", 
         "SSID_Broadcasted": "" 
        }] 
       }] 
      }] 
     }] 
    }, 
    { 
     "LocationId": "", 
     "ParentLocationId": "", 
     "DisplayFlag": "Y", 
     "DisplayOptions": "", 
     "DisplayName": "", 
     "Address": "", 
     "SecondaryAddress": "", 
     "City": "", 
     "State": "", 
     "PostalCode": "", 
     "Country": "", 
     "Latitude": 40.59485, 
     "Longitude": -73.96174, 
     "LatLonQuality": 99, 
     "BusinessLogoUrl": "", 
     "BusinessUrl": "", 
     "DisplayText": "", 
     "PhoneNumber": "", 
     "VenueGroup": 7, 
     "VenueType": 0, 
     "SubVenue": 0, 
     "IndoorFlag": "", 
     "OperatorDefined": "", 
     "AccessPoints": [{ 
      "AccessPointId": "", 
      "MACAddress": "", 
      "DisplayFlag": "", 
      "DisplayOptions": "", 
      "Latitude": 40.59485, 
      "Longitude": -73.96174, 
      "Status": "Up", 
      "OperatorDefined": "", 
      "RoamingGroups": [{ 
       "GroupName": "" 
      }, 
      { 
       "GroupName": "" 
      }], 
      "Radios": [{ 
       "RadioId": "", 
       "RadioFrequency": "", 
       "RadioProtocols": [{ 
        "Protocol": "" 
       }], 
       "WifiConnections": [{ 
        "BSSID": "", 
        "ServiceSets": [{ 
         "SSID": "", 
         "SSID_Broadcasted": "" 
        }] 
       }] 
      }] 
     }] 
    }] 
} 

我需要能夠拉出個別位置的對象，這樣我就可以查看以下

{ 
    "LocationId": "", 
    "ParentLocationId": "", 
    "DisplayFlag": "Y", 
    "DisplayOptions": "", 
    "DisplayName": "", 
    "Address": "", 
    "SecondaryAddress": "", 
    "City": "", 
    "State": "", 
    "PostalCode": "", 
    "Country": "", 
    "Latitude": 40.59485, 
    "Longitude": -73.96174, 
    "LatLonQuality": 99, 
    "BusinessLogoUrl": "", 
    "BusinessUrl": "", 
    "DisplayText": "", 
    "PhoneNumber": "", 
    "VenueGroup": 7, 
    "VenueType": 0, 
    "SubVenue": 0, 
    "IndoorFlag": "", 
    "OperatorDefined": "", 
    "AccessPoints": [{ 
     "AccessPointId": "", 
     "MACAddress": "", 
     "DisplayFlag": "", 
     "DisplayOptions": "", 
     "Latitude": 40.59485, 
     "Longitude": -73.96174, 
     "Status": "Up", 
     "OperatorDefined": "", 
     "RoamingGroups": [{ 
      "GroupName": "" 
     }, 
     { 
      "GroupName": "" 
     }], 
     "Radios": [{ 
      "RadioId": "", 
      "RadioFrequency": "", 
      "RadioProtocols": [{ 
       "Protocol": "" 
      }], 
      "WifiConnections": [{ 
       "BSSID": "", 
       "ServiceSets": [{ 
        "SSID": "", 
        "SSID_Broadcasted": "" 
       }] 
      }] 
     }] 
    }] 
} 

我試圖使用Json.NET JsonTextReader做到這一點，但是我不能讓讀者在緩衝區中包含整個位置，這是由於流中記錄的大小，讀者最初將會下降到「RadioProtocols」，該時間位於對象的中間位置流到達對象的末尾，讀者已經丟棄了對象的開始。

我使用的是嘗試讓此功能工作的代碼是

var ser = new JsonSerializer(); 
using (var reader = new JsonTextReader(new StreamReader(stream))) 
{ 
    reader.SupportMultipleContent = true; 

    while (reader.Read()) 
    { 
     if (reader.TokenType == JsonToken.StartObject && reader.Depth == 2) 
     {        
      do 
      { 
       reader.Read();         
      } while (reader.TokenType != JsonToken.EndObject && reader.Depth == 2); 

      var singleLocation = ser.Deserialize<Locations>(reader); 
     } 
    } 
} 

在這個或任何信息，這樣做將不勝感激的替代品。作爲一個附註，我們的客戶發送信息的方式目前無法改變。

Answer 1

感謝所有的幫助，我設法讓它做我想做的事情，即對各個位置對象進行反序列化。

如果項目轉換爲JObject，它將讀取完整對象並將其反序列化，這可以循環以獲得解決方案。

這是已在結算代碼

while (reader.Read()) 
{ 
    if (reader.TokenType == JsonToken.StartObject && reader.Depth == 2) 
    { 
     location = JObject.Load(reader).ToObject<Location>(); 

     var lv = new LocationValidator(location, FootprintInfo.OperatorId, FootprintInfo.RoamingGroups, true); 
     var vr = lv.IsValid(); 
     if (vr.Successful) 
     { 
      yield return location; 
     } 
     else 
     { 
      errors.Add(new Error(elNumber, location.LocationId, vr.Error.Field, vr.Error.Detail)); 
      if (errors.Count >= maxErrors) 
      { 
       yield break; 
      } 
     } 

     ++elNumber; 
    } 
} 

Answer 2

當閱讀器定位在要反序列化（在Locations陣列中的情況下的條目）的對象，你可以叫ser.Deserialize<T>(reader)，它會工作，推進到對象在該結束的開始級別，並沒有進一步的。因此，下面應通過在文件中Location對象迭代，裝載分別各一個：

public static IEnumerable<T> DeserializeNestedItems<T>(TextReader textReader) 
    { 
     var ser = new JsonSerializer(); 
     using (var reader = new JsonTextReader(textReader)) 
     { 
      reader.SupportMultipleContent = true; 

      while (reader.Read()) 
      { 
       if (reader.TokenType == JsonToken.StartObject && reader.Depth == 2) 
       { 
        var item = ser.Deserialize<T>(reader); 
        yield return item; 
       } 
      } 
     } 
    } 

和使用測試串的使用的例子：

 Debug.Assert(DeserializeNestedItems<Location>(new StringReader(json)).Count() == 2); // No assert. 

     var list = DeserializeNestedItems<Location>(new StringReader(json)).SelectMany(l => l.AccessPoints).Select(a => new { a.Latitude, a.Longitude }).ToList(); 

     Debug.WriteLine(JsonConvert.SerializeObject(list, Formatting.Indented)); 

，輸出：

[ 
    { 
    "Latitude": 40.59485, 
    "Longitude": -73.96174 
    }, 
    { 
    "Latitude": 40.59485, 
    "Longitude": -73.96174 
    } 
] 

注 - Location類來f rom張貼您的JSON到http://json2csharp.com/。