卡斯巴與大量返回的對象

Question

卡斯巴（或MongoDB的java的驅動程序）的問題似乎有應付大量返回的對象的問題。例如，下面的代碼段會產生IllegalArgumentException，並且不會返回單個結果（下面的完整堆棧跟蹤）。但是，如果我將1994年的「極限（...）」降低，一切似乎都可以正常工作。

for (link <- links; query = link $exists true) { 
    val group = new HashMap[String, Set[(String, String)]] with MultiMap[String, (String, String)] 
    log.find(query, fieldsToGet.result).limit(1996) foreach { 

     x => { 
     group.addBinding(x.get(link).toString, (x.get("_id").toString(), x.get("eventType").toString)) 
     } 

    } 
    allGroups += link -> group 
    } 

Apr 26, 2011 8:23:40 PM com.mongodb.DBTCPConnector$MyPort error 
SEVERE: MyPort.error called 
java.lang.IllegalArgumentException: response too long: 1278031173 
    at com.mongodb.Response.<init>(Response.java:40) 
    at com.mongodb.DBPort.go(DBPort.java:101) 
    at com.mongodb.DBPort.go(DBPort.java:66) 
    at com.mongodb.DBPort.call(DBPort.java:56) 
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211) 
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:266) 
    at com.mongodb.DBCursor._check(DBCursor.java:309) 
    at com.mongodb.DBCursor._hasNext(DBCursor.java:431) 
    at com.mongodb.DBCursor.hasNext(DBCursor.java:456) 
    at com.mongodb.casbah.MongoCursorBase$class.hasNext(MongoCursor.scala:72) 
    at com.mongodb.casbah.MongoCursor.hasNext(MongoCursor.scala:517) 
    at scala.collection.Iterator$class.foreach(Iterator.scala:631) 
    at com.mongodb.casbah.MongoCursor.foreach(MongoCursor.scala:517) 
    at Sequencer$$anonfun$3.apply(Sequencer.scala:23) 
    at Sequencer$$anonfun$3.apply(Sequencer.scala:20) 
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) 
    at scala.collection.immutable.List.foreach(List.scala:45) 
    at Sequencer$.<init>(Sequencer.scala:20) 
    at Sequencer$.<clinit>(Sequencer.scala) 
    at Sequencer.main(Sequencer.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115) 
Exception in thread "main" java.lang.ExceptionInInitializerError 
    at Sequencer.main(Sequencer.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) 
    at java.lang.reflect.Method.invoke(Method.java:597) 
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:115) 
Caused by: java.lang.IllegalArgumentException: response too long: 1278031173 
    at com.mongodb.Response.<init>(Response.java:40) 
    at com.mongodb.DBPort.go(DBPort.java:101) 
    at com.mongodb.DBPort.go(DBPort.java:66) 
    at com.mongodb.DBPort.call(DBPort.java:56) 
    at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:211) 
    at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:266) 
    at com.mongodb.DBCursor._check(DBCursor.java:309) 
    at com.mongodb.DBCursor._hasNext(DBCursor.java:431) 
    at com.mongodb.DBCursor.hasNext(DBCursor.java:456) 
    at com.mongodb.casbah.MongoCursorBase$class.hasNext(MongoCursor.scala:72) 
    at com.mongodb.casbah.MongoCursor.hasNext(MongoCursor.scala:517) 
    at scala.collection.Iterator$class.foreach(Iterator.scala:631) 
    at com.mongodb.casbah.MongoCursor.foreach(MongoCursor.scala:517) 
    at Sequencer$$anonfun$3.apply(Sequencer.scala:23) 
    at Sequencer$$anonfun$3.apply(Sequencer.scala:20) 
    at scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:61) 
    at scala.collection.immutable.List.foreach(List.scala:45) 
    at Sequencer$.<init>(Sequencer.scala:20) 
    at Sequencer$.<clinit>(Sequencer.scala) 
    ... 6 more 

似乎在java驅動程序中的「Response.java」中檢查了以下異常。

ByteArrayInputStream bin = new ByteArrayInputStream(b); 
_len = Bits.readInt(bin); 
if (_len > (32 * 1024 * 1024)) 
throw new IllegalArgumentException("response too long: " + _len); 

難道由該特定對象造成的退換嗎？或者這可能是關於casbah？

感謝， 德里克

Answer 1

它看起來像Java驅動程序是檢查是否當前響應塊大於32兆字節，然後拋出異常。

如果設置光標上的BATCHSIZE（FEWER_NUMBER_OF_DOCS），這將減少鎖定時間的數據庫，並返回小於32 MB的價值的數據。

我會玩的BATCHSIZE，看看有什麼是最適合你的應用程序。

http://api.mongodb.org/scala/casbah/2.1.2/scaladoc/

最大可能應該在Java驅動程序而增加。

關於你的迴應奇怪的是，它說，它返回〜1.19 GB的數據。

如果你的反應並沒有那麼多的數據，則可能是收集已損壞。