JVM狀態確定爲不穩定。有力地退出。什麼是Java Stability Inspector？爲什麼它停止DSE？

從前幾天我看到在cassandra集羣中的一些節點因爲下面的錯誤，DSE正在關閉，我需要終止Java進程並重新啓動DSE服務。JVM狀態確定爲不穩定。有力地退出。什麼是Java Stability Inspector？爲什麼它停止DSE？

我有交叉檢查讀取和寫入和壓縮沒有看起來可疑，但我看到完全Gc在問題發生之前暫停在這些服務器上。這可能是什麼原因，如何調試？如何在這些特定的節點上爲什麼會發生這種情況，當這些節點正在像羣集的其他部分一樣提供相同的請求？

是不獲取正確地執行這種情況發生，因爲完整的GC中，我們使用G1GC和DSE 4.8.3

ERROR [SharedPool-Worker-25] 2016-12-27 10:14:26,100 JVMStabilityInspector.java:117 - JVM state determined to be unstable. Exiting forcefully due to:java.lang.OutOfMemoryError: Java heap space 
    at java.util.Arrays.copyOf(Arrays.java:3181) ~[na:1.8.0_74] 
at org.apache.cassandra.db.RangeTombstoneList.copy(RangeTombstoneList.java:112) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.db.DeletionInfo.copy(DeletionInfo.java:104) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:217) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.db.Memtable.put(Memtable.java:210) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1230) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.db.Mutation.apply(Mutation.java:214) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.db.MutationVerbHandler.doVerb(MutationVerbHandler.java:54) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_74] 
      at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] 

    ERROR [SharedPool-Worker-25] 2016-12-27 10:14:28,100 SEPWorker.java:141 - Failed to execute task, unexpected exception killed worker: {} 
    java.lang.IllegalStateException: Shutdown in progress 
      at java.lang.ApplicationShutdownHooks.remove(ApplicationShutdownHooks.java:82) ~[na:1.8.0_74] 
      at java.lang.Runtime.removeShutdownHook(Runtime.java:239) ~[na:1.8.0_74] 
      at org.apache.cassandra.service.StorageService.removeShutdownHook(StorageService.java:764) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:119) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.utils.JVMStabilityInspector$Killer.killCurrentJVM(JVMStabilityInspector.java:109) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.utils.JVMStabilityInspector.inspectThrowable(JVMStabilityInspector.java:68) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
      at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:168) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
at 
org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) ~[cassandra-all-2.1.13.1131.jar:2.1.13.1131] 
at 
java.lang.Thread.run(Thread.java:745) [na:1.8.0_74] 

    INFO [Thread-6] 2016-12-27 10:14:56,150 DseDaemon.java:420 - DSE shutting down...

來源

2017-01-19 user6288321

這是一個內存不足的例外，你的堆有多大？ – phact

@phact堆是12Gb，新堆是4Gb。 – user6288321

您使用的是搜索嗎？ – phact

檢查，如果你的任何大的分區都存在於卡桑德拉表，如果有任何疑問正在閱讀很多墓碑。

來源

2017-03-15 22:15:28

JVM狀態確定爲不穩定。有力地退出。什麼是Java Stability Inspector？爲什麼它停止DSE？

回答

相關問題