我們在CentOS 6.3上運行Tomcat 7.0.34(完全修補),並且服務器啓動後,所有的應用服務器的CPU都會猛增。 Tomcat沒有連接。看起來有一些CPU內核被我們的webapp中運行的東西完全消耗掉了。Tomcat 7在無連接啓動時的CPU使用率過高
我不能爲我的生活找出什麼可能導致這個問題。有沒有人見過這個?
我應該提到這隻發生在CentOS盒子上。我的Windows tomcat服務器不顯示運行相同應用程序的此行爲。
相關技術
我們使用Tomcat的會話複製,複製的EHCache和HornetQ的JMS的。
的Java版本測試
JDK 1.7.0更新10和JDK 1.6.0更新38
殺-3運行線程
"http-apr-8080-Acceptor-1" daemon prio=10 tid=0x00007f2598183000 nid=0x46b9 runnable [0x00007f2500685000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1013)
at java.lang.Thread.run(Thread.java:722)
"http-apr-8080-Acceptor-0" daemon prio=10 tid=0x00007f2598181800 nid=0x46b8 runnable [0x00007f2500786000]
java.lang.Thread.State: RUNNABLE
at org.apache.tomcat.jni.Socket.accept(Native Method)
at org.apache.tomcat.util.net.AprEndpoint$Acceptor.run(AprEndpoint.java:1013)
at java.lang.Thread.run(Thread.java:722)
"New I/O boss #26" daemon prio=10 tid=0x00007f253d4fb000 nid=0x462f runnable [0x00007f250c49c000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006ab002568> (a sun.nio.ch.Util$2)
- locked <0x00000006ab002558> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006ab0024d0> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:41)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"New I/O worker #25" daemon prio=10 tid=0x00007f253d4e4000 nid=0x462e runnable [0x00007f250c59d000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006aefbcab8> (a sun.nio.ch.Util$2)
- locked <0x00000006aefbcaa8> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006aefbca60> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"New I/O worker #24" daemon prio=10 tid=0x00007f253d4b9000 nid=0x462d runnable [0x00007f250c69e000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006ab002c70> (a sun.nio.ch.Util$2)
- locked <0x00000006ab002c60> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006ab002bd8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
... ABOUT 20 MORE NIO WORKERS IN RUNNABLE STATE...
"New I/O worker #2" prio=10 tid=0x00007f253d054000 nid=0x4616 runnable [0x00007f250ddb5000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006adaa7070> (a sun.nio.ch.Util$2)
- locked <0x00000006adaa7060> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006adaa6fd8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"New I/O worker #1" prio=10 tid=0x00007f253d053000 nid=0x4615 runnable [0x00007f250deb6000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006adaa7bc0> (a sun.nio.ch.Util$2)
- locked <0x00000006adaa7bb0> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006adaa7b28> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.jboss.netty.channel.socket.nio.SelectorUtil.select(SelectorUtil.java:64)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.select(AbstractNioSelector.java:409)
at org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:206)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:88)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at org.jboss.netty.util.VirtualExecutorService$ChildExecutorRunnable.run(VirtualExecutorService.java:175)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
"NamingBootstrap Pool(1)-1" daemon prio=10 tid=0x00007f253c4f8800 nid=0x45fc runnable [0x00007f25842a3000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at org.jnp.server.Main$AcceptHandler.run(Main.java:481)
at org.jboss.util.threadpool.RunnableTaskWrapper.run(RunnableTaskWrapper.java:148)
at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-1098" daemon prio=10 tid=0x00007f253c4f4800 nid=0x45fb runnable [0x00007f25843a4000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-0" daemon prio=10 tid=0x00007f253cc2d800 nid=0x45e6 runnable [0x00007f25859b9000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-40001" daemon prio=10 tid=0x00007f253cc16000 nid=0x45da runnable [0x00007f25865c5000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"Multicast Heartbeat Receiver Thread" daemon prio=10 tid=0x00007f253cc18000 nid=0x45d8 runnable [0x00007f25867c7000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
- locked <0x00000006ab145b28> (a java.net.PlainDatagramSocketImpl)
at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
- locked <0x00000006ab145b28> (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:786)
- locked <0x000000067ffd9240> (a java.net.DatagramPacket)
- locked <0x00000006ab1459f0> (a java.net.MulticastSocket)
at net.sf.ehcache.distribution.MulticastKeepaliveHeartbeatReceiver$MulticastReceiverThread.run(MulticastKeepaliveHeartbeatReceiver.java:124)
"Tribes-MembershipReceiver" daemon prio=10 tid=0x00007f259895c800 nid=0x45d0 runnable [0x00007f258733a000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainDatagramSocketImpl.receive0(Native Method)
- locked <0x00000006b8163208> (a java.net.PlainDatagramSocketImpl)
at java.net.AbstractPlainDatagramSocketImpl.receive(AbstractPlainDatagramSocketImpl.java:145)
- locked <0x00000006b8163208> (a java.net.PlainDatagramSocketImpl)
at java.net.DatagramSocket.receive(DatagramSocket.java:786)
- locked <0x00000006b8158768> (a java.net.DatagramPacket)
- locked <0x00000006b7b4fc30> (a java.net.MulticastSocket)
at org.apache.catalina.tribes.membership.McastServiceImpl.receive(McastServiceImpl.java:340)
at org.apache.catalina.tribes.membership.McastServiceImpl$ReceiverThread.run(McastServiceImpl.java:534)
"NioReceiver" daemon prio=10 tid=0x00007f259892f000 nid=0x45cf runnable [0x00007f258743b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006b8155008> (a sun.nio.ch.Util$2)
- locked <0x00000006b8155020> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006b78650a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:281)
at org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:420)
at java.lang.Thread.run(Thread.java:722)
"NioReceiver" daemon prio=10 tid=0x00007f259892f000 nid=0x45cf runnable [0x00007f258743b000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:228)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:81)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
- locked <0x00000006b8155008> (a sun.nio.ch.Util$2)
- locked <0x00000006b8155020> (a java.util.Collections$UnmodifiableSet)
- locked <0x00000006b78650a8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
at org.apache.catalina.tribes.transport.nio.NioReceiver.listen(NioReceiver.java:281)
at org.apache.catalina.tribes.transport.nio.NioReceiver.run(NioReceiver.java:420)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-0" daemon prio=10 tid=0x00007f25982e0800 nid=0x45cc runnable [0x00007f258c7a6000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-9004" daemon prio=10 tid=0x00007f25982cb000 nid=0x45cb runnable [0x00007f258c8a7000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"RMI TCP Accept-0" daemon prio=10 tid=0x00007f25982a7000 nid=0x45ca runnable [0x00007f258c9a8000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)
"Service Thread" daemon prio=10 tid=0x00007f259817a800 nid=0x45c9 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread1" daemon prio=10 tid=0x00007f2598178000 nid=0x45c8 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" daemon prio=10 tid=0x00007f2598175800 nid=0x45c7 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" daemon prio=10 tid=0x00007f2598173800 nid=0x45c6 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Surrogate Locker Thread (Concurrent GC)" daemon prio=10 tid=0x00007f2598171800 nid=0x45c5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"VM Thread" prio=10 tid=0x00007f259811c000 nid=0x45c2 runnable
"Gang worker#0 (Parallel GC Threads)" prio=10 tid=0x00007f2598012000 nid=0x45bd runnable
"Gang worker#1 (Parallel GC Threads)" prio=10 tid=0x00007f2598014000 nid=0x45be runnable
"Gang worker#2 (Parallel GC Threads)" prio=10 tid=0x00007f2598015800 nid=0x45bf runnable
"Gang worker#3 (Parallel GC Threads)" prio=10 tid=0x00007f2598017800 nid=0x45c0 runnable
"Concurrent Mark-Sweep GC Thread" prio=10 tid=0x00007f25980a6800 nid=0x45c1 runnable
"VM Periodic Task Thread" prio=10 tid=0x00007f25982e3000 nid=0x45cd waiting on condition
所有的這對於剛啓動的服務器來說似乎非常繁忙,沒有複製撒尿rs,並且還沒有接受任何請求。
想法?
所有這些線程都處於空閒狀態(被阻止)。我沒有看到任何原因... 也許有本地的東西......我也會檢查4月份的設置。 我在HornetQ上看不到任何東西在BTW –
謝謝斯蒂芬,使用你的一些建議加上jvisualvm,我已經能夠將問題縮小到Netty/HornetQ。看來HornetQ正在吃掉CPU,至少我知道現在在哪裏看。 – Nobody
@Clebert使用jvisualvm,我能夠看到有2個NIO工作線程正在吃掉CPU核心。他們追溯到Netty/HornetQ。所以我決定把我所有的用戶數量減少到少數,CPU使用率隨之下降。問題是,這些消費者實際上並沒有做任何事情,只能檢查空的隊列。你有什麼建議可以解決這個問題嗎? – Nobody