You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by "shawn.du" <sh...@neulion.com.cn> on 2018/06/11 02:37:19 UTC
ignite server cpu is high
RE: ignite server cpu is high
Posted by Stanislav Lukyanov <st...@gmail.com>.
How do you monitor your CPU usage? Do you know which processes consume CPU? Are you sure it is Ignite’s process? Is CPU consumed more in user space or in system space? Can you share the actual stats?
From what I see in the thread dump, there is at least some activity on this Ignite: sys-stripe-5-#6 thread is processing an update.
In any case, that thread is the only one in the JVM that is actually performing some work, so I’d assume that the CPU load comes from other processes.
Thanks,
Stan
From: shawn.du
Sent: 11 июня 2018 г. 5:38
To: user
Subject: ignite server cpu is high
Hi Community,
My single-node ignite cluster has started since Apr 23. In past couple weeks, It worked fine.
For our case, Ignite server's CPU is very low at most time with exception that there are complex/concurrent queries.
event in case of query, ignite server's CPU will be high for a very short while. I think all above is normal behavior.
But yesterday, ignite server's CPU keep high for a long time event we don't have query.
I use jstack to dump the threads(see the attachment), We don't find any our business code.
please help and thanks in advance.
We use ignite 2.3.0.
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
Thanks
Shawn
RE: ignite server cpu is high
Posted by Stanislav Lukyanov <st...@gmail.com>.
These threads are not actually running (i.e. are not consuming CPU) – they are blocked on the socket IO.
This SO question explains why they’re RUNNABLE https://stackoverflow.com/questions/20795295/why-jstack-out-says-thread-state-is-runnable-while-socketread.
> if disk IO is very slow, it will make CPU busy?
Generally, no, not in modern systems.
Another SO question explains this https://stackoverflow.com/questions/13596997/why-is-the-cpu-not-needed-to-service-i-o-requests.
Stan
From: shawn.du
Sent: 12 июня 2018 г. 6:11
To: user
Subject: Re: ignite server cpu is high
Hi,
I see most running threads are doing below code:
"tcp-disco-sock-reader-#469" #82488 prio=10 os_prio=0 tid=0x00007febf8308000 nid=0x3e8f runnable [0x00007feb948ef000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:171)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
- locked <0x00000005cee474e0> (a java.io.BufferedInputStream)
at org.apache.ignite.marshaller.jdk.JdkMarshallerInputStreamWrapper.read(JdkMarshallerInputStreamWrapper.java:53)
at java.io.ObjectInputStream$PeekInputStream.read(ObjectInputStream.java:2657)
at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2673)
at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:3150)
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:859)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:355)
at org.apache.ignite.marshaller.jdk.JdkMarshallerObjectInputStream.<init>(JdkMarshallerObjectInputStream.java:39)
at org.apache.ignite.marshaller.jdk.JdkMarshaller.unmarshal0(JdkMarshaller.java:119)
at org.apache.ignite.marshaller.AbstractNodeNameAwareMarshaller.unmarshal(AbstractNodeNameAwareMarshaller.java:94)
at org.apache.ignite.internal.util.IgniteUtils.unmarshal(IgniteUtils.java:9740)
at org.apache.ignite.spi.discovery.tcp.ServerImpl$SocketReader.body(ServerImpl.java:5946)
at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:62)
Our data into ignite are big(at most several Mega bytes.) binary data like below:
class Data
{
byte[] data; ....
}
one question:
if disk IO is very slow, it will make CPU busy?
Thanks
Shawn
On 6/12/2018 00:01，Stanislav Lukyanov<st...@gmail.com> wrote：
Sorry, but there isn’t much else to be said without additional data.
If consistent usage of resources on that host is important, I suggest to setup some monitoring so that if it happens again there’ll be at least some place to start.
Stan
From: shawn.du
Sent: 11 июня 2018 г. 16:34
To: user@ignite.apache.org
Cc: user@ignite.apache.org
Subject: Re: ignite server cpu is high
the server is a dedicate ignite server。i am sure it is ignite consume cpu. I can’t get more info now, the cpu issue is gone as now there are very few traffic on our system.
For our case,we store data into ignite every 30 seconds. If no query, the cpu should be very low. But yesterday it is a exception. Very strange.
shawn.du
邮箱：shawn.du@neulion.com.cn
Signature is customized by Netease Mail Master
On 06/11/2018 17:43, Stanislav Lukyanov wrote:
How do you monitor your CPU usage? Do you know which processes consume CPU? Are you sure it is Ignite’s process? Is CPU consumed more in user space or in system space? Can you share the actual stats?
From what I see in the thread dump, there is at least some activity on this Ignite: sys-stripe-5-#6 thread is processing an update.
In any case, that thread is the only one in the JVM that is actually performing some work, so I’d assume that the CPU load comes from other processes.
Thanks,
Stan
From: shawn.du
Sent: 11 июня 2018 г. 5:38
To: user
Subject: ignite server cpu is high
Hi Community,
My single-node ignite cluster has started since Apr 23. In past couple weeks, It worked fine.
For our case, Ignite server's CPU is very low at most time with exception that there are complex/concurrent queries.
event in case of query, ignite server's CPU will be high for a very short while. I think all above is normal behavior.
But yesterday, ignite server's CPU keep high for a long time event we don't have query.
I use jstack to dump the threads(see the attachment), We don't find any our business code.
please help and thanks in advance.
We use ignite 2.3.0.
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
Thanks
Shawn
RE: ignite server cpu is high
Posted by Stanislav Lukyanov <st...@gmail.com>.
Sorry, but there isn’t much else to be said without additional data.
If consistent usage of resources on that host is important, I suggest to setup some monitoring so that if it happens again there’ll be at least some place to start.
Stan
From: shawn.du
Sent: 11 июня 2018 г. 16:34
To: user@ignite.apache.org
Cc: user@ignite.apache.org
Subject: Re: ignite server cpu is high
the server is a dedicate ignite server。i am sure it is ignite consume cpu. I can’t get more info now, the cpu issue is gone as now there are very few traffic on our system.
For our case,we store data into ignite every 30 seconds. If no query, the cpu should be very low. But yesterday it is a exception. Very strange.
shawn.du
邮箱：shawn.du@neulion.com.cn
Signature is customized by Netease Mail Master
On 06/11/2018 17:43, Stanislav Lukyanov wrote:
How do you monitor your CPU usage? Do you know which processes consume CPU? Are you sure it is Ignite’s process? Is CPU consumed more in user space or in system space? Can you share the actual stats?
From what I see in the thread dump, there is at least some activity on this Ignite: sys-stripe-5-#6 thread is processing an update.
In any case, that thread is the only one in the JVM that is actually performing some work, so I’d assume that the CPU load comes from other processes.
Thanks,
Stan
From: shawn.du
Sent: 11 июня 2018 г. 5:38
To: user
Subject: ignite server cpu is high
Hi Community,
My single-node ignite cluster has started since Apr 23. In past couple weeks, It worked fine.
For our case, Ignite server's CPU is very low at most time with exception that there are complex/concurrent queries.
event in case of query, ignite server's CPU will be high for a very short while. I think all above is normal behavior.
But yesterday, ignite server's CPU keep high for a long time event we don't have query.
I use jstack to dump the threads(see the attachment), We don't find any our business code.
please help and thanks in advance.
We use ignite 2.3.0.
java version "1.8.0_151"
Java(TM) SE Runtime Environment (build 1.8.0_151-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.151-b12, mixed mode)
Thanks
Shawn