You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@ignite.apache.org by Ropugg <re...@gmail.com> on 2019/05/21 01:14:59 UTC
The CPU of Ignite NODEs in the same cluster has 20% gap
We have a cluster has 20 nodes.
18 nodes work as http servers, 2 nodes work as cache servers.
The 18 nodes query data from the 2 cache servers via IgniteService and
IgniteCache.
But on the load testing, the CPU of 2 cache severs has 20% gap.
<http://apache-ignite-users.70518.x6.nabble.com/file/t2330/cache_sever_cpu_gap.png>
I added a counter in my code, it shows the services access was balanced, but
the cpu wasn't.
<http://apache-ignite-users.70518.x6.nabble.com/file/t2330/two_cpu_compairson.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/t2330/cpu_comparison_2.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/t2330/cpu_comparison_hot_spots.png>
Here is the cpu snapshot, you can find more details via ./jvisualvm.
ignite01w-snap.nps
<http://apache-ignite-users.70518.x6.nabble.com/file/t2330/ignite01w-snap.nps>
ignite02w-snap.nps
<http://apache-ignite-users.70518.x6.nabble.com/file/t2330/ignite02w-snap.nps>
The Http servers were not balance either.
Do we know why they are not balance?
Is there anyway to resolve the unbalance issue?
All ignite cache are replicated on the two cache servers.
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The CPU of Ignite NODEs in the same cluster has 20% gap
Posted by Ropugg <re...@gmail.com>.
6 CPU
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
stepping : 2
microcode : 1808
cpu MHz : 2700.000
cache size : 20480 KB
physical id : 10
siblings : 1
core id : 0
cpu cores : 1
apicid : 10
initial apicid : 10
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc
arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc unfair_spinlock
eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave
avx hypervisor lahf_lm arat pti retpoline
bogomips : 5400.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
-----------------------------
Linux version 2.6.32-754.11.1.el6.x86_64 (mockbuild@x86-01.bsys.centos.org)
(gcc version 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC) ) #1 SMP Tue Feb 26
15:38:56 UTC 2019
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The CPU of Ignite NODEs in the same cluster has 20% gap
Posted by wbyeh <wb...@hotmail.com>.
Hi Ropugg,
If these 2 machines' spec are the same, please check whether they are
running at the same performance mode (for Linux version 2.6.32).
Also, if they are HTTP servers, please check whether the requests are
balanced.
Typically, sar command will help.
(sar -w; sar -u; sar -q ; sar -n DEV etc)
Via ignitevisorcmd.sh , we may find the cache internal statistics (more
detailed when the cache's property "statisticsEnabled" is 'true').
If we have found the different statistics from above results, we may go to
the next steps.
It has been found that the code has differences on filter & Hashtable (if
monitor synchronized) that will make the unbalanced.
<http://apache-ignite-users.70518.x6.nabble.com/file/t645/question1.png>
As my previous suggestions from other veteran, using Java Flight recording
is helpful.
If we have other findings, we may study here.
-WENPIN
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The CPU of Ignite NODEs in the same cluster has 20% gap
Posted by wbyeh <wb...@hotmail.com>.
Hi Ropugg,
If these 2 machines' spec are the same, please check whether they are
running at the same performance mode (for Linux version 2.6.32).
Also, if they are HTTP servers, please check whether the requests are
balanced.
Typically, sar command will help.
(sar -w; sar -u; sar -q ; sar -n DEV etc)
Via ignitevisorcmd.sh , we may find the cache internal statistics (more
detailed when the cache's property "statisticsEnabled" is 'true').
If we have found the different statistics from above results, we may go to
the next steps.
It has been found that the code has differences on filter & Hashtable (if
monitor synchronized) that will make the unbalanced.
<http://apache-ignite-users.70518.x6.nabble.com/file/t645/question1.png>
As my previous suggestions from other veteran, using Java Flight recording
is helpful.
If we have other findings, we may study here.
-WENPIN
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The CPU of Ignite NODEs in the same cluster has 20% gap
Posted by Ropugg <re...@gmail.com>.
6 CPU
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 45
model name : Intel(R) Xeon(R) CPU E5-2680 0 @ 2.70GHz
stepping : 2
microcode : 1808
cpu MHz : 2700.000
cache size : 20480 KB
physical id : 10
siblings : 1
core id : 0
cpu cores : 1
apicid : 10
initial apicid : 10
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat
pse36 clflush dts mmx fxsr sse sse2 ss syscall nx rdtscp lm constant_tsc
arch_perfmon pebs bts xtopology tsc_reliable nonstop_tsc unfair_spinlock
eagerfpu pni pclmulqdq ssse3 cx16 pcid sse4_1 sse4_2 x2apic popcnt aes xsave
avx hypervisor lahf_lm arat pti retpoline
bogomips : 5400.00
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:
----------
Linux version 2.6.32-754.11.1.el6.x86_64 (mockbuild@x86-01.bsys.centos.org)
(gcc version 4.4.7 20120313 (Red Hat 4.4.7-23) (GCC) ) #1 SMP Tue Feb 26
15:38:56 UTC 2019
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The CPU of Ignite NODEs in the same cluster has 20% gap
Posted by wbyeh <wb...@hotmail.com>.
May we have a Linux '/proc/cpuinfo' ?
and what OS/kernel version?
-WENPIN
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/
Re: The CPU of Ignite NODEs in the same cluster has 20% gap
Posted by Ropugg <re...@gmail.com>.
Update the jvisualvm screenshot.
<http://apache-ignite-users.70518.x6.nabble.com/file/t2330/two_cpu_compairson.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/t2330/cpu_comparison_2.png>
<http://apache-ignite-users.70518.x6.nabble.com/file/t2330/cpu_comparison_hot_spots.png>
--
Sent from: http://apache-ignite-users.70518.x6.nabble.com/