LeaderSelector running away with CPU

["Cantrell, Curtis" <Cu...@bkfs.com>]

We have 5 servers in the ensemble with a LeaderSelector in each server.    In our performance environment, we are noticing that the server that takes leadership causes CPU utilization to run up.  Test yesterday say CPU utilization at 95%.  Today was better at 33%.  The odd thing is that even after the test are completed, the CPU utilization does not go down.

We determined what thread was causing the high CPU utilization with

ps -C java -L -o pid,tid,pcpu,state,stime,time,nlwp

The TID is the thread id.   We then converted the TID to hex and searched the javacore thread dump to see what was causing the CPU to hang.  Every time the consuming thread was running the LeaderSelector.

3XMTHREADINFO      "LeaderSelector-2" J9VMThread:0x00000000C54D8800, j9thread_t:0x00007F8C51E5CC80, java/lang/Thread:0x0000000006FAB4D8, state:CW, prio=5
3XMJAVALTHREAD            (java/lang/Thread getId:0x3A8, isDaemon:true)
3XMTHREADINFO1            (native thread ID:0x2BDE, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO2            (native stack address range from:0x00007F8BD298F000, to:0x00007F8BD29D0000, size:0x41000)
3XMHEAPALLOC             Heap bytes allocated since last GC cycle=0 (0x0)
3XMTHREADINFO3           Java callstack:
4XESTACKTRACE                at com/lps/mg/inlet/sadevents/Leader.takeLeadership(Leader.java:127(Compiled Code))
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector$WrappedListener.takeLeadership(LeaderSelector.java:536)
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector.doWork(LeaderSelector.java:391)
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector.doWorkLoop(LeaderSelector.java:436)
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector.access$100(LeaderSelector.java:64)
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector$2.call(LeaderSelector.java:241)
4XESTACKTRACE                at org/apache/curator/framework/recipes/leader/LeaderSelector$2.call(LeaderSelector.java:235)
4XESTACKTRACE                at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314)
4XESTACKTRACE                at java/util/concurrent/FutureTask.run(FutureTask.java:149)
4XESTACKTRACE                at java/util/concurrent/Executors$RunnableAdapter.call(Executors.java:450)
4XESTACKTRACE                at java/util/concurrent/FutureTask$Sync.innerRun(FutureTask.java:314)
4XESTACKTRACE                at java/util/concurrent/FutureTask.run(FutureTask.java:149)
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:908)
4XESTACKTRACE                at java/util/concurrent/ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:931)
4XESTACKTRACE                at java/lang/Thread.run(Thread.java:773(Compiled Code))


Is this a reason that the LeaderSelector would run away with the CPU?

Thank you,
Curtis Cantrell




The information contained in this message is proprietary and/or confidential. If you are not the intended recipient, please: (i) delete the message and all copies; (ii) do not disclose, distribute or use the message in any manner; and (iii) notify the sender immediately. In addition, please be aware that any message addressed to our domain is subject to archiving and review by persons other than the intended recipient. Thank you.