You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Lukas Weiss <Lu...@raiffeisen.it> on 2019/02/27 10:04:00 UTC
High CPU usage with Solr 7.7.0
Hello,
we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
have problems with the server's CPU usage.
We have two Solr cores configured, but even if we clear all indexes and do
not start the index process, we see 100 CPU usage for both cores.
Here's what our top says:
root@solr:~ # top
top - 09:25:24 up 17:40, 1 user, load average: 2,28, 2,56, 2,68
Threads: 74 total, 3 running, 71 sleeping, 0 stopped, 0 zombie
%Cpu0 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
0,0 st
%Cpu1 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
0,0 st
%Cpu2 : 11,3 us, 1,0 sy, 0,0 ni, 86,7 id, 0,7 wa, 0,0 hi, 0,3 si,
0,0 st
%Cpu3 : 3,0 us, 3,0 sy, 0,0 ni, 93,7 id, 0,3 wa, 0,0 hi, 0,0 si,
0,0 st
KiB Mem : 8388608 total, 7859168 free, 496744 used, 32696
buff/cache
KiB Swap: 2097152 total, 2097152 free, 0 used. 7859168 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
P
10209 solr 20 0 6138468 452520 25740 R 99,9 5,4 29:43.45 java
-server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
10214 solr 20 0 6138468 452520 25740 R 99,9 5,4 28:42.91 java
-server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux LXC
dedicated Container.
Some more server info:
root@solr:~ # java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
root@solr:~ # free -m
total used free shared buff/cache
available
Mem: 8192 484 7675 701 31 7675
Swap: 2048 0 2048
We also found something strange if we do an strace of the main process, we
get lots of ongoing connection timeouts:
root@solr:~ # strace -F -p 4136
strace: Process 4136 attached with 48 threads
strace: [ Process PID=11089 runs in x32 mode. ]
[pid 4937] epoll_wait(139, <unfinished ...>
[pid 4936] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4909] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4618] epoll_wait(136, <unfinished ...>
[pid 4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
...>
[pid 4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
<unfinished ...>
[pid 4244] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
<unfinished ...>
[pid 4243] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
<unfinished ...>
[pid 4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL <unfinished
...>
[pid 4205] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4204] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4196] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4195] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4194] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4193] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4187] restart_syscall(<... resuming interrupted restart_syscall ...>
<unfinished ...>
[pid 4180] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4179] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4177] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4174] accept(133, <unfinished ...>
[pid 4173] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4172] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4171] restart_syscall(<... resuming interrupted restart_syscall ...>
<unfinished ...>
[pid 4165] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
...>
[pid 4163] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4162] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4161] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4160] futex(0x7ff623d52c20,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
<unfinished ...>
[pid 4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid 4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL <unfinished
...>
[pid 4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL <unfinished
...>
[pid 4156] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4155] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid 4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
...>
[pid 4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid 4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid 4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
<unfinished ...>
[pid 4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
[pid 4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
<unfinished ...>
[pid 4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
<unfinished ...>
[pid 4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
<unfinished ...>
[pid 4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
...>
[pid 4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished ...>
[pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT (Connection
timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
[pid 4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
[pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
detached
strace: Process 4144 detached
strace: Process 4145 detached
strace: Process 4146 detached
strace: Process 4147 detached
strace: Process 4148 detached
strace: Process 4150 detached
strace: Process 4151 detached
strace: Process 4152 detached
strace: Process 4153 detached
....
Could you help us to determine what's wrong with our setup?
Thank you very much,
Kind regards
Lukas Weiss
Antwort: Re: Re: High CPU usage with Solr 7.7.0
Posted by Lukas Weiss <Lu...@raiffeisen.it>.
This is the information of the Thread Dump screen of the Solr web
interface:
process reaper (8195)
java.util.concurrent.SynchronousQueue$TransferStack@23ec2c53
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:460)
java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362)
java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:941)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.8959ms
0.0000ms
commitScheduler-14-thread-35 (8174)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
644010.9423ms
643930.0000ms
commitScheduler-16-thread-62 (8173)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.poll(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
644831.4905ms
644740.0000ms
qtp1282287470-8051 (8051)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d2bd65e
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
java.lang.Thread.run(Thread.java:748)
14.5521ms
10.0000ms
qtp1282287470-8050 (8050)
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
java.lang.Thread.run(Thread.java:748)
10.8397ms
10.0000ms
qtp1282287470-8049 (8049)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4a97a2f0
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
java.lang.Thread.run(Thread.java:748)
2.4610ms
0.0000ms
qtp1282287470-8047 (8047)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d2bd65e
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
java.lang.Thread.run(Thread.java:748)
8.3763ms
0.0000ms
Scheduler-235237152 (136)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@6c16dc65
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
0.7083ms
0.0000ms
SolrRrdBackendFactory-7-thread-2 (41)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2edafb00
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
59.4289ms
50.0000ms
autoExpireDocs-19-thread-1 (40)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@5db051a3
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
2600.6087ms
2200.0000ms
autoExpireDocs-18-thread-1 (39)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@56962869
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
3731.4479ms
3730.0000ms
searcherExecutor-10-thread-1 (38)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@c39232a
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
111.4577ms
110.0000ms
searcherExecutor-11-thread-1 (37)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@7542db56
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
94.7639ms
90.0000ms
DestroyJavaVM (34)
1409.1001ms
1350.0000ms
Thread-13 (32)
java.lang.Object@2aa0a114
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
org.apache.solr.core.CloserThread.run(CoreContainer.java:1901)
0.1739ms
0.0000ms
MetricsHistoryHandler-8-thread-1 (29)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@4d920149
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
5333.8818ms
4450.0000ms
SolrRrdBackendFactory-7-thread-1 (28)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2edafb00
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1088)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
205.1954ms
90.0000ms
Connection evictor (27)
java.lang.Thread.sleep(Native Method)
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
java.lang.Thread.run(Thread.java:748)
144.2396ms
110.0000ms
Connection evictor (26)
java.lang.Thread.sleep(Native Method)
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
java.lang.Thread.run(Thread.java:748)
122.9604ms
120.0000ms
Connection evictor (25)
java.lang.Thread.sleep(Native Method)
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
java.lang.Thread.run(Thread.java:748)
117.9532ms
110.0000ms
Connection evictor (24)
java.lang.Thread.sleep(Native Method)
org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
java.lang.Thread.run(Thread.java:748)
147.2869ms
120.0000ms
Scheduler-2044366277 (23)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@8775ba2
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
java.lang.Thread.run(Thread.java:748)
46.1530ms
40.0000ms
qtp1282287470-22 (22)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@439b00fa
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2163)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.reservedWait(ReservedThreadExecutor.java:292)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:357)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
java.lang.Thread.run(Thread.java:748)
59.4703ms
50.0000ms
qtp1282287470-20 (20)
sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:423)
org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:360)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.produceTask(EatWhatYouKill.java:357)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:181)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
java.lang.Thread.run(Thread.java:748)
117.0453ms
110.0000ms
qtp1282287470-19 (19)
sun.management.OperatingSystemImpl.getSystemCpuLoad(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
java.lang.reflect.Method.invoke(Method.java:498)
org.apache.solr.util.stats.MetricUtils.addMXBeanMetrics(MetricUtils.java:587)
org.apache.solr.util.stats.MetricUtils.addMXBeanMetrics(MetricUtils.java:634)
org.apache.solr.handler.admin.SystemInfoHandler.getSystemInfo(SystemInfoHandler.java:213)
org.apache.solr.handler.admin.SystemInfoHandler.handleRequestBody(SystemInfoHandler.java:149)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
org.apache.solr.handler.admin.InfoHandler.handle(InfoHandler.java:91)
org.apache.solr.handler.admin.InfoHandler.handleRequestBody(InfoHandler.java:81)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
org.eclipse.jetty.server.Server.handle(Server.java:502)
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
java.lang.Thread.run(Thread.java:748)
58.4642ms
20.0000ms
qtp1282287470-16-acceptor-0@28e7f409-ServerConnector@26200f65{HTTP/1.1,[http/1.1]}{0.0.0.0:8983}
(16)
sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:385)
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:648)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
java.lang.Thread.run(Thread.java:748)
4.6343ms
0.0000ms
qtp1282287470-15 (15)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@2d2bd65e
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:656)
org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:46)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:720)
java.lang.Thread.run(Thread.java:748)
80.6062ms
70.0000ms
qtp1282287470-14 (14)
sun.management.ThreadImpl.getThreadInfo1(Native Method)
sun.management.ThreadImpl.getThreadInfo(ThreadImpl.java:176)
org.apache.solr.handler.admin.ThreadDumpHandler.handleRequestBody(ThreadDumpHandler.java:71)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
org.apache.solr.handler.admin.InfoHandler.handle(InfoHandler.java:91)
org.apache.solr.handler.admin.InfoHandler.handleRequestBody(InfoHandler.java:81)
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:199)
org.apache.solr.servlet.HttpSolrCall.handleAdmin(HttpSolrCall.java:735)
org.apache.solr.servlet.HttpSolrCall.handleAdminRequest(HttpSolrCall.java:716)
org.apache.solr.servlet.HttpSolrCall.call(HttpSolrCall.java:496)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:395)
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:341)
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1602)
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:548)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1588)
org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1557)
org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:220)
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:126)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
org.eclipse.jetty.rewrite.handler.RewriteHandler.handle(RewriteHandler.java:335)
org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
org.eclipse.jetty.server.Server.handle(Server.java:502)
org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:364)
org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:765)
org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:683)
java.lang.Thread.run(Thread.java:748)
75.6621ms
20.0000ms
ShutdownMonitor (12)
java.net.PlainSocketImpl.socketAccept(Native Method)
java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:409)
java.net.ServerSocket.implAccept(ServerSocket.java:545)
java.net.ServerSocket.accept(ServerSocket.java:513)
org.eclipse.jetty.server.ShutdownMonitor$ShutdownMonitorRunnable.run(ShutdownMonitor.java:335)
java.lang.Thread.run(Thread.java:748)
0.3767ms
0.0000ms
Signal Dispatcher (5)
0.0362ms
0.0000ms
Finalizer (3)
java.lang.ref.ReferenceQueue$Lock@448b0df5
java.lang.Object.wait(Native Method)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
8.2488ms
0.0000ms
Reference Handler (2)
java.lang.ref.Reference$Lock@19ced464
java.lang.Object.wait(Native Method)
java.lang.Object.wait(Object.java:502)
java.lang.ref.Reference.tryHandlePending(Reference.java:191)
java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
Von: "Tomás Fernández Löbbe" <to...@gmail.com>
An: solr-user@lucene.apache.org,
Datum: 27.02.2019 19:34
Betreff: Re: Re: High CPU usage with Solr 7.7.0
Maybe a thread dump would be useful if you still have some instance
running
on 7.7
On Wed, Feb 27, 2019 at 7:28 AM Lukas Weiss <Lu...@raiffeisen.it>
wrote:
> I can confirm this. Downgrading to 7.6.0 solved the issue.
> Thanks for the hint.
>
>
>
> Von: "Joe Obernberger" <jo...@gmail.com>
> An: solr-user@lucene.apache.org, "Lukas Weiss"
> <Lu...@raiffeisen.it>,
> Datum: 27.02.2019 15:59
> Betreff: Re: High CPU usage with Solr 7.7.0
>
>
>
> Just to add to this. We upgraded to 7.7.0 and saw very large CPU usage
> on multi core boxes - sustained in the 1200% range. We then switched to
> 7.6.0 (no other configuration changes) and the problem went away.
>
> We have a 40 node cluster and all 40 nodes had high CPU usage with 3
> indexes stored on HDFS.
>
> -Joe
>
> On 2/27/2019 5:04 AM, Lukas Weiss wrote:
> > Hello,
> >
> > we recently updated our Solr server from 6.6.5 to 7.7.0. Since then,
we
> > have problems with the server's CPU usage.
> > We have two Solr cores configured, but even if we clear all indexes
and
> do
> > not start the index process, we see 100 CPU usage for both cores.
> >
> > Here's what our top says:
> >
> > root@solr:~ # top
> > top - 09:25:24 up 17:40, 1 user, load average: 2,28, 2,56, 2,68
> > Threads: 74 total, 3 running, 71 sleeping, 0 stopped, 0 zombie
> > %Cpu0 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0
si,
> > 0,0 st
> > %Cpu1 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0
si,
> > 0,0 st
> > %Cpu2 : 11,3 us, 1,0 sy, 0,0 ni, 86,7 id, 0,7 wa, 0,0 hi, 0,3
si,
> > 0,0 st
> > %Cpu3 : 3,0 us, 3,0 sy, 0,0 ni, 93,7 id, 0,3 wa, 0,0 hi, 0,0
si,
> > 0,0 st
> > KiB Mem : 8388608 total, 7859168 free, 496744 used, 32696
> > buff/cache
> > KiB Swap: 2097152 total, 2097152 free, 0 used. 7859168 avail
> Mem
> >
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> > P
> > 10209 solr 20 0 6138468 452520 25740 R 99,9 5,4 29:43.45
java
> > -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
> > 10214 solr 20 0 6138468 452520 25740 R 99,9 5,4 28:42.91
java
> > -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
> >
> > The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux
> LXC
> > dedicated Container.
> >
> > Some more server info:
> >
> > root@solr:~ # java -version
> > openjdk version "1.8.0_181"
> > OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
> > OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
> >
> > root@solr:~ # free -m
> > total used free shared buff/cache
> > available
> > Mem: 8192 484 7675 701 31
7675
> > Swap: 2048 0 2048
> >
> > We also found something strange if we do an strace of the main
process,
> we
> > get lots of ongoing connection timeouts:
> >
> > root@solr:~ # strace -F -p 4136
> > strace: Process 4136 attached with 48 threads
> > strace: [ Process PID=11089 runs in x32 mode. ]
> > [pid 4937] epoll_wait(139, <unfinished ...>
> > [pid 4936] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4909] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4618] epoll_wait(136, <unfinished ...>
> > [pid 4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL
> <unfinished
> > ...>
> > [pid 4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
> > <unfinished ...>
> > [pid 4244] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
> > <unfinished ...>
> > [pid 4243] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
> > <unfinished ...>
> > [pid 4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL
> <unfinished
> > ...>
> > [pid 4205] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4204] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4196] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4195] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4194] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4193] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4187] restart_syscall(<... resuming interrupted restart_syscall
> ...>
> > <unfinished ...>
> > [pid 4180] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4179] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4177] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4174] accept(133, <unfinished ...>
> > [pid 4173] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4172] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4171] restart_syscall(<... resuming interrupted restart_syscall
> ...>
> > <unfinished ...>
> > [pid 4165] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL
> <unfinished
> > ...>
> > [pid 4163] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4162] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4161] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4160] futex(0x7ff623d52c20,
> > FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
> > <unfinished ...>
> > [pid 4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid 4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL
> <unfinished
> > ...>
> > [pid 4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL
> <unfinished
> > ...>
> > [pid 4156] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4155] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid 4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL
> <unfinished
> > ...>
> > [pid 4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid 4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid 4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
> > <unfinished ...>
> > [pid 4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> > timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
> > [pid 4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
> > <unfinished ...>
> > [pid 4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
> > <unfinished ...>
> > [pid 4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
> > <unfinished ...>
> > [pid 4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL
> <unfinished
> > ...>
> > [pid 4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished
> ...>
> > [pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection
timed
> > out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT
> (Connection
> > timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
> > [pid 4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> > timed out)
> > [pid 4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
> > [pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection
timed
> > out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
> > detached
> > strace: Process 4144 detached
> > strace: Process 4145 detached
> > strace: Process 4146 detached
> > strace: Process 4147 detached
> > strace: Process 4148 detached
> > strace: Process 4150 detached
> > strace: Process 4151 detached
> > strace: Process 4152 detached
> > strace: Process 4153 detached
> > ....
> >
> >
> > Could you help us to determine what's wrong with our setup?
> >
> > Thank you very much,
> >
> > Kind regards
> > Lukas Weiss
> >
> > ---
> > This email has been checked for viruses by AVG.
> > https://www.avg.com
> >
>
>
Re: Re: High CPU usage with Solr 7.7.0
Posted by Tomás Fernández Löbbe <to...@gmail.com>.
Maybe a thread dump would be useful if you still have some instance running
on 7.7
On Wed, Feb 27, 2019 at 7:28 AM Lukas Weiss <Lu...@raiffeisen.it>
wrote:
> I can confirm this. Downgrading to 7.6.0 solved the issue.
> Thanks for the hint.
>
>
>
> Von: "Joe Obernberger" <jo...@gmail.com>
> An: solr-user@lucene.apache.org, "Lukas Weiss"
> <Lu...@raiffeisen.it>,
> Datum: 27.02.2019 15:59
> Betreff: Re: High CPU usage with Solr 7.7.0
>
>
>
> Just to add to this. We upgraded to 7.7.0 and saw very large CPU usage
> on multi core boxes - sustained in the 1200% range. We then switched to
> 7.6.0 (no other configuration changes) and the problem went away.
>
> We have a 40 node cluster and all 40 nodes had high CPU usage with 3
> indexes stored on HDFS.
>
> -Joe
>
> On 2/27/2019 5:04 AM, Lukas Weiss wrote:
> > Hello,
> >
> > we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
> > have problems with the server's CPU usage.
> > We have two Solr cores configured, but even if we clear all indexes and
> do
> > not start the index process, we see 100 CPU usage for both cores.
> >
> > Here's what our top says:
> >
> > root@solr:~ # top
> > top - 09:25:24 up 17:40, 1 user, load average: 2,28, 2,56, 2,68
> > Threads: 74 total, 3 running, 71 sleeping, 0 stopped, 0 zombie
> > %Cpu0 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
> > 0,0 st
> > %Cpu1 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
> > 0,0 st
> > %Cpu2 : 11,3 us, 1,0 sy, 0,0 ni, 86,7 id, 0,7 wa, 0,0 hi, 0,3 si,
> > 0,0 st
> > %Cpu3 : 3,0 us, 3,0 sy, 0,0 ni, 93,7 id, 0,3 wa, 0,0 hi, 0,0 si,
> > 0,0 st
> > KiB Mem : 8388608 total, 7859168 free, 496744 used, 32696
> > buff/cache
> > KiB Swap: 2097152 total, 2097152 free, 0 used. 7859168 avail
> Mem
> >
> >
> > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
> COMMAND
> > P
> > 10209 solr 20 0 6138468 452520 25740 R 99,9 5,4 29:43.45 java
> > -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
> > 10214 solr 20 0 6138468 452520 25740 R 99,9 5,4 28:42.91 java
> > -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> > -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> > -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
> >
> > The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux
> LXC
> > dedicated Container.
> >
> > Some more server info:
> >
> > root@solr:~ # java -version
> > openjdk version "1.8.0_181"
> > OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
> > OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
> >
> > root@solr:~ # free -m
> > total used free shared buff/cache
> > available
> > Mem: 8192 484 7675 701 31 7675
> > Swap: 2048 0 2048
> >
> > We also found something strange if we do an strace of the main process,
> we
> > get lots of ongoing connection timeouts:
> >
> > root@solr:~ # strace -F -p 4136
> > strace: Process 4136 attached with 48 threads
> > strace: [ Process PID=11089 runs in x32 mode. ]
> > [pid 4937] epoll_wait(139, <unfinished ...>
> > [pid 4936] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4909] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4618] epoll_wait(136, <unfinished ...>
> > [pid 4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL
> <unfinished
> > ...>
> > [pid 4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
> > <unfinished ...>
> > [pid 4244] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
> > <unfinished ...>
> > [pid 4243] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
> > <unfinished ...>
> > [pid 4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL
> <unfinished
> > ...>
> > [pid 4205] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4204] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4196] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4195] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4194] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4193] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4187] restart_syscall(<... resuming interrupted restart_syscall
> ...>
> > <unfinished ...>
> > [pid 4180] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4179] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4177] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4174] accept(133, <unfinished ...>
> > [pid 4173] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4172] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4171] restart_syscall(<... resuming interrupted restart_syscall
> ...>
> > <unfinished ...>
> > [pid 4165] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL
> <unfinished
> > ...>
> > [pid 4163] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4162] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4161] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4160] futex(0x7ff623d52c20,
> > FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
> > <unfinished ...>
> > [pid 4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid 4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL
> <unfinished
> > ...>
> > [pid 4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL
> <unfinished
> > ...>
> > [pid 4156] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4155] restart_syscall(<... resuming interrupted futex ...>
> > <unfinished ...>
> > [pid 4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid 4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL
> <unfinished
> > ...>
> > [pid 4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid 4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL
> <unfinished
> > ...>
> > [pid 4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
> > <unfinished ...>
> > [pid 4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> > timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
> > [pid 4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
> > <unfinished ...>
> > [pid 4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
> > <unfinished ...>
> > [pid 4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
> > <unfinished ...>
> > [pid 4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL
> <unfinished
> > ...>
> > [pid 4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished
> ...>
> > [pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
> > out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT
> (Connection
> > timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
> > [pid 4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> > timed out)
> > [pid 4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
> > [pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
> > out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
> > (Connection timed out)
> > [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> > [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> > {tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
> > detached
> > strace: Process 4144 detached
> > strace: Process 4145 detached
> > strace: Process 4146 detached
> > strace: Process 4147 detached
> > strace: Process 4148 detached
> > strace: Process 4150 detached
> > strace: Process 4151 detached
> > strace: Process 4152 detached
> > strace: Process 4153 detached
> > ....
> >
> >
> > Could you help us to determine what's wrong with our setup?
> >
> > Thank you very much,
> >
> > Kind regards
> > Lukas Weiss
> >
> > ---
> > This email has been checked for viruses by AVG.
> > https://www.avg.com
> >
>
>
Antwort: Re: High CPU usage with Solr 7.7.0
Posted by Lukas Weiss <Lu...@raiffeisen.it>.
I can confirm this. Downgrading to 7.6.0 solved the issue.
Thanks for the hint.
Von: "Joe Obernberger" <jo...@gmail.com>
An: solr-user@lucene.apache.org, "Lukas Weiss"
<Lu...@raiffeisen.it>,
Datum: 27.02.2019 15:59
Betreff: Re: High CPU usage with Solr 7.7.0
Just to add to this. We upgraded to 7.7.0 and saw very large CPU usage
on multi core boxes - sustained in the 1200% range. We then switched to
7.6.0 (no other configuration changes) and the problem went away.
We have a 40 node cluster and all 40 nodes had high CPU usage with 3
indexes stored on HDFS.
-Joe
On 2/27/2019 5:04 AM, Lukas Weiss wrote:
> Hello,
>
> we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
> have problems with the server's CPU usage.
> We have two Solr cores configured, but even if we clear all indexes and
do
> not start the index process, we see 100 CPU usage for both cores.
>
> Here's what our top says:
>
> root@solr:~ # top
> top - 09:25:24 up 17:40, 1 user, load average: 2,28, 2,56, 2,68
> Threads: 74 total, 3 running, 71 sleeping, 0 stopped, 0 zombie
> %Cpu0 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
> 0,0 st
> %Cpu1 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
> 0,0 st
> %Cpu2 : 11,3 us, 1,0 sy, 0,0 ni, 86,7 id, 0,7 wa, 0,0 hi, 0,3 si,
> 0,0 st
> %Cpu3 : 3,0 us, 3,0 sy, 0,0 ni, 93,7 id, 0,3 wa, 0,0 hi, 0,0 si,
> 0,0 st
> KiB Mem : 8388608 total, 7859168 free, 496744 used, 32696
> buff/cache
> KiB Swap: 2097152 total, 2097152 free, 0 used. 7859168 avail
Mem
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+
COMMAND
> P
> 10209 solr 20 0 6138468 452520 25740 R 99,9 5,4 29:43.45 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
> 10214 solr 20 0 6138468 452520 25740 R 99,9 5,4 28:42.91 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
>
> The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux
LXC
> dedicated Container.
>
> Some more server info:
>
> root@solr:~ # java -version
> openjdk version "1.8.0_181"
> OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
> OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
>
> root@solr:~ # free -m
> total used free shared buff/cache
> available
> Mem: 8192 484 7675 701 31 7675
> Swap: 2048 0 2048
>
> We also found something strange if we do an strace of the main process,
we
> get lots of ongoing connection timeouts:
>
> root@solr:~ # strace -F -p 4136
> strace: Process 4136 attached with 48 threads
> strace: [ Process PID=11089 runs in x32 mode. ]
> [pid 4937] epoll_wait(139, <unfinished ...>
> [pid 4936] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4909] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4618] epoll_wait(136, <unfinished ...>
> [pid 4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL
<unfinished
> ...>
> [pid 4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
> <unfinished ...>
> [pid 4244] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid 4243] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid 4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL
<unfinished
> ...>
> [pid 4205] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4204] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4196] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4195] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4194] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4193] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4187] restart_syscall(<... resuming interrupted restart_syscall
...>
> <unfinished ...>
> [pid 4180] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4179] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4177] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4174] accept(133, <unfinished ...>
> [pid 4173] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4172] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4171] restart_syscall(<... resuming interrupted restart_syscall
...>
> <unfinished ...>
> [pid 4165] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL
<unfinished
> ...>
> [pid 4163] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4162] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4161] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4160] futex(0x7ff623d52c20,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
> <unfinished ...>
> [pid 4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL
<unfinished
> ...>
> [pid 4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL
<unfinished
> ...>
> [pid 4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL
<unfinished
> ...>
> [pid 4156] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4155] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL
<unfinished
> ...>
> [pid 4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL
<unfinished
> ...>
> [pid 4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL
<unfinished
> ...>
> [pid 4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL
<unfinished
> ...>
> [pid 4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
> <unfinished ...>
> [pid 4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
> [pid 4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
> <unfinished ...>
> [pid 4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
> <unfinished ...>
> [pid 4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
> <unfinished ...>
> [pid 4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL
<unfinished
> ...>
> [pid 4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished
...>
> [pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT
(Connection
> timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
> [pid 4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid 4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
> [pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
> detached
> strace: Process 4144 detached
> strace: Process 4145 detached
> strace: Process 4146 detached
> strace: Process 4147 detached
> strace: Process 4148 detached
> strace: Process 4150 detached
> strace: Process 4151 detached
> strace: Process 4152 detached
> strace: Process 4153 detached
> ....
>
>
> Could you help us to determine what's wrong with our setup?
>
> Thank you very much,
>
> Kind regards
> Lukas Weiss
>
> ---
> This email has been checked for viruses by AVG.
> https://www.avg.com
>
Re: High CPU usage with Solr 7.7.0
Posted by Joe Obernberger <jo...@gmail.com>.
Just to add to this. We upgraded to 7.7.0 and saw very large CPU usage
on multi core boxes - sustained in the 1200% range. We then switched to
7.6.0 (no other configuration changes) and the problem went away.
We have a 40 node cluster and all 40 nodes had high CPU usage with 3
indexes stored on HDFS.
-Joe
On 2/27/2019 5:04 AM, Lukas Weiss wrote:
> Hello,
>
> we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
> have problems with the server's CPU usage.
> We have two Solr cores configured, but even if we clear all indexes and do
> not start the index process, we see 100 CPU usage for both cores.
>
> Here's what our top says:
>
> root@solr:~ # top
> top - 09:25:24 up 17:40, 1 user, load average: 2,28, 2,56, 2,68
> Threads: 74 total, 3 running, 71 sleeping, 0 stopped, 0 zombie
> %Cpu0 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
> 0,0 st
> %Cpu1 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
> 0,0 st
> %Cpu2 : 11,3 us, 1,0 sy, 0,0 ni, 86,7 id, 0,7 wa, 0,0 hi, 0,3 si,
> 0,0 st
> %Cpu3 : 3,0 us, 3,0 sy, 0,0 ni, 93,7 id, 0,3 wa, 0,0 hi, 0,0 si,
> 0,0 st
> KiB Mem : 8388608 total, 7859168 free, 496744 used, 32696
> buff/cache
> KiB Swap: 2097152 total, 2097152 free, 0 used. 7859168 avail Mem
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> P
> 10209 solr 20 0 6138468 452520 25740 R 99,9 5,4 29:43.45 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
> 10214 solr 20 0 6138468 452520 25740 R 99,9 5,4 28:42.91 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
>
> The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux LXC
> dedicated Container.
>
> Some more server info:
>
> root@solr:~ # java -version
> openjdk version "1.8.0_181"
> OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
> OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
>
> root@solr:~ # free -m
> total used free shared buff/cache
> available
> Mem: 8192 484 7675 701 31 7675
> Swap: 2048 0 2048
>
> We also found something strange if we do an strace of the main process, we
> get lots of ongoing connection timeouts:
>
> root@solr:~ # strace -F -p 4136
> strace: Process 4136 attached with 48 threads
> strace: [ Process PID=11089 runs in x32 mode. ]
> [pid 4937] epoll_wait(139, <unfinished ...>
> [pid 4936] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4909] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4618] epoll_wait(136, <unfinished ...>
> [pid 4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
> ...>
> [pid 4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
> <unfinished ...>
> [pid 4244] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid 4243] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid 4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL <unfinished
> ...>
> [pid 4205] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4204] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4196] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4195] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4194] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4193] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4187] restart_syscall(<... resuming interrupted restart_syscall ...>
> <unfinished ...>
> [pid 4180] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4179] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4177] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4174] accept(133, <unfinished ...>
> [pid 4173] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4172] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4171] restart_syscall(<... resuming interrupted restart_syscall ...>
> <unfinished ...>
> [pid 4165] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
> ...>
> [pid 4163] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4162] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4161] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4160] futex(0x7ff623d52c20,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
> <unfinished ...>
> [pid 4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid 4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL <unfinished
> ...>
> [pid 4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL <unfinished
> ...>
> [pid 4156] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4155] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid 4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
> ...>
> [pid 4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid 4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid 4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
> <unfinished ...>
> [pid 4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
> [pid 4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
> <unfinished ...>
> [pid 4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
> <unfinished ...>
> [pid 4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
> <unfinished ...>
> [pid 4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
> ...>
> [pid 4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished ...>
> [pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT (Connection
> timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
> [pid 4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid 4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
> [pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
> detached
> strace: Process 4144 detached
> strace: Process 4145 detached
> strace: Process 4146 detached
> strace: Process 4147 detached
> strace: Process 4148 detached
> strace: Process 4150 detached
> strace: Process 4151 detached
> strace: Process 4152 detached
> strace: Process 4153 detached
> ....
>
>
> Could you help us to determine what's wrong with our setup?
>
> Thank you very much,
>
> Kind regards
> Lukas Weiss
>
> ---
> This email has been checked for viruses by AVG.
> https://www.avg.com
>
Re: High CPU usage with Solr 7.7.0
Posted by Erick Erickson <er...@gmail.com>.
Thanks all. I pushed changes last night, this should be fixed in 7.7.2, 8.1 and master.
Meanwhile, this is a trivial change to one line, so two ways to get by would be
1> just make the change yourself locally. Building Solr from scratch is actually not hard. The “ant package” target will get you the same thing you’d get from downloading the distribution.
2> use Java 9 or greater.
Best,
Erick
> On Mar 25, 2019, at 1:58 AM, Lukas Weiss <Lu...@raiffeisen.it> wrote:
>
> I forward this message. Thanks Adam.
>
> Hi,
> Apologies, I can’t figure out how to reply to the Solr mailing list.
> I just ran across the same high CPU usage issue. I believe it’’s caused by
> this commit which was introduced in Solr 7.7.0
> https://github.com/apache/lucene-solr/commit/eb652b84edf441d8369f5188cdd5e3ae2b151434#diff-e54b251d166135a1afb7938cfe152bb5
> There is a bug in JDK versions <=8 where using 0 threads in the
> ScheduledThreadPool causes high CPU usage:
> https://bugs.openjdk.java.net/browse/JDK-8129861
> Oddly, the latest version
> of solr/core/src/java/org/apache/solr/update/CommitTracker.java on
> master still uses 0 executors as the default. Presumably most everyone is
> using JDK 9 or greater which has the bug fixed, so they don’t experience
> the bug.
> Feel free to relay this back to the mailing list.
> Thanks,
> Adam Guthrie
>
>
>
>
>
> Von: "Lukas Weiss" <Lu...@raiffeisen.it>
> An: solr-user@lucene.apache.org,
> Datum: 27.02.2019 11:13
> Betreff: High CPU usage with Solr 7.7.0
>
>
>
> Hello,
>
> we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
> have problems with the server's CPU usage.
> We have two Solr cores configured, but even if we clear all indexes and do
>
> not start the index process, we see 100 CPU usage for both cores.
>
> Here's what our top says:
>
> root@solr:~ # top
> top - 09:25:24 up 17:40, 1 user, load average: 2,28, 2,56, 2,68
> Threads: 74 total, 3 running, 71 sleeping, 0 stopped, 0 zombie
> %Cpu0 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
> 0,0 st
> %Cpu1 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
> 0,0 st
> %Cpu2 : 11,3 us, 1,0 sy, 0,0 ni, 86,7 id, 0,7 wa, 0,0 hi, 0,3 si,
> 0,0 st
> %Cpu3 : 3,0 us, 3,0 sy, 0,0 ni, 93,7 id, 0,3 wa, 0,0 hi, 0,0 si,
> 0,0 st
> KiB Mem : 8388608 total, 7859168 free, 496744 used, 32696
> buff/cache
> KiB Swap: 2097152 total, 2097152 free, 0 used. 7859168 avail Mem
>
>
>
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>
> P
> 10209 solr 20 0 6138468 452520 25740 R 99,9 5,4 29:43.45 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
> 10214 solr 20 0 6138468 452520 25740 R 99,9 5,4 28:42.91 java
> -server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
> -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
> -XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
>
> The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux LXC
> dedicated Container.
>
> Some more server info:
>
> root@solr:~ # java -version
> openjdk version "1.8.0_181"
> OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
> OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
>
> root@solr:~ # free -m
> total used free shared buff/cache
> available
> Mem: 8192 484 7675 701 31 7675
> Swap: 2048 0 2048
>
> We also found something strange if we do an strace of the main process, we
>
> get lots of ongoing connection timeouts:
>
> root@solr:~ # strace -F -p 4136
> strace: Process 4136 attached with 48 threads
> strace: [ Process PID=11089 runs in x32 mode. ]
> [pid 4937] epoll_wait(139, <unfinished ...>
> [pid 4936] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4909] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4618] epoll_wait(136, <unfinished ...>
> [pid 4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
> ...>
> [pid 4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
> <unfinished ...>
> [pid 4244] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid 4243] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
> <unfinished ...>
> [pid 4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL <unfinished
> ...>
> [pid 4205] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4204] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4196] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4195] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4194] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4193] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4187] restart_syscall(<... resuming interrupted restart_syscall ...>
>
> <unfinished ...>
> [pid 4180] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4179] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4177] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4174] accept(133, <unfinished ...>
> [pid 4173] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4172] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4171] restart_syscall(<... resuming interrupted restart_syscall ...>
>
> <unfinished ...>
> [pid 4165] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
> ...>
> [pid 4163] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4162] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4161] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4160] futex(0x7ff623d52c20,
> FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
> <unfinished ...>
> [pid 4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid 4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL <unfinished
>
> ...>
> [pid 4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL <unfinished
>
> ...>
> [pid 4156] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4155] restart_syscall(<... resuming interrupted futex ...>
> <unfinished ...>
> [pid 4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid 4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
> ...>
> [pid 4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid 4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
> ...>
> [pid 4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
> <unfinished ...>
> [pid 4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
> [pid 4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
> <unfinished ...>
> [pid 4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
> <unfinished ...>
> [pid 4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
> <unfinished ...>
> [pid 4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
> ...>
> [pid 4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished ...>
> [pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT (Connection
> timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
> [pid 4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
> timed out)
> [pid 4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
> [pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
> out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
> (Connection timed out)
> [pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
> [pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
> {tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
> detached
> strace: Process 4144 detached
> strace: Process 4145 detached
> strace: Process 4146 detached
> strace: Process 4147 detached
> strace: Process 4148 detached
> strace: Process 4150 detached
> strace: Process 4151 detached
> strace: Process 4152 detached
> strace: Process 4153 detached
> ....
>
>
> Could you help us to determine what's wrong with our setup?
>
> Thank you very much,
>
> Kind regards
> Lukas Weiss
>
Re: High CPU usage with Solr 7.7.0
Posted by Lukas Weiss <Lu...@raiffeisen.it>.
I forward this message. Thanks Adam.
Hi,
Apologies, I can’t figure out how to reply to the Solr mailing list.
I just ran across the same high CPU usage issue. I believe it’’s caused by
this commit which was introduced in Solr 7.7.0
https://github.com/apache/lucene-solr/commit/eb652b84edf441d8369f5188cdd5e3ae2b151434#diff-e54b251d166135a1afb7938cfe152bb5
There is a bug in JDK versions <=8 where using 0 threads in the
ScheduledThreadPool causes high CPU usage:
https://bugs.openjdk.java.net/browse/JDK-8129861
Oddly, the latest version
of solr/core/src/java/org/apache/solr/update/CommitTracker.java on
master still uses 0 executors as the default. Presumably most everyone is
using JDK 9 or greater which has the bug fixed, so they don’t experience
the bug.
Feel free to relay this back to the mailing list.
Thanks,
Adam Guthrie
Von: "Lukas Weiss" <Lu...@raiffeisen.it>
An: solr-user@lucene.apache.org,
Datum: 27.02.2019 11:13
Betreff: High CPU usage with Solr 7.7.0
Hello,
we recently updated our Solr server from 6.6.5 to 7.7.0. Since then, we
have problems with the server's CPU usage.
We have two Solr cores configured, but even if we clear all indexes and do
not start the index process, we see 100 CPU usage for both cores.
Here's what our top says:
root@solr:~ # top
top - 09:25:24 up 17:40, 1 user, load average: 2,28, 2,56, 2,68
Threads: 74 total, 3 running, 71 sleeping, 0 stopped, 0 zombie
%Cpu0 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
0,0 st
%Cpu1 :100,0 us, 0,0 sy, 0,0 ni, 0,0 id, 0,0 wa, 0,0 hi, 0,0 si,
0,0 st
%Cpu2 : 11,3 us, 1,0 sy, 0,0 ni, 86,7 id, 0,7 wa, 0,0 hi, 0,3 si,
0,0 st
%Cpu3 : 3,0 us, 3,0 sy, 0,0 ni, 93,7 id, 0,3 wa, 0,0 hi, 0,0 si,
0,0 st
KiB Mem : 8388608 total, 7859168 free, 496744 used, 32696
buff/cache
KiB Swap: 2097152 total, 2097152 free, 0 used. 7859168 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
P
10209 solr 20 0 6138468 452520 25740 R 99,9 5,4 29:43.45 java
-server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 24
10214 solr 20 0 6138468 452520 25740 R 99,9 5,4 28:42.91 java
-server -Xms1024m -Xmx1024m -XX:NewRatio=3 -XX:SurvivorRatio=4
-XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=8
-XX:+UseConcMarkSweepGC -XX:ConcGCThreads=4 + 25
The solr server is installed on a Debian Stretch 9.8 (64bit) on Linux LXC
dedicated Container.
Some more server info:
root@solr:~ # java -version
openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-2~deb9u1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)
root@solr:~ # free -m
total used free shared buff/cache
available
Mem: 8192 484 7675 701 31 7675
Swap: 2048 0 2048
We also found something strange if we do an strace of the main process, we
get lots of ongoing connection timeouts:
root@solr:~ # strace -F -p 4136
strace: Process 4136 attached with 48 threads
strace: [ Process PID=11089 runs in x32 mode. ]
[pid 4937] epoll_wait(139, <unfinished ...>
[pid 4936] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4909] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4618] epoll_wait(136, <unfinished ...>
[pid 4576] futex(0x7ff61ce66474, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
...>
[pid 4279] futex(0x7ff61ce62b34, FUTEX_WAIT_PRIVATE, 2203, NULL
<unfinished ...>
[pid 4244] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4227] futex(0x7ff56c71ae14, FUTEX_WAIT_PRIVATE, 2237, NULL
<unfinished ...>
[pid 4243] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4228] futex(0x7ff5608331a4, FUTEX_WAIT_PRIVATE, 2237, NULL
<unfinished ...>
[pid 4208] futex(0x7ff61ce63e54, FUTEX_WAIT_PRIVATE, 5, NULL <unfinished
...>
[pid 4205] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4204] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4196] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4195] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4194] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4193] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4187] restart_syscall(<... resuming interrupted restart_syscall ...>
<unfinished ...>
[pid 4180] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4179] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4177] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4174] accept(133, <unfinished ...>
[pid 4173] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4172] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4171] restart_syscall(<... resuming interrupted restart_syscall ...>
<unfinished ...>
[pid 4165] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4164] futex(0x7ff61c1f5054, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
...>
[pid 4163] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4162] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4161] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4160] futex(0x7ff623d52c20,
FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, 0xffffffff
<unfinished ...>
[pid 4159] futex(0x7ff61c1e9d54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid 4158] futex(0x7ff61c1b7f54, FUTEX_WAIT_PRIVATE, 15, NULL <unfinished
...>
[pid 4157] futex(0x7ff61c1b5554, FUTEX_WAIT_PRIVATE, 19, NULL <unfinished
...>
[pid 4156] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4155] restart_syscall(<... resuming interrupted futex ...>
<unfinished ...>
[pid 4153] futex(0x7ff61c06c754, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid 4152] futex(0x7ff61c06ab54, FUTEX_WAIT_PRIVATE, 3, NULL <unfinished
...>
[pid 4151] futex(0x7ff61c068f54, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid 4150] futex(0x7ff61c067354, FUTEX_WAIT_PRIVATE, 7, NULL <unfinished
...>
[pid 4148] futex(0x7ff61c024a54, FUTEX_WAIT_PRIVATE, 403, NULL
<unfinished ...>
[pid 4165] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=849859736}, 0xffffffff <unfinished ...>
[pid 4147] futex(0x7ff61c022e54, FUTEX_WAIT_PRIVATE, 415, NULL
<unfinished ...>
[pid 4146] futex(0x7ff61c021254, FUTEX_WAIT_PRIVATE, 397, NULL
<unfinished ...>
[pid 4145] futex(0x7ff61c01f654, FUTEX_WAIT_PRIVATE, 405, NULL
<unfinished ...>
[pid 4144] futex(0x7ff61c00e354, FUTEX_WAIT_PRIVATE, 1, NULL <unfinished
...>
[pid 4136] futex(0x7ff624b729d0, FUTEX_WAIT, 4144, NULL <unfinished ...>
[pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=900162344}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564856, tv_nsec=950365105}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=586325}, 0xffffffff) = -1 ETIMEDOUT (Connection
timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=50791977}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=100997890}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=151206817}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=201402531}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=251616284}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=301813556}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=352036802}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=402239182}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=452439835}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=502635489}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=552844020}, 0xffffffff <unfinished ...>
[pid 4156] <... restart_syscall resumed> ) = -1 ETIMEDOUT (Connection
timed out)
[pid 4156] futex(0x7ff61c1aba28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4156] futex(0x7ff61c1aba54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564858, tv_nsec=506449064}, 0xffffffff <unfinished ...>
[pid 4165] <... futex resumed> ) = -1 ETIMEDOUT (Connection timed
out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=603013734}, 0xffffffff) = -1 ETIMEDOUT
(Connection timed out)
[pid 4165] futex(0x7ff61c1f7a28, FUTEX_WAKE_PRIVATE, 1) = 0
[pid 4165] futex(0x7ff61c1f7a54, FUTEX_WAIT_BITSET_PRIVATE, 1,
{tv_sec=32564857, tv_nsec=653149664}, 0xffffffff^Cstrace: Process 4136
detached
strace: Process 4144 detached
strace: Process 4145 detached
strace: Process 4146 detached
strace: Process 4147 detached
strace: Process 4148 detached
strace: Process 4150 detached
strace: Process 4151 detached
strace: Process 4152 detached
strace: Process 4153 detached
....
Could you help us to determine what's wrong with our setup?
Thank you very much,
Kind regards
Lukas Weiss