You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2019/02/15 03:50:10 UTC

[GitHub] wu-sheng edited a comment on issue #2247: Backend streaming thread model improvement

wu-sheng edited a comment on issue #2247: Backend streaming thread model improvement
URL: https://github.com/apache/incubator-skywalking/pull/2247#issuecomment-463894944
 
 
   I run `jstack` at my local machine, and show this.
   - One `DataCarrier.REGISTER_L1.BulkConsumePool`, `DataCarrier.REGISTER_L2.BulkConsumePool`.
   - One `DataCarrier.RECORD_PERSISTENT.BulkConsumePool`
   - 7 `DataCarrier.INDICATOR_L1_AGGREGATION`s
   - 1 `DataCarrier.INDICATOR_L2_AGGREGATION`
   
   Consider my laptop has 8 cores, this should be the right result.
   
   ```
   2019-02-15 10:50:51
   Full thread dump Java HotSpot(TM) 64-Bit Server VM (25.91-b14 mixed mode):
   
   "Attach Listener" #68 daemon prio=9 os_prio=31 tid=0x00007fb1ff994800 nid=0x14e07 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "qtp818093527-67" #67 prio=5 os_prio=31 tid=0x00007fb204121000 nid=0xae0b waiting on condition [0x00007000124f9000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x00000006c02dc1e0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:392)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:564)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.access$800(QueuedThreadPool.java:49)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:627)
   	at java.lang.Thread.run(Thread.java:745)
   
   "elasticsearch[scheduler][T#1]" #65 daemon prio=5 os_prio=31 tid=0x00007fb1fcb6d000 nid=0x15003 waiting on condition [0x0000700012a08000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x000000077872a960> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "DestroyJavaVM" #64 prio=5 os_prio=31 tid=0x00007fb2028e5000 nid=0x2503 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "pool-15-thread-1" #63 prio=5 os_prio=31 tid=0x00007fb203414800 nid=0xb403 waiting on condition [0x0000700012905000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x0000000778840600> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "pool-14-thread-1" #62 prio=5 os_prio=31 tid=0x00007fb20138b800 nid=0xb303 waiting on condition [0x0000700012802000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x0000000778840e10> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "pool-13-thread-1" #61 prio=5 os_prio=31 tid=0x00007fb200f97000 nid=0x15203 waiting on condition [0x00007000126ff000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x000000077880f1d8> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "qtp818093527-59-acceptor-0@25028f54-ServerConnector@4567fb2b{HTTP/1.1,[http/1.1]}{0.0.0.0:12800}" #59 prio=3 os_prio=31 tid=0x00007fb201997800 nid=0xb003 runnable [0x00007000125fc000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
   	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:422)
   	at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:250)
   	- locked <0x00000007787f7988> (a java.lang.Object)
   	at org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:352)
   	at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:603)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   	at java.lang.Thread.run(Thread.java:745)
   
   "qtp818093527-57-lowPrioritySelector" #57 prio=1 os_prio=31 tid=0x00007fb1fc917000 nid=0xad03 waiting for monitor entry [0x00007000123f6000]
      java.lang.Thread.State: BLOCKED (on object monitor)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:244)
   	- waiting to lock <0x00000007787f4ae8> (a org.eclipse.jetty.io.ManagedSelector$SelectorProducer)
   	at org.eclipse.jetty.util.thread.strategy.ProduceExecuteConsume.produce(ProduceExecuteConsume.java:75)
   	at org.eclipse.jetty.io.ManagedSelector.lowPriorityProduce(ManagedSelector.java:113)
   	at org.eclipse.jetty.io.ManagedSelector$$Lambda$194/1809392952.run(Unknown Source)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   	at java.lang.Thread.run(Thread.java:745)
   
   "qtp818093527-56" #56 prio=5 os_prio=31 tid=0x00007fb1fc951000 nid=0x15403 runnable [0x00007000122f3000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x00000007787f4cb8> (a sun.nio.ch.Util$2)
   	- locked <0x00000007787f4ca8> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x00000007787f4b78> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:302)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:254)
   	- locked <0x00000007787f4ae8> (a org.eclipse.jetty.io.ManagedSelector$SelectorProducer)
   	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:147)
   	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produce(ExecuteProduceConsume.java:97)
   	at org.eclipse.jetty.io.ManagedSelector$$Lambda$193/1873947363.run(Unknown Source)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   	at java.lang.Thread.run(Thread.java:745)
   
   "qtp818093527-55-lowPrioritySelector" #55 prio=1 os_prio=31 tid=0x00007fb1fca0d000 nid=0xaa03 waiting for monitor entry [0x00007000121f0000]
      java.lang.Thread.State: BLOCKED (on object monitor)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:244)
   	- waiting to lock <0x00000007787f4f80> (a org.eclipse.jetty.io.ManagedSelector$SelectorProducer)
   	at org.eclipse.jetty.util.thread.strategy.ProduceExecuteConsume.produce(ProduceExecuteConsume.java:75)
   	at org.eclipse.jetty.io.ManagedSelector.lowPriorityProduce(ManagedSelector.java:113)
   	at org.eclipse.jetty.io.ManagedSelector$$Lambda$194/1809392952.run(Unknown Source)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   	at java.lang.Thread.run(Thread.java:745)
   
   "qtp818093527-54" #54 prio=5 os_prio=31 tid=0x00007fb201f63000 nid=0x7903 runnable [0x00007000120ed000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x00000007787f5150> (a sun.nio.ch.Util$2)
   	- locked <0x00000007787f5140> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x00000007787f5010> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:302)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:254)
   	- locked <0x00000007787f4f80> (a org.eclipse.jetty.io.ManagedSelector$SelectorProducer)
   	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:147)
   	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produce(ExecuteProduceConsume.java:97)
   	at org.eclipse.jetty.io.ManagedSelector$$Lambda$193/1873947363.run(Unknown Source)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   	at java.lang.Thread.run(Thread.java:745)
   
   "qtp818093527-53-lowPrioritySelector" #53 prio=1 os_prio=31 tid=0x00007fb1fdc00800 nid=0x7803 waiting for monitor entry [0x0000700011fea000]
      java.lang.Thread.State: BLOCKED (on object monitor)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:244)
   	- waiting to lock <0x00000007787f5418> (a org.eclipse.jetty.io.ManagedSelector$SelectorProducer)
   	at org.eclipse.jetty.util.thread.strategy.ProduceExecuteConsume.produce(ProduceExecuteConsume.java:75)
   	at org.eclipse.jetty.io.ManagedSelector.lowPriorityProduce(ManagedSelector.java:113)
   	at org.eclipse.jetty.io.ManagedSelector$$Lambda$194/1809392952.run(Unknown Source)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   	at java.lang.Thread.run(Thread.java:745)
   
   "qtp818093527-52" #52 prio=5 os_prio=31 tid=0x00007fb1ff97a000 nid=0x7703 runnable [0x0000700011ee7000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x00000007787f55e8> (a sun.nio.ch.Util$2)
   	- locked <0x00000007787f55d8> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x00000007787f54a8> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:302)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:254)
   	- locked <0x00000007787f5418> (a org.eclipse.jetty.io.ManagedSelector$SelectorProducer)
   	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:147)
   	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produce(ExecuteProduceConsume.java:97)
   	at org.eclipse.jetty.io.ManagedSelector$$Lambda$193/1873947363.run(Unknown Source)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   	at java.lang.Thread.run(Thread.java:745)
   
   "qtp818093527-51-lowPrioritySelector" #51 prio=1 os_prio=31 tid=0x00007fb20285e800 nid=0x7e03 waiting for monitor entry [0x0000700011de4000]
      java.lang.Thread.State: BLOCKED (on object monitor)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:244)
   	- waiting to lock <0x00000007787f58b0> (a org.eclipse.jetty.io.ManagedSelector$SelectorProducer)
   	at org.eclipse.jetty.util.thread.strategy.ProduceExecuteConsume.produce(ProduceExecuteConsume.java:75)
   	at org.eclipse.jetty.io.ManagedSelector.lowPriorityProduce(ManagedSelector.java:113)
   	at org.eclipse.jetty.io.ManagedSelector$$Lambda$194/1809392952.run(Unknown Source)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   	at java.lang.Thread.run(Thread.java:745)
   
   "qtp818093527-50" #50 prio=5 os_prio=31 tid=0x00007fb204119000 nid=0x7f03 runnable [0x0000700011ce1000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x00000007787f5a80> (a sun.nio.ch.Util$2)
   	- locked <0x00000007787f5a70> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x00000007787f5940> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.select(ManagedSelector.java:302)
   	at org.eclipse.jetty.io.ManagedSelector$SelectorProducer.produce(ManagedSelector.java:254)
   	- locked <0x00000007787f58b0> (a org.eclipse.jetty.io.ManagedSelector$SelectorProducer)
   	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produceConsume(ExecuteProduceConsume.java:147)
   	at org.eclipse.jetty.util.thread.strategy.ExecuteProduceConsume.produce(ExecuteProduceConsume.java:97)
   	at org.eclipse.jetty.io.ManagedSelector$$Lambda$193/1873947363.run(Unknown Source)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:672)
   	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:590)
   	at java.lang.Thread.run(Thread.java:745)
   
   "grpc-default-boss-ELG-1-1" #49 daemon prio=5 os_prio=31 tid=0x00007fb201bdb000 nid=0x7403 runnable [0x0000700011bde000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x00000007787d3e60> (a io.netty.channel.nio.SelectedSelectionKeySet)
   	- locked <0x000000077883fb40> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x00000007787d3dc0> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62)
   	at io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:755)
   	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:410)
   	at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:884)
   	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
   	at java.lang.Thread.run(Thread.java:745)
   
   "DataCarrier.default.Consumser.0.Thread" #48 daemon prio=5 os_prio=31 tid=0x00007fb201bdb800 nid=0x8003 waiting on condition [0x0000700011adb000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:72)
   
   "pool-11-thread-1" #47 prio=5 os_prio=31 tid=0x00007fb2009ae000 nid=0x8203 waiting on condition [0x00007000119d8000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x000000077880ccb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "pool-10-thread-1" #46 prio=5 os_prio=31 tid=0x00007fb203d62000 nid=0x7203 waiting on condition [0x00007000118d5000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x000000077880c3f0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "DataCarrier.default.Consumser.0.Thread" #45 daemon prio=5 os_prio=31 tid=0x00007fb200ad7800 nid=0x8403 waiting on condition [0x00007000117d2000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:72)
   
   "pool-9-thread-1" #44 prio=5 os_prio=31 tid=0x00007fb1ff378000 nid=0x7003 runnable [0x00007000116cf000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x000000077880a498> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "pool-8-thread-1" #43 prio=5 os_prio=31 tid=0x00007fb2028b6000 nid=0x8603 waiting on condition [0x00007000115cc000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x0000000778833df0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "I/O dispatcher 8" #42 prio=5 os_prio=31 tid=0x00007fb1fd9c4000 nid=0x6d03 runnable [0x00007000114c9000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x00000007788564e8> (a sun.nio.ch.Util$2)
   	- locked <0x00000007788564d8> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x0000000778856398> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
   	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   	at java.lang.Thread.run(Thread.java:745)
   
   "I/O dispatcher 7" #41 prio=5 os_prio=31 tid=0x00007fb201d44000 nid=0x8803 runnable [0x00007000113c6000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x0000000778833b98> (a sun.nio.ch.Util$2)
   	- locked <0x0000000778833b88> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x0000000778833a48> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
   	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   	at java.lang.Thread.run(Thread.java:745)
   
   "I/O dispatcher 6" #40 prio=5 os_prio=31 tid=0x00007fb203c6f000 nid=0x8a03 runnable [0x00007000112c3000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x000000077880ed18> (a sun.nio.ch.Util$2)
   	- locked <0x000000077880ed08> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x000000077880ebc8> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
   	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   	at java.lang.Thread.run(Thread.java:745)
   
   "I/O dispatcher 5" #39 prio=5 os_prio=31 tid=0x00007fb1febda800 nid=0x8b03 runnable [0x00007000111c0000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x000000077880e730> (a sun.nio.ch.Util$2)
   	- locked <0x000000077880e720> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x000000077880e5e0> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
   	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   	at java.lang.Thread.run(Thread.java:745)
   
   "I/O dispatcher 4" #38 prio=5 os_prio=31 tid=0x00007fb1ffc6b800 nid=0x8d03 runnable [0x00007000110bd000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x0000000778856890> (a sun.nio.ch.Util$2)
   	- locked <0x0000000778856880> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x0000000778856740> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
   	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   	at java.lang.Thread.run(Thread.java:745)
   
   "I/O dispatcher 3" #37 prio=5 os_prio=31 tid=0x00007fb1fc932000 nid=0x6b03 runnable [0x0000700010fba000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x0000000778856c38> (a sun.nio.ch.Util$2)
   	- locked <0x0000000778856c28> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x0000000778856ae8> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
   	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   	at java.lang.Thread.run(Thread.java:745)
   
   "I/O dispatcher 2" #36 prio=5 os_prio=31 tid=0x00007fb201c76800 nid=0x9003 runnable [0x0000700010eb7000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x0000000778856fe0> (a sun.nio.ch.Util$2)
   	- locked <0x0000000778856fd0> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x0000000778856e90> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
   	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   	at java.lang.Thread.run(Thread.java:745)
   
   "I/O dispatcher 1" #35 prio=5 os_prio=31 tid=0x00007fb20431e000 nid=0x6a03 runnable [0x0000700010db4000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x0000000778855778> (a sun.nio.ch.Util$2)
   	- locked <0x0000000778855768> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x0000000778855558> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:255)
   	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
   	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)
   	at java.lang.Thread.run(Thread.java:745)
   
   "pool-7-thread-1" #34 prio=5 os_prio=31 tid=0x00007fb201c75800 nid=0x6903 runnable [0x0000700010cb1000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x00000006c0c5b030> (a sun.nio.ch.Util$2)
   	- locked <0x00000006c0c5b020> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x00000006c0c5af00> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor.execute(AbstractMultiworkerIOReactor.java:340)
   	at org.apache.http.impl.nio.conn.PoolingNHttpClientConnectionManager.execute(PoolingNHttpClientConnectionManager.java:192)
   	at org.apache.http.impl.nio.client.CloseableHttpAsyncClientBase$1.run(CloseableHttpAsyncClientBase.java:64)
   	at java.lang.Thread.run(Thread.java:745)
   
   "pool-6-thread-1" #33 prio=5 os_prio=31 tid=0x00007fb200fe5000 nid=0x6803 waiting on condition [0x0000700010bae000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x00000006c0748228> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "pool-5-thread-1" #32 prio=5 os_prio=31 tid=0x00007fb202989800 nid=0x6703 waiting on condition [0x0000700010aab000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x00000006c0749270> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "DataCarrier.MeshDataBufferFileCache.Consumser.0.Thread" #31 daemon prio=5 os_prio=31 tid=0x00007fb1ff3f2000 nid=0x9503 waiting on condition [0x00007000109a8000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:72)
   
   "DataCarrier.default.Consumser.0.Thread" #30 daemon prio=5 os_prio=31 tid=0x00007fb201ff6800 nid=0x6503 waiting on condition [0x00007000108a5000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.ConsumerThread.run(ConsumerThread.java:72)
   
   "DataCarrier.RECORD_PERSISTENT.BulkConsumePool.0.Thread" #29 daemon prio=5 os_prio=31 tid=0x00007fb1ff7f7800 nid=0x6403 sleeping[0x00007000107a2000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "DataCarrier.REGISTER_L1.BulkConsumePool.0.Thread" #28 daemon prio=5 os_prio=31 tid=0x00007fb201af0800 nid=0x9803 waiting on condition [0x000070001069f000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "DataCarrier.REGISTER_L2.BulkConsumePool.0.Thread" #27 daemon prio=5 os_prio=31 tid=0x00007fb1fe87f000 nid=0x6203 waiting on condition [0x000070001059c000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "DataCarrier.INDICATOR_L1_AGGREGATION.BulkConsumePool.6.Thread" #26 daemon prio=5 os_prio=31 tid=0x00007fb1fda08000 nid=0x6103 sleeping[0x0000700010499000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "DataCarrier.INDICATOR_L1_AGGREGATION.BulkConsumePool.5.Thread" #25 daemon prio=5 os_prio=31 tid=0x00007fb1fda07000 nid=0x6003 waiting on condition [0x0000700010396000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "DataCarrier.INDICATOR_L1_AGGREGATION.BulkConsumePool.4.Thread" #24 daemon prio=5 os_prio=31 tid=0x00007fb1ff940000 nid=0x9d03 waiting on condition [0x0000700010293000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "DataCarrier.INDICATOR_L1_AGGREGATION.BulkConsumePool.3.Thread" #23 daemon prio=5 os_prio=31 tid=0x00007fb2009ca000 nid=0x9f03 waiting on condition [0x0000700010190000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "DataCarrier.INDICATOR_L1_AGGREGATION.BulkConsumePool.2.Thread" #22 daemon prio=5 os_prio=31 tid=0x00007fb2009e2800 nid=0xa103 waiting on condition [0x000070001008d000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "DataCarrier.INDICATOR_L1_AGGREGATION.BulkConsumePool.1.Thread" #21 daemon prio=5 os_prio=31 tid=0x00007fb1fe956800 nid=0x5e03 waiting on condition [0x000070000ff8a000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "DataCarrier.INDICATOR_L1_AGGREGATION.BulkConsumePool.0.Thread" #20 daemon prio=5 os_prio=31 tid=0x00007fb1fda99800 nid=0xa303 waiting on condition [0x000070000fe87000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "DataCarrier.INDICATOR_L2_AGGREGATION.BulkConsumePool.0.Thread" #19 daemon prio=5 os_prio=31 tid=0x00007fb201ac5800 nid=0x1207 waiting on condition [0x000070000fd84000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
   	at java.lang.Thread.sleep(Native Method)
   	at org.apache.skywalking.apm.commons.datacarrier.consumer.MultipleChannelsConsumer.run(MultipleChannelsConsumer.java:54)
   
   "pool-4-thread-1" #18 prio=5 os_prio=31 tid=0x00007fb1fd1b5000 nid=0x5c03 waiting on condition [0x000070000fc81000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x00000006c0567090> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "Thread-2" #17 prio=5 os_prio=31 tid=0x00007fb1fca6a800 nid=0xa603 runnable [0x000070000fb7e000]
      java.lang.Thread.State: RUNNABLE
   	at sun.nio.ch.KQueueArrayWrapper.kevent0(Native Method)
   	at sun.nio.ch.KQueueArrayWrapper.poll(KQueueArrayWrapper.java:198)
   	at sun.nio.ch.KQueueSelectorImpl.doSelect(KQueueSelectorImpl.java:117)
   	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
   	- locked <0x00000006c054ff98> (a sun.nio.ch.Util$2)
   	- locked <0x00000006c054ff88> (a java.util.Collections$UnmodifiableSet)
   	- locked <0x00000006c054fde8> (a sun.nio.ch.KQueueSelectorImpl)
   	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
   	at sun.net.httpserver.ServerImpl$Dispatcher.run(ServerImpl.java:352)
   	at java.lang.Thread.run(Thread.java:745)
   
   "server-timer" #16 daemon prio=5 os_prio=31 tid=0x00007fb1fc91f000 nid=0x5a03 in Object.wait() [0x000070000fa7b000]
      java.lang.Thread.State: TIMED_WAITING (on object monitor)
   	at java.lang.Object.wait(Native Method)
   	- waiting on <0x00000006c0550468> (a java.util.TaskQueue)
   	at java.util.TimerThread.mainLoop(Timer.java:552)
   	- locked <0x00000006c0550468> (a java.util.TaskQueue)
   	at java.util.TimerThread.run(Timer.java:505)
   
   "pool-2-thread-1" #15 prio=5 os_prio=31 tid=0x00007fb1fca6c800 nid=0x5903 waiting on condition [0x000070000f978000]
      java.lang.Thread.State: TIMED_WAITING (parking)
   	at sun.misc.Unsafe.park(Native Method)
   	- parking to wait for  <0x00000006c055c598> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
   	at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
   	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1093)
   	at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
   	at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1067)
   	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1127)
   	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
   	at java.lang.Thread.run(Thread.java:745)
   
   "Service Thread" #12 daemon prio=9 os_prio=31 tid=0x00007fb1fc842800 nid=0x5603 runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "C1 CompilerThread3" #11 daemon prio=9 os_prio=31 tid=0x00007fb1fd1b9000 nid=0x5503 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "C2 CompilerThread2" #10 daemon prio=9 os_prio=31 tid=0x00007fb1fd1b8000 nid=0x3903 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "C2 CompilerThread1" #9 daemon prio=9 os_prio=31 tid=0x00007fb1fd1b7800 nid=0x3b03 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "C2 CompilerThread0" #8 daemon prio=9 os_prio=31 tid=0x00007fb1fd1b6800 nid=0x3703 waiting on condition [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "JDWP Command Reader" #7 daemon prio=10 os_prio=31 tid=0x00007fb200007000 nid=0x3e03 runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "JDWP Event Helper Thread" #6 daemon prio=10 os_prio=31 tid=0x00007fb1fc824000 nid=0x4003 runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "JDWP Transport Listener: dt_socket" #5 daemon prio=10 os_prio=31 tid=0x00007fb1fe81c800 nid=0x3607 runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "Signal Dispatcher" #4 daemon prio=9 os_prio=31 tid=0x00007fb1fd011000 nid=0x3503 runnable [0x0000000000000000]
      java.lang.Thread.State: RUNNABLE
   
   "Finalizer" #3 daemon prio=8 os_prio=31 tid=0x00007fb1fc821800 nid=0x4a03 in Object.wait() [0x000070000ee57000]
      java.lang.Thread.State: WAITING (on object monitor)
   	at java.lang.Object.wait(Native Method)
   	- waiting on <0x00000006c001d9e8> (a java.lang.ref.ReferenceQueue$Lock)
   	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:143)
   	- locked <0x00000006c001d9e8> (a java.lang.ref.ReferenceQueue$Lock)
   	at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:164)
   	at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:209)
   
   "Reference Handler" #2 daemon prio=10 os_prio=31 tid=0x00007fb1ff00b000 nid=0x3103 in Object.wait() [0x000070000ed54000]
      java.lang.Thread.State: WAITING (on object monitor)
   	at java.lang.Object.wait(Native Method)
   	- waiting on <0x00000006c0035dc8> (a java.lang.ref.Reference$Lock)
   	at java.lang.Object.wait(Object.java:502)
   	at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
   	- locked <0x00000006c0035dc8> (a java.lang.ref.Reference$Lock)
   	at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
   
   "VM Thread" os_prio=31 tid=0x00007fb20002e800 nid=0x3003 runnable 
   
   "GC task thread#0 (ParallelGC)" os_prio=31 tid=0x00007fb1fe81b800 nid=0x1d07 runnable 
   
   "GC task thread#1 (ParallelGC)" os_prio=31 tid=0x00007fb1fc800800 nid=0x2a03 runnable 
   
   "GC task thread#2 (ParallelGC)" os_prio=31 tid=0x00007fb1fc81c800 nid=0x5303 runnable 
   
   "GC task thread#3 (ParallelGC)" os_prio=31 tid=0x00007fb1fc81d000 nid=0x5103 runnable 
   
   "GC task thread#4 (ParallelGC)" os_prio=31 tid=0x00007fb1fc81d800 nid=0x2b03 runnable 
   
   "GC task thread#5 (ParallelGC)" os_prio=31 tid=0x00007fb1fc81e800 nid=0x2d03 runnable 
   
   "GC task thread#6 (ParallelGC)" os_prio=31 tid=0x00007fb1fc81f000 nid=0x4f03 runnable 
   
   "GC task thread#7 (ParallelGC)" os_prio=31 tid=0x00007fb200000000 nid=0x4e03 runnable 
   
   "VM Periodic Task Thread" os_prio=31 tid=0x00007fb1fd80a000 nid=0x5803 waiting on condition 
   
   JNI global references: 21941
   ```

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services