[GitHub] chenpengfei opened a new issue #2168: Collecter server's load average is very big

[GitBox <gi...@apache.org>]

chenpengfei opened a new issue #2168: Collecter server's load average is very big
URL: https://github.com/apache/incubator-skywalking/issues/2168
 
 
   Please answer these questions before submitting your issue.
   
   - Why do you submit this issue?
   - [ ] Question or discussion
   - [ ] Bug
   - [ * ] Requirement
   - [ ] Feature or performance improvement
   
   ___
   ### Question
   - What do you want to know?
   
   ___
   ### Bug
   - Which version of SkyWalking, OS and JRE?
   
   - Which company or project?
   
   - What happen?
   If possible, provide a way for reproducing the error. e.g. demo application, component version.
   
   ___
   ### Requirement or improvement
   - Please describe about your requirements or improvement suggestions.
   
   Collector server's memory is enough, but cpu usage rate and load average are very high.
   
   `jstack` print info 
   
   `
   "grpc-default-executor-5112" #13959 daemon prio=5 os_prio=0 tid=0x00007fb7b4fae800 nid=0x714 sleeping[0x00007fb774b21000]
      java.lang.Thread.State: TIMED_WAITING (sleeping)
           at java.lang.Thread.sleep(Native Method)
           at org.apache.skywalking.apm.commons.datacarrier.buffer.Buffer.save(Buffer.java:64)
           at org.apache.skywalking.apm.commons.datacarrier.buffer.Channels.save(Channels.java:52)
           at org.apache.skywalking.apm.commons.datacarrier.DataCarrier.produce(DataCarrier.java:82)
           at org.apache.skywalking.apm.collector.analysis.worker.model.base.LocalAsyncWorkerRef.in(LocalAsyncWorkerRef.java:74)
           at org.apache.skywalking.apm.collector.analysis.worker.model.base.LocalAsyncWorkerRef.in(LocalAsyncWorkerRef.java:33)
           at org.apache.skywalking.apm.collector.core.graph.Next.lambda$execute$0(Next.java:49)
           at org.apache.skywalking.apm.collector.core.graph.Next$$Lambda$805/635779307.accept(Unknown Source)
           at java.lang.Iterable.forEach(Iterable.java:75)
           at org.apache.skywalking.apm.collector.core.graph.Next.execute(Next.java:49)
           at org.apache.skywalking.apm.collector.analysis.metric.provider.worker.service.refmetric.ServiceReferenceHourMetricTransformNode.process(ServiceReferenceHourMetricTransformNode.java:44)
           at org.apache.skywalking.apm.collector.analysis.metric.provider.worker.service.refmetric.ServiceReferenceHourMetricTransformNode.process(ServiceReferenceHourMetricTransformNode.java:31)
           at org.apache.skywalking.apm.collector.core.graph.Node.execute(Node.java:52)
           at org.apache.skywalking.apm.collector.core.graph.WayToNode.out(WayToNode.java:40)
           at org.apache.skywalking.apm.collector.core.graph.DirectWay.in(DirectWay.java:31)
           at org.apache.skywalking.apm.collector.core.graph.Next.lambda$execute$0(Next.java:49)
           at org.apache.skywalking.apm.collector.core.graph.Next$$Lambda$805/635779307.accept(Unknown Source)
           at java.lang.Iterable.forEach(Iterable.java:75)
           at org.apache.skywalking.apm.collector.core.graph.Next.execute(Next.java:49)
           at org.apache.skywalking.apm.collector.remote.grpc.handler.RemoteCommonServiceHandler$1.onNext(RemoteCommonServiceHandler.java:63)
           at org.apache.skywalking.apm.collector.remote.grpc.handler.RemoteCommonServiceHandler$1.onNext(RemoteCommonServiceHandler.java:52)
           at io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onMessage(ServerCalls.java:248)
           at io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.messagesAvailable(ServerCallImpl.java:263)
           at io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1MessagesAvailable.runInContext(ServerImpl.java:683)
           at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
           at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
           at java.lang.Thread.run(Thread.java:745)
   `
   
   I hope, we can config the sleep time, total thread count, buffer size by config file according my machine.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services