[GitHub] [skywalking] jadezhong opened a new issue #3441: Grpc server thread pool is full, rejecting the task

[GitBox <gi...@apache.org>]

jadezhong opened a new issue #3441: Grpc server thread pool is full, rejecting the task
URL: https://github.com/apache/skywalking/issues/3441
 
 
   Please answer these questions before submitting your issue.
   
   - Why do you submit this issue?
   - [x] Question or discussion
   - [ ] Bug
   - [ ] Requirement
   - [ ] Feature or performance improvement
   
   ___
   ### Question
   - What do you want to know?
   We have two oapServer, one of it report "Grpc server thread pool is full, rejecting the task" after running some days with fix GrpcServer's threadPoolSize and queueSize.
   
   ___
   ### Bug
   - Which version of SkyWalking, OS and JRE?
   Skywalking 6.3.0
   CentOS 7.x, JDK 1.8, 8C CPU, 16G RAM
   
   - Which company or project?
   
   - What happen?
   If possible, provide a way for reproducing the error. e.g. demo application, component version.
   Following solution of #3405, fixed 6.3.0 according to master branch(6.4.0 under testing) by setting GrpcServer's threadPoolSize and queueSize, oapServer seems ok within one day(no grpc-default-executor-* exponential growth and no heap OOM). but after running two ~ three days, one of our oapServer continue report "Grpc server thread pool is full, rejecting the task"(屏幕快照 2019-09-09 10 09 53) till now(屏幕快照 2019-09-09 10 10 36) in log.
   in jConsole, difference between two oapServer is the one reported  "Grpc server thread pool is full, rejecting the task" has no grpc-default-executor-*(屏幕快照 2019-09-09 10 11 42).
   
   <img width="1414" alt="屏幕快照 2019-09-09 10 09 53" src="https://user-images.githubusercontent.com/49858897/64500172-b4dde080-d2ee-11e9-9c96-b6eb7bf873d0.png">
   <img width="1281" alt="屏幕快照 2019-09-09 10 10 36" src="https://user-images.githubusercontent.com/49858897/64500215-dfc83480-d2ee-11e9-8aeb-dbd7e824217c.png">
   <img width="1640" alt="屏幕快照 2019-09-09 10 11 42" src="https://user-images.githubusercontent.com/49858897/64500248-fbcbd600-d2ee-11e9-929f-a2d59cd5694d.png">
   
   ___
   ### Requirement or improvement
   - Please describe about your requirements or improvement suggestions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services