You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@skywalking.apache.org by "cylrk (via GitHub)" <gi...@apache.org> on 2023/05/17 02:02:05 UTC

[GitHub] [skywalking] cylrk opened a new issue, #10816: [Bug]

cylrk opened a new issue, #10816:
URL: https://github.com/apache/skywalking/issues/10816

   ### Search before asking
   
   - [X] I had searched in the [issues](https://github.com/apache/skywalking/issues?q=is%3Aissue) and found no similar issues.
   
   
   ### Apache SkyWalking Component
   
   OAP server (apache/skywalking)
   
   ### What happened
   
   hello! Please ask two questions. During the process of using Skywalking 9.1 (deploying 5 8-core 16G servers on Alibaba ECS) in the production environment, there were some situations where data loss was reported from the agent to the OAP cluster (as can be seen from the logs in the first version, there were many GRPC timeouts)
   So in the later stage, it was changed to kafka reporter for reporting data. However, there is still some data loss.
   2. During the log investigation process, many Grpc server thread pools were found to be full, rejecting the task. The queue size is the default of 10000. I would like to know if after reporting data nodes through Kafka, shouldn't there be so many Grpc tasks? 
   <img width="1444" alt="image" src="https://github.com/apache/skywalking/assets/65347924/cd5cb2d0-e72b-41b0-8466-1f47773f90aa">
   
   
   ### What you expected to happen
   
   I hope there will be no grpc task!
   
   ### How to reproduce
   
   The Grpc server thread pool is full, rejecting the task after replacing it with kafka after a large amount of timeout was originally reported by Grpc
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on issue #10816: [Bug]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10816:
URL: https://github.com/apache/skywalking/issues/10816#issuecomment-1550581974

   I can see having anything relative to bug report. Thread pool is full, means the OAP can't process and flush the reported data, then it blocks the gRPC channel to avoid potential OOM.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] cylrk commented on issue #10816: [Bug]

Posted by "cylrk (via GitHub)" <gi...@apache.org>.
cylrk commented on issue #10816:
URL: https://github.com/apache/skywalking/issues/10816#issuecomment-1550946445

   > We never said the Kafka reporter would replace the whole gRPC reports. Some information still relies on gRPC channel. This means more that your OAP is not working healthy, especially, can't write all data in the storage or analyzed.
   
   From the logs printed by OAP, it can be seen that TraceSegmentReportServiceHandler is still receiving TraceSegment information. It should be that this has caused the grpc queue to be full. I would like to ask if after kafka reported is enabled, it still uses grpc to report TraceSegment information
   <img width="1433" alt="image" src="https://github.com/apache/skywalking/assets/65347924/6f70c6ba-ebc9-47b9-a97f-df343e4eb84d">
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on issue #10816: [Bug]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10816:
URL: https://github.com/apache/skywalking/issues/10816#issuecomment-1551058726

   About segment report, yes. gRPC reporter would be replaced.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] cylrk commented on issue #10816: [Bug]

Posted by "cylrk (via GitHub)" <gi...@apache.org>.
cylrk commented on issue #10816:
URL: https://github.com/apache/skywalking/issues/10816#issuecomment-1550586674

   > 我们只接受GitHub上的英文讨论。
   
   
   
   > We only accept English discussions on GitHub.
   
   May I ask why there are so many GRPC tasks after Kafka Reporter reports data?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on issue #10816: [Bug]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10816:
URL: https://github.com/apache/skywalking/issues/10816#issuecomment-1550588488

   We never said the Kafka reporter would replace the whole gRPC reports. Some information still repies on gRPC channel.
   This means more that your OAP is not working healthy, especially, can't write all data in the storage or analyzed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on issue #10816: [Bug]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10816:
URL: https://github.com/apache/skywalking/issues/10816#issuecomment-1550585173

   We only accept English discussions on GitHub. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng commented on issue #10816: [Bug]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng commented on issue #10816:
URL: https://github.com/apache/skywalking/issues/10816#issuecomment-1550972678

   How I can know that? That is your environment. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] wu-sheng closed issue #10816: [Bug]

Posted by "wu-sheng (via GitHub)" <gi...@apache.org>.
wu-sheng closed issue #10816: [Bug] 
URL: https://github.com/apache/skywalking/issues/10816


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] cylrk commented on issue #10816: [Bug]

Posted by "cylrk (via GitHub)" <gi...@apache.org>.
cylrk commented on issue #10816:
URL: https://github.com/apache/skywalking/issues/10816#issuecomment-1550981229

   > How I can know that? That is your environment.
   
   Thank you for your answer. I just want to confirm that according to the default skywalking9.1, theoretically, once Kafka Reporter - Kafka TraceSegmentServiceClient is enabled, TraceSegmentServiceClient will no longer be used to report data
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] cylrk commented on issue #10816: [Bug]

Posted by "cylrk (via GitHub)" <gi...@apache.org>.
cylrk commented on issue #10816:
URL: https://github.com/apache/skywalking/issues/10816#issuecomment-1550584529

   > 我可以看到与错误报告相关的任何内容。线程池已满,意味着 OAP 无法处理和刷新报告的数据,然后它会阻塞 gRPC 通道以避免潜在的 OOM。
   
   
   请问Kafka Reporter上报数据后为什么会有这么多GRPC任务?
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [skywalking] cylrk commented on issue #10816: [Bug]

Posted by "cylrk (via GitHub)" <gi...@apache.org>.
cylrk commented on issue #10816:
URL: https://github.com/apache/skywalking/issues/10816#issuecomment-1550584278

   May I ask why there are so many GRPC tasks after Kafka Reporter reports data?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@skywalking.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org