You are viewing a plain text version of this content. The canonical link for it is here.

Posted to notifications@skywalking.apache.org by GitBox <gi...@apache.org> on 2020/10/21 03:49:43 UTC

[GitHub] [skywalking] liyanyan-dev opened a new issue #5705: Agent cannot reconnect OAP

liyanyan-dev opened a new issue #5705:
URL: https://github.com/apache/skywalking/issues/5705

Please answer these questions before submitting your issue.

- Why do you submit this issue?
- [√ ] Question or discussion
- [ ] Bug
- [ ] Requirement
- [ ] Feature or performance improvement

___
### Question
I occasionally find that after the OAP is restarted, the agent can't be reconnected to the OAP for about half an hour. After reading the source code of the agent, I find that the following problems may exist:
1. Multiple threads will report errors and update the connection status. When a certain thread reports an error, it happens that the GRPCChannelManager thread starts scheduling，After changing the connection status to connected, another thread reports an error and changes the status to disconnected. In extreme cases, the agent cannot be reconnected with OAP all the time.
___
### Bug
- Which version of SkyWalking, OS and JRE?
skywalking 8.1.0
centos7
- Which company or project?

- What happened?
If possible, provide a way to reproduce the error. e.g. demo application, component version.

___
### Requirement or improvement
- Please describe your requirements or improvement suggestions.

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] liyanyan-dev commented on issue #5705: Agent cannot reconnect OAP

Posted by GitBox <gi...@apache.org>.

liyanyan-dev commented on issue #5705:
URL: https://github.com/apache/skywalking/issues/5705#issuecomment-713302760


   OAP server is healthy, other agent can reconnnect to oap;Only one agent can't reconnect，but restarting the agent is normal .The agent debug log always reports that it is unable to connect and the segment data is ignored;The log file was removed when the agent was restarted


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] kezhenxu94 edited a comment on issue #5705: Agent cannot reconnect OAP

Posted by GitBox <gi...@apache.org>.

kezhenxu94 edited a comment on issue #5705:
URL: https://github.com/apache/skywalking/issues/5705#issuecomment-713285867


   > Multiple threads will report errors and update the connection status. When a certain thread reports an error, it happens that the GRPCChannelManager thread starts scheduling，After changing the connection status to connected, another thread reports an error and changes the status to disconnected. In extreme cases, the agent cannot be reconnected with OAP all the time.
   
   How could that be possible if your OAP server is back to healthy, if you OAP server is healthy finally, all the threads should be able to finally change the status to connected. Otherwise there must be something else wrong. Do you have any extra information/logs?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] wu-sheng commented on issue #5705: Agent cannot reconnect OAP

Posted by GitBox <gi...@apache.org>.

wu-sheng commented on issue #5705:
URL: https://github.com/apache/skywalking/issues/5705#issuecomment-713537815


   If it is not connected, it should continue to retry. In this case, can't happen the race condition you described.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] kezhenxu94 commented on issue #5705: Agent cannot reconnect OAP

Posted by GitBox <gi...@apache.org>.

kezhenxu94 commented on issue #5705:
URL: https://github.com/apache/skywalking/issues/5705#issuecomment-713285867


   > Multiple threads will report errors and update the connection status. When a certain thread reports an error, it happens that the GRPCChannelManager thread starts scheduling，After changing the connection status to connected, another thread reports an error and changes the status to disconnected. In extreme cases, the agent cannot be reconnected with OAP all the time.
   
   How could that be possible if your OAP server is back to healthy, if you OAP server is healthy finally, all the threads should be able to finally change the status to connected. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] kezhenxu94 edited a comment on issue #5705: Agent cannot reconnect OAP

Posted by GitBox <gi...@apache.org>.

kezhenxu94 edited a comment on issue #5705:
URL: https://github.com/apache/skywalking/issues/5705#issuecomment-713319785


   > Only one agent can't reconnect，but restarting the agent is normal.
   
   What's the difference between that agent and the others? Is that agent reproducible always? If yes, can you minimize the codes for us to reproduce this issue and fix?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] kezhenxu94 closed issue #5705: Agent cannot reconnect OAP

Posted by GitBox <gi...@apache.org>.

kezhenxu94 closed issue #5705:
URL: https://github.com/apache/skywalking/issues/5705


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [skywalking] kezhenxu94 commented on issue #5705: Agent cannot reconnect OAP

Posted by GitBox <gi...@apache.org>.

kezhenxu94 commented on issue #5705:
URL: https://github.com/apache/skywalking/issues/5705#issuecomment-713319785


   > Only one agent can't reconnect，but restarting the agent is normal.
   
   Is that agent reproducible always? If yes, can you minimize the codes for us to reproduce this issue and fix?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org