You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by kiwiflyer <gi...@git.apache.org> on 2016/03/04 19:36:57 UTC
[GitHub] cloudstack pull request: CLOUDSTACK-9285 - Agent throws an excepti...
GitHub user kiwiflyer opened a pull request:
https://github.com/apache/cloudstack/pull/1429
CLOUDSTACK-9285 - Agent throws an exception on reconnect and fails
Agent throws an exception that can never be recovered from when the agent attempts to reconnect and is sent a RST. This case occurs when a load balancer (haproxy) is proxying the traffic and there is no management server active to serve the request on the backend.
Original issue logs from agent.log:
2016-03-03 17:15:36,527 INFO [utils.nio.NioClient] (Agent-Handler-2:null) (logid:) NioClient connection closed
2016-03-03 17:15:36,527 INFO [cloud.agent.Agent] (Agent-Handler-2:null) (logid:) Reconnecting...
2016-03-03 17:15:36,527 INFO [utils.nio.NioClient] (Agent-Handler-2:null) (logid:) Connecting to 10.103.0.154:8250
2016-03-03 17:15:36,540 ERROR [utils.nio.NioConnection] (Agent-Handler-2:null) (logid:) Unable to initialize the threads.
java.io.IOException: Connection closed with -1 on reading size.
at com.cloud.utils.nio.Link.doHandshake(Link.java:513)
at com.cloud.utils.nio.NioClient.init(NioClient.java:80)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.reconnect(Agent.java:413)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:868)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-03-03 17:15:36,545 INFO [utils.exception.CSExceptionErrorCode] (Agent-Handler-2:null) (logid:) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
<agent thread is dead>
After the patch is applied:
2016-03-03 17:50:05,190 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:) NioClient connection closed
2016-03-03 17:50:05,190 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:) Reconnecting...
2016-03-03 17:50:05,190 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:) Connecting to 10.103.0.154:8250
2016-03-03 17:50:05,206 ERROR [utils.nio.NioConnection] (Agent-Handler-3:null) (logid:) Unable to initialize the threads.
java.io.IOException: Connection closed with -1 on reading size.
at com.cloud.utils.nio.Link.doHandshake(Link.java:513)
at com.cloud.utils.nio.NioClient.init(NioClient.java:80)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:88)
at com.cloud.agent.Agent.reconnect(Agent.java:413)
at com.cloud.agent.Agent$ServerHandler.doTask(Agent.java:869)
at com.cloud.utils.nio.Task.call(Task.java:83)
at com.cloud.utils.nio.Task.call(Task.java:29)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-03-03 17:50:05,210 INFO [utils.exception.CSExceptionErrorCode] (Agent-Handler-3:null) (logid:) Could not find exception: com.cloud.utils.exception.NioConnectionException in error code list for exceptions
2016-03-03 17:50:05,211 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again...
2016-03-03 17:50:10,211 INFO [cloud.agent.Agent] (Agent-Handler-3:null) (logid:) Reconnecting...
2016-03-03 17:50:10,211 INFO [utils.nio.NioClient] (Agent-Handler-3:null) (logid:) Connecting to 10.103.0.154:8250
This has been tested on 4.7.1 and master.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/myENA/cloudstack CLOUDSTACK-9285
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1429.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1429
----
commit c17250da5aca931438b5b031839e1dad20955f16
Author: Simon Weller <sw...@ena.com>
Date: 2016-03-04T13:40:27Z
CLOUDSTACK-9285 - Agent thows an exception that can never be recovered from
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9285 - Agent throws an excepti...
Posted by kiwiflyer <gi...@git.apache.org>.
Github user kiwiflyer commented on the pull request:
https://github.com/apache/cloudstack/pull/1429#issuecomment-198513018
Closing this PR, as this issue has been address more cleanly in my other PR against 4.7 - https://github.com/apache/cloudstack/pull/1430
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9285 - Agent throws an excepti...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1429#issuecomment-192864151
@kiwiflyer your change makes sense to me, (un)fortunately we have quite strict checkstyle settings. Can you replace the tab with spaces?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9285 - Agent throws an excepti...
Posted by kiwiflyer <gi...@git.apache.org>.
Github user kiwiflyer closed the pull request at:
https://github.com/apache/cloudstack/pull/1429
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9285 - Agent throws an excepti...
Posted by kiwiflyer <gi...@git.apache.org>.
Github user kiwiflyer commented on the pull request:
https://github.com/apache/cloudstack/pull/1429#issuecomment-192913591
Daan,
Sorry about that. I've fixed it, pushed the commit and checks are now passing.
Simon
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9285 - Agent throws an excepti...
Posted by remibergsma <gi...@git.apache.org>.
Github user remibergsma commented on the pull request:
https://github.com/apache/cloudstack/pull/1429#issuecomment-193384072
@kiwiflyer Thanks, have seen this as well and will try your fix when I have some time :-)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request: CLOUDSTACK-9285 - Agent throws an excepti...
Posted by DaanHoogland <gi...@git.apache.org>.
Github user DaanHoogland commented on the pull request:
https://github.com/apache/cloudstack/pull/1429#issuecomment-193220496
@kiwiflyer Simon, I'm a nag but can you recreate this PR against 4.7? We've confirmed it on 4.7 s well. I think it was introduced in 4.6 even but that release is intermediate in our release process change so we might want to skip it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---