You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2020/06/23 21:08:04 UTC
[GitHub] [cloudstack] gaaray2k opened a new issue #4168: Problems when using multiple management servers.
gaaray2k opened a new issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168
CS version: 4.14
setup:
LB: haproxy
2 management nodes
2 compute nodes
I am playing around with adding a second mgmt host to my setup. the problem is that when I configured the haproxy LB for port 8250, the management servers are seeing the compute node certificates but the mgmt servers are seeing the ip of the load balancer as the source of the request. because of this the connection is failing. the ip of the LB isnt listed in the cert therefore getting the error below. I changed the 'host' property to be the LB ip and I see the nodes trying to connect but they fail to do so.
how is a setup like supposed to be configured?
log on management:
2020-06-22 05:04:27,010 DEBUG [o.a.c.c.p.RootCACustomTrustManager] (pool-222-thread-1:null) (logid:) A client/agent attempting connection from address=10.0.0.45 has presented these certificate(s):
Certificate [1] :
Serial: 71a519c2e7762bf4
Not Before:Thu Jun 18 22:40:36 EDT 2020
Not After:Sat Jun 19 10:40:36 EDT 2021
Signature Algorithm:SHA256withRSA
Version:3
Subject DN:C=cloudstack, O=cloudstack, OU=cloudstack, CN=cs-compute-01
Issuer DN:CN=ca.cloudstack.apache.org
Alternative Names:[[7, fe80:0:0:0:3801:1f5e:9ab2:bc9e], [7, fe80:0:0:0:19c:6dc6:62e9:59cb], [7, fe80:0:0:0:20c:29ff:fecf:7565], [7, 192.168.122.1], [7, 10.10.0.52], [7, fe80:0:0:0:c400:4519:2866:f915], [7, 10.0.0.52], [2, 10.0.0.52]]
Certificate [2] :
Serial: 7ab4d7014adc5901
Not Before:Thu Jun 18 21:19:22 EDT 2020
Not After:Sun Jun 12 09:19:22 EDT 2050
Signature Algorithm:SHA256withRSA
Version:3
Subject DN:CN=ca.cloudstack.apache.org
Issuer DN:CN=ca.cloudstack.apache.org
Alternative Names:null
2020-06-22 05:04:27,014 ERROR [o.a.c.c.p.RootCACustomTrustManager] (pool-222-thread-1:null) (logid:) Certificate ownership verification failed for client: 10.0.0.45
2020-06-22 05:04:27,014 ERROR [c.c.u.n.Link] (AgentManager-SSLHandshakeHandler-79:null) (logid:) SSL error caught during wrap data: Certificate ownership verification failed for client: 10.0.0.45, for local address=/10.0.0.66:8250, remote address=/10.0.0.45:46844.
2020-06-22 05:04:27,018 INFO [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-12:null) (logid:) Connection from /10.0.0.45 closed but no cleanup was done.
log on compute node:
2020-06-23 16:25:36,084 INFO [cloud.agent.Agent] (main:null) (logid:) Connecting to host:10.0.0.45
2020-06-23 16:25:36,085 INFO [utils.nio.NioClient] (main:null) (logid:) Connecting to 10.0.0.45:8250
2020-06-23 16:25:36,086 INFO [utils.nio.Link] (main:null) (logid:) Conf file found: /etc/cloudstack/agent/agent.properties
2020-06-23 16:25:36,396 ERROR [utils.nio.Link] (main:null) (logid:) SSL error caught during unwrap data: Received fatal alert: certificate_unknown, for local address=/10.0.0.52:34340, remote address=/10.0.0.45:8250. The client may have invalid ca-certificates.
2020-06-23 16:25:36,396 ERROR [utils.nio.NioClient] (main:null) (logid:) SSL Handshake failed while connecting to host: 10.0.0.45 port: 8250
2020-06-23 16:25:36,397 ERROR [utils.nio.NioConnection] (main:null) (logid:) Unable to initialize the threads.
java.io.IOException: SSL Handshake failed while connecting to host: 10.0.0.45 port: 8250
at com.cloud.utils.nio.NioClient.init(NioClient.java:67)
at com.cloud.utils.nio.NioConnection.start(NioConnection.java:95)
at com.cloud.agent.Agent.start(Agent.java:293)
at com.cloud.agent.AgentShell.launchNewAgent(AgentShell.java:455)
at com.cloud.agent.AgentShell.launchAgentFromClassInfo(AgentShell.java:422)
at com.cloud.agent.AgentShell.launchAgent(AgentShell.java:406)
at com.cloud.agent.AgentShell.start(AgentShell.java:512)
at com.cloud.agent.AgentShell.main(AgentShell.java:547)
2020-06-23 16:25:36,397 INFO [utils.nio.NioClient] (main:null) (logid:) NioClient connection closed
2020-06-23 16:25:36,398 INFO [cloud.agent.Agent] (main:null) (logid:) Attempted to connect to the server, but received an unexpected exception, trying again...
LB config:
frontend agent_front
mode tcp
option forwardfor
option tcplog
cookie SESSIONID prefix
bind 10.0.0.45:8250
stats uri /haproxy?stats
default_backend agent_back
backend agent_back
mode tcp
balance source
stats enable
stats auth x:x
server cs1-agent 10.0.0.66:8250 check cookie p
server cs2-agent 10.0.0.64:8250 check cookie l
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] gaaray2k commented on issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
gaaray2k commented on issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168#issuecomment-648822682
I followed this doc and I see that the agents are being load balanced between the two mgmt servers, but I found another issue. when the console proxy sys vm connects to one mgmt server, I cannot open console from the other mgmt server. I get the message below. that is a problem when the load balancer for the frontend sends a user to the mgmt server that isnt being used by the console proxy vm.
'Access is denied for the console session. Please close the window and retry again'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168#issuecomment-668079540
> I am not using haproxy. I am using the management server internal load balancer. Haproxy for agent connections are not necessary. The problem is that when a MGMT server fails, the agents connecting to that server don't detect the failure and stay "connected" to the failed server.
@gaaray2k ok. you mentioned "LB: haproxy" in description, maybe it changed.
ps: it is suggested to use 3 or more management servers for HA.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] gaaray2k commented on issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
gaaray2k commented on issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168#issuecomment-667996960
I am not using haproxy. I am using the management server internal load balancer. Haproxy for agent connections are not necessary. The problem is that when a MGMT server fails, the agents connecting to that server don't detect the failure and stay "connected" to the failed server.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] gaaray2k edited a comment on issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
gaaray2k edited a comment on issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168#issuecomment-648822682
I followed this doc and I see that the agents are being load balanced between the two mgmt servers, but I found another issue. when the console proxy sys vm connects to one mgmt server, I cannot open console from the other mgmt server. I get the message below.. that is a problem when the load balancer for the frontend sends a user to the mgmt server that isnt being used by the console proxy vm.
'Access is denied for the console session. Please close the window and retry again'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] gaaray2k commented on issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
gaaray2k commented on issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168#issuecomment-667540715
is there a setting I need to enable for the agents to failover to a new mgmt server in case of a failure? right now this isnt happening. I shutdown the mgmt server where the agents were connected and they all went to disconnected state. I waited about 10 mins but they didnt reconnect to another server.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] gaaray2k edited a comment on issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
gaaray2k edited a comment on issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168#issuecomment-648822682
I followed this doc and I see that the agents are being load balanced between the two mgmt servers, but I found another issue. when the console proxy sys vm connects to one mgmt server, I cannot open console from the other mgmt server. I get the message below. only after opening a console from the working mgmt server a console would open from the other(I guess a cookie or something in the browser). that is a problem when the load balancer for the frontend sends a user to the mgmt server that isnt being used by the console proxy vm.
'Access is denied for the console session. Please close the window and retry again'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168#issuecomment-667989968
the sessions are not shared between mgt servers.
you have to enable sticky sessions in your haproxy config
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] gaaray2k removed a comment on issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
gaaray2k removed a comment on issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168#issuecomment-648822682
I followed this doc and I see that the agents are being load balanced between the two mgmt servers, but I found another issue. when the console proxy sys vm connects to one mgmt server, I cannot open console from the other mgmt server. I get the message below.. that is a problem when the load balancer for the frontend sends a user to the mgmt server that isnt being used by the console proxy vm.
'Access is denied for the console session. Please close the window and retry again'
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168#issuecomment-648530929
@gaaray2k no need to use external LB now as cloudstack has an internal software based LB, use comma separated list of mgmt IPs in the 'host' global settings. For existing kvm hosts, fix the same in agent.properties. See https://cwiki.apache.org/confluence/display/CLOUDSTACK/Multiple+Management+Servers+Support+on+agents
and
https://www.shapeblue.com/software-based-agent-lb-for-cloudstack/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] gaaray2k commented on issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
gaaray2k commented on issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168#issuecomment-668118811
I have there MGMT servers. The problem is if one fails the agents aren't reconnecting on their own to the other 2. I have to restart the agents for them to reconnect
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] gaaray2k closed issue #4168: Problems when using multiple management servers.
Posted by GitBox <gi...@apache.org>.
gaaray2k closed issue #4168:
URL: https://github.com/apache/cloudstack/issues/4168
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org