You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Yiping Zhang <yz...@marketo.com> on 2015/07/09 20:22:01 UTC
[Urgent]: xenserver hosts stuck in alert state after 4.3.2 ->
4.5.1 upgrade
Hi, all:
We just did an upgrade from CS 4.3.2 -> 4.5.1. Our environment is rhel 6 + Adv. Zone with SecurtyGroup + XenServer 6.2.
After the upgrade, all xenserver hosts were in alert state and MS can’t connect to them with following errors:
INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Host 10.0.100.25 OpaqueRef:996c575a-ad04-5f3f-cd6d-56b7daa16844: Host 10.0.100.25 is already setup.
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Failed to configure brige firewall
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Check host 10.0.100.25 for CSP is installed or not and check network mode for bridge
WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-13:ctx-b4517020) Unable to setup agent 5 due to Failed to configure brige firewall
INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-13:ctx-b4517020) Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions
WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-13:ctx-b4517020) Monitor XcpServerDiscoverer says there is an error in the connect process for 5 due to Reinitialize agent after setup.
INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-13:ctx-b4517020) Host 5 is disconnecting with event AgentDisconnected
WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-13:ctx-b4517020) Unable to connect due to
com.cloud.exception.ConnectionException: Reinitialize agent after setup.
at com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer.processConnect(XcpServerDiscoverer.java:621)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:539)
at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1447)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1794)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1920)
at sun.reflect.GeneratedMethodAccessor142.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy149.createHostAndAgent(Unknown Source)
at com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.runInContext(AgentManagerImpl.java:1078)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
The error implies that it was expecting linux bridge on XenServer and couldn’t find it. After checking on XenServer hosts, we have been using openvswitch instead of linux bridge as network backend all this time.
So the question comes down to:
1. If CS advanced zone with SecurityGroup only supported on XenServer using linux bridge backend, why did 4.3.x allow us to work with unsupported configurations for so long without any apparent issues, and does CS 4.5.1 now enforce this requirement ?
2. If we switch network backend from openvswitch to linux bridge, would this fix the problem? We are hoping to avoid this step, as it requires rebooting all XenServer hosts and shuffling around hundreds of VM instances.
3. Is there any other solutions to make newly upgraded cloudstack 4.5.1 management server to reconnect to our XenServer hosts ?
Thanks in advance,
Yiping
Re: [Urgent]: xenserver hosts stuck in alert state after 4.3.2 ->
4.5.1 upgrade
Posted by Nux! <nu...@li.nux.ro>.
I cannot be sure without (being able of) reading code, but I'm thinking there were no proper checks in place for this within Cloudstack until now.
Your instances probably worked somehow with the openvswitch bridge and you did not realise the problem since you have not actually used the security groups.
If that's the case I'd just downgrade to whatever version you tried last that worked and give myself some more time to plan this out in the longer term..
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Yiping Zhang" <yz...@marketo.com>
> To: users@cloudstack.apache.org
> Sent: Friday, 10 July, 2015 01:28:44
> Subject: Re: [Urgent]: xenserver hosts stuck in alert state after 4.3.2 -> 4.5.1 upgrade
> Hi, Lucian:
>
> Thanks for the reply. When I said it worked all this time I really meant
> that the CS instance worked as expected for what we were doing with it,
> not to mean that the SecurityGroup feature worked.
>
> To be honest, we did not really use security group feature per se, as
> this is a private cloud running on our own networks and we picked advanced
> networking with SG only to avoid assigning a public network IP range for
> guests. The CS instance went through upgrades from 4.3.0 -> 4.3.1 ->
> 4.3.2 without a hiccup, until we tried to upgrade to 4.5.1.
>
> Yiping
>
>
> On 7/9/15, 12:31 PM, "Nux!" <nu...@li.nux.ro> wrote:
>
>>Hello,
>>
>>As far as I can tell, Xenserver has always required the network to be in
>>bridge mode for security groups to work, at least since 4.1 that I've
>>been playing with. Not sure how exactly it was working in your case ...
>>did you actually test the firewall rules were doing anything? (Sorry for
>>dumb question)
>>
>>Lucian
>>
>>--
>>Sent from the Delta quadrant using Borg technology!
>>
>>Nux!
>>www.nux.ro
>>
>>----- Original Message -----
>>> From: "Yiping Zhang" <yz...@marketo.com>
>>> To: users@cloudstack.apache.org
>>> Sent: Thursday, 9 July, 2015 19:22:01
>>> Subject: [Urgent]: xenserver hosts stuck in alert state after 4.3.2 ->
>>>4.5.1 upgrade
>>
>>> Hi, all:
>>>
>>> We just did an upgrade from CS 4.3.2 -> 4.5.1. Our environment is rhel
>>>6 + Adv.
>>> Zone with SecurtyGroup + XenServer 6.2.
>>>
>>> After the upgrade, all xenserver hosts were in alert state and MS can¹t
>>>connect
>>> to them with following errors:
>>>
>>>
>>> INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Host
>>> 10.0.100.25 OpaqueRef:996c575a-ad04-5f3f-cd6d-56b7daa16844: Host
>>>10.0.100.25 is
>>> already setup.
>>>
>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce)
>>>Failed to
>>> configure brige firewall
>>>
>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Check
>>>host
>>> 10.0.100.25 for CSP is installed or not and check network mode for
>>>bridge
>>>
>>> WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-13:ctx-b4517020)
>>>Unable to
>>> setup agent 5 due to Failed to configure brige firewall
>>>
>>> INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-13:ctx-b4517020)
>>>Could not
>>> find exception: com.cloud.exception.ConnectionException in error code
>>>list for
>>> exceptions
>>>
>>> WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-13:ctx-b4517020) Monitor
>>> XcpServerDiscoverer says there is an error in the connect process for 5
>>>due to
>>> Reinitialize agent after setup.
>>>
>>> INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-13:ctx-b4517020) Host 5
>>>is
>>> disconnecting with event AgentDisconnected
>>>
>>> WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-13:ctx-b4517020)
>>>Unable to
>>> connect due to
>>>
>>> com.cloud.exception.ConnectionException: Reinitialize agent after setup.
>>>
>>> at
>>>
>>>com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer.processConn
>>>ect(XcpServerDiscoverer.java:621)
>>>
>>> at
>>>
>>>com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(Agent
>>>ManagerImpl.java:539)
>>>
>>> at
>>>
>>>com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentMa
>>>nagerImpl.java:1447)
>>>
>>> at
>>>
>>>com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManager
>>>Impl.java:1794)
>>>
>>> at
>>>
>>>com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManager
>>>Impl.java:1920)
>>>
>>> at sun.reflect.GeneratedMethodAccessor142.invoke(Unknown Source)
>>>
>>> at
>>>
>>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>>>mpl.java:43)
>>>
>>> at java.lang.reflect.Method.invoke(Method.java:606)
>>>
>>> at
>>>
>>>org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(A
>>>opUtils.java:317)
>>>
>>> at
>>>
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpo
>>>int(ReflectiveMethodInvocation.java:183)
>>>
>>> at
>>>
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl
>>>ectiveMethodInvocation.java:150)
>>>
>>> at
>>>
>>>org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Ex
>>>poseInvocationInterceptor.java:91)
>>>
>>> at
>>>
>>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl
>>>ectiveMethodInvocation.java:172)
>>>
>>> at
>>>
>>>org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAop
>>>Proxy.java:204)
>>>
>>> at com.sun.proxy.$Proxy149.createHostAndAgent(Unknown Source)
>>>
>>> at
>>>
>>>com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.runInContext(A
>>>gentManagerImpl.java:1078)
>>>
>>> at
>>>
>>>org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Manage
>>>dContextRunnable.java:49)
>>>
>>> at
>>>
>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(D
>>>efaultManagedContext.java:56)
>>>
>>> at
>>>
>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWith
>>>Context(DefaultManagedContext.java:103)
>>>
>>> at
>>>
>>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithC
>>>ontext(DefaultManagedContext.java:53)
>>>
>>> at
>>>
>>>org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedC
>>>ontextRunnable.java:46)
>>>
>>> at
>>>
>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>>:1145)
>>>
>>> at
>>>
>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>>a:615)
>>>
>>> at java.lang.Thread.run(Thread.java:745)
>>>
>>> The error implies that it was expecting linux bridge on XenServer and
>>>couldn¹t
>>> find it. After checking on XenServer hosts, we have been using
>>>openvswitch
>>> instead of linux bridge as network backend all this time.
>>>
>>> So the question comes down to:
>>>
>>> 1. If CS advanced zone with SecurityGroup only supported on XenServer
>>>using
>>> linux bridge backend, why did 4.3.x allow us to work with unsupported
>>> configurations for so long without any apparent issues, and does CS
>>>4.5.1 now
>>> enforce this requirement ?
>>> 2. If we switch network backend from openvswitch to linux bridge,
>>>would this
>>> fix the problem? We are hoping to avoid this step, as it requires
>>>rebooting
>>> all XenServer hosts and shuffling around hundreds of VM instances.
>>> 3. Is there any other solutions to make newly upgraded cloudstack 4.5.1
>>> management server to reconnect to our XenServer hosts ?
>>>
>>> Thanks in advance,
>>>
> >> Yiping
Re: [Urgent]: xenserver hosts stuck in alert state after 4.3.2 ->
4.5.1 upgrade
Posted by Yiping Zhang <yz...@marketo.com>.
Hi, Lucian:
Thanks for the reply. When I said it worked all this time I really meant
that the CS instance worked as expected for what we were doing with it,
not to mean that the SecurityGroup feature worked.
To be honest, we did not really use security group feature per se, as
this is a private cloud running on our own networks and we picked advanced
networking with SG only to avoid assigning a public network IP range for
guests. The CS instance went through upgrades from 4.3.0 -> 4.3.1 ->
4.3.2 without a hiccup, until we tried to upgrade to 4.5.1.
Yiping
On 7/9/15, 12:31 PM, "Nux!" <nu...@li.nux.ro> wrote:
>Hello,
>
>As far as I can tell, Xenserver has always required the network to be in
>bridge mode for security groups to work, at least since 4.1 that I've
>been playing with. Not sure how exactly it was working in your case ...
>did you actually test the firewall rules were doing anything? (Sorry for
>dumb question)
>
>Lucian
>
>--
>Sent from the Delta quadrant using Borg technology!
>
>Nux!
>www.nux.ro
>
>----- Original Message -----
>> From: "Yiping Zhang" <yz...@marketo.com>
>> To: users@cloudstack.apache.org
>> Sent: Thursday, 9 July, 2015 19:22:01
>> Subject: [Urgent]: xenserver hosts stuck in alert state after 4.3.2 ->
>>4.5.1 upgrade
>
>> Hi, all:
>>
>> We just did an upgrade from CS 4.3.2 -> 4.5.1. Our environment is rhel
>>6 + Adv.
>> Zone with SecurtyGroup + XenServer 6.2.
>>
>> After the upgrade, all xenserver hosts were in alert state and MS can¹t
>>connect
>> to them with following errors:
>>
>>
>> INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Host
>> 10.0.100.25 OpaqueRef:996c575a-ad04-5f3f-cd6d-56b7daa16844: Host
>>10.0.100.25 is
>> already setup.
>>
>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce)
>>Failed to
>> configure brige firewall
>>
>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Check
>>host
>> 10.0.100.25 for CSP is installed or not and check network mode for
>>bridge
>>
>> WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-13:ctx-b4517020)
>>Unable to
>> setup agent 5 due to Failed to configure brige firewall
>>
>> INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-13:ctx-b4517020)
>>Could not
>> find exception: com.cloud.exception.ConnectionException in error code
>>list for
>> exceptions
>>
>> WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-13:ctx-b4517020) Monitor
>> XcpServerDiscoverer says there is an error in the connect process for 5
>>due to
>> Reinitialize agent after setup.
>>
>> INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-13:ctx-b4517020) Host 5
>>is
>> disconnecting with event AgentDisconnected
>>
>> WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-13:ctx-b4517020)
>>Unable to
>> connect due to
>>
>> com.cloud.exception.ConnectionException: Reinitialize agent after setup.
>>
>> at
>>
>>com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer.processConn
>>ect(XcpServerDiscoverer.java:621)
>>
>> at
>>
>>com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(Agent
>>ManagerImpl.java:539)
>>
>> at
>>
>>com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentMa
>>nagerImpl.java:1447)
>>
>> at
>>
>>com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManager
>>Impl.java:1794)
>>
>> at
>>
>>com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManager
>>Impl.java:1920)
>>
>> at sun.reflect.GeneratedMethodAccessor142.invoke(Unknown Source)
>>
>> at
>>
>>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI
>>mpl.java:43)
>>
>> at java.lang.reflect.Method.invoke(Method.java:606)
>>
>> at
>>
>>org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(A
>>opUtils.java:317)
>>
>> at
>>
>>org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpo
>>int(ReflectiveMethodInvocation.java:183)
>>
>> at
>>
>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl
>>ectiveMethodInvocation.java:150)
>>
>> at
>>
>>org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(Ex
>>poseInvocationInterceptor.java:91)
>>
>> at
>>
>>org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(Refl
>>ectiveMethodInvocation.java:172)
>>
>> at
>>
>>org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAop
>>Proxy.java:204)
>>
>> at com.sun.proxy.$Proxy149.createHostAndAgent(Unknown Source)
>>
>> at
>>
>>com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.runInContext(A
>>gentManagerImpl.java:1078)
>>
>> at
>>
>>org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(Manage
>>dContextRunnable.java:49)
>>
>> at
>>
>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(D
>>efaultManagedContext.java:56)
>>
>> at
>>
>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWith
>>Context(DefaultManagedContext.java:103)
>>
>> at
>>
>>org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithC
>>ontext(DefaultManagedContext.java:53)
>>
>> at
>>
>>org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedC
>>ontextRunnable.java:46)
>>
>> at
>>
>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java
>>:1145)
>>
>> at
>>
>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>a:615)
>>
>> at java.lang.Thread.run(Thread.java:745)
>>
>> The error implies that it was expecting linux bridge on XenServer and
>>couldn¹t
>> find it. After checking on XenServer hosts, we have been using
>>openvswitch
>> instead of linux bridge as network backend all this time.
>>
>> So the question comes down to:
>>
>> 1. If CS advanced zone with SecurityGroup only supported on XenServer
>>using
>> linux bridge backend, why did 4.3.x allow us to work with unsupported
>> configurations for so long without any apparent issues, and does CS
>>4.5.1 now
>> enforce this requirement ?
>> 2. If we switch network backend from openvswitch to linux bridge,
>>would this
>> fix the problem? We are hoping to avoid this step, as it requires
>>rebooting
>> all XenServer hosts and shuffling around hundreds of VM instances.
>> 3. Is there any other solutions to make newly upgraded cloudstack 4.5.1
>> management server to reconnect to our XenServer hosts ?
>>
>> Thanks in advance,
>>
>> Yiping
Re: [Urgent]: xenserver hosts stuck in alert state after 4.3.2 ->
4.5.1 upgrade
Posted by Nux! <nu...@li.nux.ro>.
Hello,
As far as I can tell, Xenserver has always required the network to be in bridge mode for security groups to work, at least since 4.1 that I've been playing with. Not sure how exactly it was working in your case ... did you actually test the firewall rules were doing anything? (Sorry for dumb question)
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Yiping Zhang" <yz...@marketo.com>
> To: users@cloudstack.apache.org
> Sent: Thursday, 9 July, 2015 19:22:01
> Subject: [Urgent]: xenserver hosts stuck in alert state after 4.3.2 -> 4.5.1 upgrade
> Hi, all:
>
> We just did an upgrade from CS 4.3.2 -> 4.5.1. Our environment is rhel 6 + Adv.
> Zone with SecurtyGroup + XenServer 6.2.
>
> After the upgrade, all xenserver hosts were in alert state and MS can’t connect
> to them with following errors:
>
>
> INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Host
> 10.0.100.25 OpaqueRef:996c575a-ad04-5f3f-cd6d-56b7daa16844: Host 10.0.100.25 is
> already setup.
>
> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Failed to
> configure brige firewall
>
> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-9:ctx-7fc226ce) Check host
> 10.0.100.25 for CSP is installed or not and check network mode for bridge
>
> WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-13:ctx-b4517020) Unable to
> setup agent 5 due to Failed to configure brige firewall
>
> INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-13:ctx-b4517020) Could not
> find exception: com.cloud.exception.ConnectionException in error code list for
> exceptions
>
> WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-13:ctx-b4517020) Monitor
> XcpServerDiscoverer says there is an error in the connect process for 5 due to
> Reinitialize agent after setup.
>
> INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-13:ctx-b4517020) Host 5 is
> disconnecting with event AgentDisconnected
>
> WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-13:ctx-b4517020) Unable to
> connect due to
>
> com.cloud.exception.ConnectionException: Reinitialize agent after setup.
>
> at
> com.cloud.hypervisor.xenserver.discoverer.XcpServerDiscoverer.processConnect(XcpServerDiscoverer.java:621)
>
> at
> com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:539)
>
> at
> com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1447)
>
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1794)
>
> at
> com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1920)
>
> at sun.reflect.GeneratedMethodAccessor142.invoke(Unknown Source)
>
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> at java.lang.reflect.Method.invoke(Method.java:606)
>
> at
> org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
>
> at
> org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>
> at
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
>
> at
> org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
>
> at com.sun.proxy.$Proxy149.createHostAndAgent(Unknown Source)
>
> at
> com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.runInContext(AgentManagerImpl.java:1078)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>
> at java.lang.Thread.run(Thread.java:745)
>
> The error implies that it was expecting linux bridge on XenServer and couldn’t
> find it. After checking on XenServer hosts, we have been using openvswitch
> instead of linux bridge as network backend all this time.
>
> So the question comes down to:
>
> 1. If CS advanced zone with SecurityGroup only supported on XenServer using
> linux bridge backend, why did 4.3.x allow us to work with unsupported
> configurations for so long without any apparent issues, and does CS 4.5.1 now
> enforce this requirement ?
> 2. If we switch network backend from openvswitch to linux bridge, would this
> fix the problem? We are hoping to avoid this step, as it requires rebooting
> all XenServer hosts and shuffling around hundreds of VM instances.
> 3. Is there any other solutions to make newly upgraded cloudstack 4.5.1
> management server to reconnect to our XenServer hosts ?
>
> Thanks in advance,
>
> Yiping