You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Carlos Reátegui <cr...@gmail.com> on 2014/06/28 05:55:21 UTC
4.4 upgrade issues
I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
-----
Notes on the docs:
8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
——
Issues:
Saw the following in catalina.out, not sure if it is an issues:
Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=…” on each of the system vms? Will CloudStack notice they are ton and start new ones?
Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
Re: 4.4 upgrade issues
Posted by ilya musayev <il...@gmail.com>.
Also, the password is salted with timestamp (i assume), it will look
different each time - but should decrypt correctly with proper key.
On 6/30/14, 1:16 PM, Carlos Reátegui wrote:
> I found the comments in: https://issues.apache.org/jira/browse/CLOUDSTACK-3990 useful but how do I find out the database key so that I can set the pw.
>
> Also in looking at my previous backups for the host_details table it seems like the password entry changes on a regular basis.
>
> Is there something the keeps updating the db key and re-ecrypts the host passwords?
>
> On Jun 30, 2014, at 1:01 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> Hi All,
>>
>> I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
>>
>> thanks.
>> Carlos
>>
>>
>> 2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
>> 2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>> 2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>> 2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> 2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
>> 2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
>> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
>> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
>> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
>> at java.util.TimerThread.mainLoop(Timer.java:555)
>> at java.util.TimerThread.run(Timer.java:505)
>> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
>> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
>> at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
>> ... 17 more
>> 2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
>> 2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
>> 2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
>> 2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
>> 2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
>>
>>
>>
>> On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>>
>>> Hi Sudha,
>>> Thanks for checking in. I was out for the weekend and just getting back to this now.
>>>
>>> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>>>
>>> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>>>
>>> thanks,
>>> Carlos
>>>
>>>
>>>
>>> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>>>
>>>> Hi Carlos,
>>>>
>>>> Were you able to resolve the following? Was your upgrade successful?
>>>>
>>>> Thanks
>>>> /sudha
>>>>
>>>> -----Original Message-----
>>>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>>>> Sent: Friday, June 27, 2014 8:55 PM
>>>> To: CloudStack-Users
>>>> Cc: dev@cloudstack.apache.org
>>>> Subject: 4.4 upgrade issues
>>>>
>>>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>>>
>>>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>>>
>>>> -----
>>>> Notes on the docs:
>>>>
>>>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>>>
>>>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>>>
>>>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>>>
>>>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>>>
>>>> --
>>>> Issues:
>>>>
>>>> Saw the following in catalina.out, not sure if it is an issues:
>>>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>>>
>>>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>>>
>>>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>>>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>>>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>>>
>>>>
>>>>
>>>>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
An update on this.
I finally succeeded in getting my hosts out of alert state by reverting to an earlier version of the kernel (see other thread). Unfortunately when it came up I realized I had installed the 4.3 systemvm template and not the 4.4 one so I just reverted to my 4.1.1 installation and did the upgrade to 4.3 which appears to work. I am seeing some errors in the log which I’ll post to a separate thread. I’ll wait to test 4.4 when it is released.
With regards to the db.properties file. I looked into it in more detail and the problem was that the file had somehow gotten re-ordered in a random fashion which is why I could not make sense out of the diff during the install. Not sure how that happened but my concern mentioned below is probably a non-issue.
regards,
-Carlos
On Jun 30, 2014, at 5:11 PM, Carlos Reátegui <cr...@gmail.com> wrote:
> I set encryption to none in db.properties and updated the passwords in host_details to unencrypted versions so I could make progress.
>
> I don’t know what exactly the problem was but this is probably something that needs better testing. I’m pretty sure I had all the encryption stuff correct in the db.properties file but could not get it to work.
>
> It would be nice if there was a specialized merging utility for the db.properties given the change in the organization of the file. I am guessing if the file had not been reorganized it would have been more obvious how to merge the 2 and I may have avoided this issue.
>
> Now my hosts come up in an alert state and I’m not sure where to go from here. Please note I am not using bridge mode because I wanted to to a 4 nic bridge which bridge does not allow (only 2 nics). This was working fine in 4.1 so hopefully this is not a requirement for 4.4. I am not using security groups which was my understanding is what requires bridge networking:
>
> The error in the log is this:
> 2014-06-30 14:06:50,073 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-1:ctx-35941dc7) Failed to configure brige firewall
> 2014-06-30 14:06:50,073 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-1:ctx-35941dc7) Check host 172.30.45.32 for CSP is installed or not and check network mode for bridge
> 2014-06-30 14:06:50,074 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: Response Received:
> 2014-06-30 14:06:50,075 DEBUG [c.c.a.t.Request] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: Processing: { Ans: , MgmtId: 233845174730253, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api
> .SetupAnswer":{"_reconnect":true,"result":false,"details":"Failed to configure brige firewall","wait":0}}] }
> 2014-06-30 14:06:50,075 DEBUG [c.c.a.t.Request] (AgentTaskPool-2:ctx-b360d1bb) Seq 2-6232418934327345153: Received: { Ans: , MgmtId: 233845174730253, via: 2, Ver: v1, Flags: 110, { SetupAnswer } }
> 2014-06-30 14:06:50,076 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: No more commands found
> 2014-06-30 14:06:50,076 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-2:ctx-b360d1bb) Unable to setup agent 2 due to Failed to configure brige firewall
> 2014-06-30 14:06:50,079 INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-2:ctx-b360d1bb) Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions
> 2014-06-30 14:06:50,079 WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Monitor XcpServerDiscoverer says there is an error in the connect process for 2 due to Reinitialize agent after se
> tup.
> 2014-06-30 14:06:50,079 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Host 2 is disconnecting with event AgentDisconnected
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) The next status of agent 2is Alert, current status is Connecting
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Deregistering link for 2 with state Alert
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Remove Agent : 2
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.DirectAgentAttache] (AgentTaskPool-2:ctx-b360d1bb) Processing disconnect 2(srvengxen02)
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: org.apache.cloudstack.engine.orchestration.NetworkOrchestrator
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.deploy.DeploymentPlanningManagerImpl
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
> 2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
> 2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl
> 2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener
> 2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
> 2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
> 2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
> 2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
> 2014-06-30 14:06:50,093 DEBUG [c.c.n.NetworkUsageManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Disconnected called on 2 with status Alert
> 2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.agent.manager.AgentManagerImpl$BehindOnPingListener
> 2014-06-30 14:06:50,093 DEBUG [c.c.h.Status] (AgentTaskPool-2:ctx-b360d1bb) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 2, name = srvengxen02]
> 2014-06-30 14:06:50,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Notifying other nodes of to disconnect
> 2014-06-30 14:06:50,108 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:ctx-59808af2) Copying /usr/share/cloudstack-management/webapps/client/WEB-INF/classes/scripts/vm/hypervisor/xenserver/xenserver60/../../../../network/domr//router_proxy.sh to /opt/cloud/bin on 172.30.45.32 with permission 0755
> 2014-06-30 14:06:50,108 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:ctx-59808af2) Unable to create destination path: /opt/cloud/bin on 172.30.45.32 but trying anyway
> 2014-06-30 14:06:50,110 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Unable to connect due to
> com.cloud.exception.ConnectionException: Reinitialize agent after setup.
> at com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer.processConnect(XcpServerDiscoverer.java:656)
> at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:514)
> at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1427)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1765)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1891)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>
>
>
>
> On Jun 30, 2014, at 1:57 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> Making a little progress but still stuck…
>>
>> I realized that when I did the upgrade it had asked me if to keep the old dp.properties or use the new one. The structure of the file seemed different enough and I did not recall using anything but the defaults so I went ahead and told it to use the new one. Seems this was not the right thing to do.
>>
>> I have updated the password/ecryption settings to match the old file but it is still not working. Now I am getting stuck here:
>>
>> 2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Found parameter routing unique name null
>> 2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Use console proxy template : routing
>> 2014-06-30 13:50:32,143 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HostPodDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,157 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DedicatedResourceDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,168 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HypervisorCapabilitiesDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 100 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,175 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = UserDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 5000 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 300 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,180 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = ServiceOfferingDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,187 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DataCenterDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,188 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = Ip Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,189 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = vnet Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,198 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = VlanDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,232 DEBUG [c.c.u.c.DBEncryptionUtil] (main:null) Error while decrypting: true
>>
>> The key is still the default password and I have decrypted all the ENC parameters from the db.properties file and they seem ok. What am I missing?
>>
>> thanks,
>> Carlos
>>
>>
>> On Jun 30, 2014, at 1:16 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>>
>>> I found the comments in: https://issues.apache.org/jira/browse/CLOUDSTACK-3990 useful but how do I find out the database key so that I can set the pw.
>>>
>>> Also in looking at my previous backups for the host_details table it seems like the password entry changes on a regular basis.
>>>
>>> Is there something the keeps updating the db key and re-ecrypts the host passwords?
>>>
>>> On Jun 30, 2014, at 1:01 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
>>>>
>>>> thanks.
>>>> Carlos
>>>>
>>>>
>>>> 2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
>>>> 2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>>>> 2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>>>> 2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>>> 2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
>>>> 2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
>>>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
>>>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
>>>> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
>>>> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
>>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
>>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
>>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
>>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
>>>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
>>>> at java.util.TimerThread.mainLoop(Timer.java:555)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>>> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
>>>> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
>>>> at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
>>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
>>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
>>>> ... 17 more
>>>> 2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
>>>> 2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
>>>> 2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
>>>> 2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
>>>> 2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
>>>>
>>>>
>>>>
>>>> On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>>>>
>>>>> Hi Sudha,
>>>>> Thanks for checking in. I was out for the weekend and just getting back to this now.
>>>>>
>>>>> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>>>>>
>>>>> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>>>>>
>>>>> thanks,
>>>>> Carlos
>>>>>
>>>>>
>>>>>
>>>>> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>>>>>
>>>>>> Hi Carlos,
>>>>>>
>>>>>> Were you able to resolve the following? Was your upgrade successful?
>>>>>>
>>>>>> Thanks
>>>>>> /sudha
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>>>>>> Sent: Friday, June 27, 2014 8:55 PM
>>>>>> To: CloudStack-Users
>>>>>> Cc: dev@cloudstack.apache.org
>>>>>> Subject: 4.4 upgrade issues
>>>>>>
>>>>>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>>>>>
>>>>>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>>>>>
>>>>>> -----
>>>>>> Notes on the docs:
>>>>>>
>>>>>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>>>>>
>>>>>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>>>>>
>>>>>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>>>>>
>>>>>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>>>>>
>>>>>> --
>>>>>> Issues:
>>>>>>
>>>>>> Saw the following in catalina.out, not sure if it is an issues:
>>>>>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>>>>>
>>>>>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>>>>>
>>>>>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>>>>>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>>>>>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
An update on this.
I finally succeeded in getting my hosts out of alert state by reverting to an earlier version of the kernel (see other thread). Unfortunately when it came up I realized I had installed the 4.3 systemvm template and not the 4.4 one so I just reverted to my 4.1.1 installation and did the upgrade to 4.3 which appears to work. I am seeing some errors in the log which I’ll post to a separate thread. I’ll wait to test 4.4 when it is released.
With regards to the db.properties file. I looked into it in more detail and the problem was that the file had somehow gotten re-ordered in a random fashion which is why I could not make sense out of the diff during the install. Not sure how that happened but my concern mentioned below is probably a non-issue.
regards,
-Carlos
On Jun 30, 2014, at 5:11 PM, Carlos Reátegui <cr...@gmail.com> wrote:
> I set encryption to none in db.properties and updated the passwords in host_details to unencrypted versions so I could make progress.
>
> I don’t know what exactly the problem was but this is probably something that needs better testing. I’m pretty sure I had all the encryption stuff correct in the db.properties file but could not get it to work.
>
> It would be nice if there was a specialized merging utility for the db.properties given the change in the organization of the file. I am guessing if the file had not been reorganized it would have been more obvious how to merge the 2 and I may have avoided this issue.
>
> Now my hosts come up in an alert state and I’m not sure where to go from here. Please note I am not using bridge mode because I wanted to to a 4 nic bridge which bridge does not allow (only 2 nics). This was working fine in 4.1 so hopefully this is not a requirement for 4.4. I am not using security groups which was my understanding is what requires bridge networking:
>
> The error in the log is this:
> 2014-06-30 14:06:50,073 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-1:ctx-35941dc7) Failed to configure brige firewall
> 2014-06-30 14:06:50,073 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-1:ctx-35941dc7) Check host 172.30.45.32 for CSP is installed or not and check network mode for bridge
> 2014-06-30 14:06:50,074 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: Response Received:
> 2014-06-30 14:06:50,075 DEBUG [c.c.a.t.Request] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: Processing: { Ans: , MgmtId: 233845174730253, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api
> .SetupAnswer":{"_reconnect":true,"result":false,"details":"Failed to configure brige firewall","wait":0}}] }
> 2014-06-30 14:06:50,075 DEBUG [c.c.a.t.Request] (AgentTaskPool-2:ctx-b360d1bb) Seq 2-6232418934327345153: Received: { Ans: , MgmtId: 233845174730253, via: 2, Ver: v1, Flags: 110, { SetupAnswer } }
> 2014-06-30 14:06:50,076 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: No more commands found
> 2014-06-30 14:06:50,076 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-2:ctx-b360d1bb) Unable to setup agent 2 due to Failed to configure brige firewall
> 2014-06-30 14:06:50,079 INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-2:ctx-b360d1bb) Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions
> 2014-06-30 14:06:50,079 WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Monitor XcpServerDiscoverer says there is an error in the connect process for 2 due to Reinitialize agent after se
> tup.
> 2014-06-30 14:06:50,079 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Host 2 is disconnecting with event AgentDisconnected
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) The next status of agent 2is Alert, current status is Connecting
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Deregistering link for 2 with state Alert
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Remove Agent : 2
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.DirectAgentAttache] (AgentTaskPool-2:ctx-b360d1bb) Processing disconnect 2(srvengxen02)
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
> 2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: org.apache.cloudstack.engine.orchestration.NetworkOrchestrator
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.deploy.DeploymentPlanningManagerImpl
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl
> 2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
> 2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
> 2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl
> 2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener
> 2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
> 2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
> 2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
> 2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
> 2014-06-30 14:06:50,093 DEBUG [c.c.n.NetworkUsageManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Disconnected called on 2 with status Alert
> 2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.agent.manager.AgentManagerImpl$BehindOnPingListener
> 2014-06-30 14:06:50,093 DEBUG [c.c.h.Status] (AgentTaskPool-2:ctx-b360d1bb) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 2, name = srvengxen02]
> 2014-06-30 14:06:50,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Notifying other nodes of to disconnect
> 2014-06-30 14:06:50,108 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:ctx-59808af2) Copying /usr/share/cloudstack-management/webapps/client/WEB-INF/classes/scripts/vm/hypervisor/xenserver/xenserver60/../../../../network/domr//router_proxy.sh to /opt/cloud/bin on 172.30.45.32 with permission 0755
> 2014-06-30 14:06:50,108 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:ctx-59808af2) Unable to create destination path: /opt/cloud/bin on 172.30.45.32 but trying anyway
> 2014-06-30 14:06:50,110 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Unable to connect due to
> com.cloud.exception.ConnectionException: Reinitialize agent after setup.
> at com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer.processConnect(XcpServerDiscoverer.java:656)
> at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:514)
> at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1427)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1765)
> at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1891)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
>
>
>
>
> On Jun 30, 2014, at 1:57 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> Making a little progress but still stuck…
>>
>> I realized that when I did the upgrade it had asked me if to keep the old dp.properties or use the new one. The structure of the file seemed different enough and I did not recall using anything but the defaults so I went ahead and told it to use the new one. Seems this was not the right thing to do.
>>
>> I have updated the password/ecryption settings to match the old file but it is still not working. Now I am getting stuck here:
>>
>> 2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Found parameter routing unique name null
>> 2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Use console proxy template : routing
>> 2014-06-30 13:50:32,143 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HostPodDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,157 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DedicatedResourceDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,168 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HypervisorCapabilitiesDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 100 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,175 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = UserDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 5000 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 300 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,180 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = ServiceOfferingDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,187 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DataCenterDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,188 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = Ip Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,189 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = vnet Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,198 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = VlanDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
>> 2014-06-30 13:50:32,232 DEBUG [c.c.u.c.DBEncryptionUtil] (main:null) Error while decrypting: true
>>
>> The key is still the default password and I have decrypted all the ENC parameters from the db.properties file and they seem ok. What am I missing?
>>
>> thanks,
>> Carlos
>>
>>
>> On Jun 30, 2014, at 1:16 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>>
>>> I found the comments in: https://issues.apache.org/jira/browse/CLOUDSTACK-3990 useful but how do I find out the database key so that I can set the pw.
>>>
>>> Also in looking at my previous backups for the host_details table it seems like the password entry changes on a regular basis.
>>>
>>> Is there something the keeps updating the db key and re-ecrypts the host passwords?
>>>
>>> On Jun 30, 2014, at 1:01 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>>>
>>>> Hi All,
>>>>
>>>> I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
>>>>
>>>> thanks.
>>>> Carlos
>>>>
>>>>
>>>> 2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
>>>> 2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>>>> 2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>>>> 2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>>> 2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
>>>> 2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
>>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
>>>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
>>>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
>>>> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
>>>> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
>>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
>>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
>>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
>>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
>>>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
>>>> at java.util.TimerThread.mainLoop(Timer.java:555)
>>>> at java.util.TimerThread.run(Timer.java:505)
>>>> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>>> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
>>>> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
>>>> at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
>>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
>>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
>>>> ... 17 more
>>>> 2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
>>>> 2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
>>>> 2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
>>>> 2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
>>>> 2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
>>>>
>>>>
>>>>
>>>> On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>>>>
>>>>> Hi Sudha,
>>>>> Thanks for checking in. I was out for the weekend and just getting back to this now.
>>>>>
>>>>> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>>>>>
>>>>> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>>>>>
>>>>> thanks,
>>>>> Carlos
>>>>>
>>>>>
>>>>>
>>>>> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>>>>>
>>>>>> Hi Carlos,
>>>>>>
>>>>>> Were you able to resolve the following? Was your upgrade successful?
>>>>>>
>>>>>> Thanks
>>>>>> /sudha
>>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>>>>>> Sent: Friday, June 27, 2014 8:55 PM
>>>>>> To: CloudStack-Users
>>>>>> Cc: dev@cloudstack.apache.org
>>>>>> Subject: 4.4 upgrade issues
>>>>>>
>>>>>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>>>>>
>>>>>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>>>>>
>>>>>> -----
>>>>>> Notes on the docs:
>>>>>>
>>>>>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>>>>>
>>>>>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>>>>>
>>>>>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>>>>>
>>>>>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>>>>>
>>>>>> --
>>>>>> Issues:
>>>>>>
>>>>>> Saw the following in catalina.out, not sure if it is an issues:
>>>>>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>>>>>
>>>>>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>>>>>
>>>>>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>>>>>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>>>>>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
I set encryption to none in db.properties and updated the passwords in host_details to unencrypted versions so I could make progress.
I don’t know what exactly the problem was but this is probably something that needs better testing. I’m pretty sure I had all the encryption stuff correct in the db.properties file but could not get it to work.
It would be nice if there was a specialized merging utility for the db.properties given the change in the organization of the file. I am guessing if the file had not been reorganized it would have been more obvious how to merge the 2 and I may have avoided this issue.
Now my hosts come up in an alert state and I’m not sure where to go from here. Please note I am not using bridge mode because I wanted to to a 4 nic bridge which bridge does not allow (only 2 nics). This was working fine in 4.1 so hopefully this is not a requirement for 4.4. I am not using security groups which was my understanding is what requires bridge networking:
The error in the log is this:
2014-06-30 14:06:50,073 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-1:ctx-35941dc7) Failed to configure brige firewall
2014-06-30 14:06:50,073 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-1:ctx-35941dc7) Check host 172.30.45.32 for CSP is installed or not and check network mode for bridge
2014-06-30 14:06:50,074 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: Response Received:
2014-06-30 14:06:50,075 DEBUG [c.c.a.t.Request] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: Processing: { Ans: , MgmtId: 233845174730253, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api
.SetupAnswer":{"_reconnect":true,"result":false,"details":"Failed to configure brige firewall","wait":0}}] }
2014-06-30 14:06:50,075 DEBUG [c.c.a.t.Request] (AgentTaskPool-2:ctx-b360d1bb) Seq 2-6232418934327345153: Received: { Ans: , MgmtId: 233845174730253, via: 2, Ver: v1, Flags: 110, { SetupAnswer } }
2014-06-30 14:06:50,076 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: No more commands found
2014-06-30 14:06:50,076 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-2:ctx-b360d1bb) Unable to setup agent 2 due to Failed to configure brige firewall
2014-06-30 14:06:50,079 INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-2:ctx-b360d1bb) Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions
2014-06-30 14:06:50,079 WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Monitor XcpServerDiscoverer says there is an error in the connect process for 2 due to Reinitialize agent after se
tup.
2014-06-30 14:06:50,079 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Host 2 is disconnecting with event AgentDisconnected
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) The next status of agent 2is Alert, current status is Connecting
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Deregistering link for 2 with state Alert
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Remove Agent : 2
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.DirectAgentAttache] (AgentTaskPool-2:ctx-b360d1bb) Processing disconnect 2(srvengxen02)
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: org.apache.cloudstack.engine.orchestration.NetworkOrchestrator
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.deploy.DeploymentPlanningManagerImpl
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl
2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener
2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-06-30 14:06:50,093 DEBUG [c.c.n.NetworkUsageManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Disconnected called on 2 with status Alert
2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.agent.manager.AgentManagerImpl$BehindOnPingListener
2014-06-30 14:06:50,093 DEBUG [c.c.h.Status] (AgentTaskPool-2:ctx-b360d1bb) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 2, name = srvengxen02]
2014-06-30 14:06:50,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Notifying other nodes of to disconnect
2014-06-30 14:06:50,108 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:ctx-59808af2) Copying /usr/share/cloudstack-management/webapps/client/WEB-INF/classes/scripts/vm/hypervisor/xenserver/xenserver60/../../../../network/domr//router_proxy.sh to /opt/cloud/bin on 172.30.45.32 with permission 0755
2014-06-30 14:06:50,108 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:ctx-59808af2) Unable to create destination path: /opt/cloud/bin on 172.30.45.32 but trying anyway
2014-06-30 14:06:50,110 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Unable to connect due to
com.cloud.exception.ConnectionException: Reinitialize agent after setup.
at com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer.processConnect(XcpServerDiscoverer.java:656)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:514)
at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1427)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1765)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1891)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
On Jun 30, 2014, at 1:57 PM, Carlos Reátegui <cr...@gmail.com> wrote:
> Making a little progress but still stuck…
>
> I realized that when I did the upgrade it had asked me if to keep the old dp.properties or use the new one. The structure of the file seemed different enough and I did not recall using anything but the defaults so I went ahead and told it to use the new one. Seems this was not the right thing to do.
>
> I have updated the password/ecryption settings to match the old file but it is still not working. Now I am getting stuck here:
>
> 2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Found parameter routing unique name null
> 2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Use console proxy template : routing
> 2014-06-30 13:50:32,143 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HostPodDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,157 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DedicatedResourceDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,168 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HypervisorCapabilitiesDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 100 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,175 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = UserDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 5000 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 300 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,180 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = ServiceOfferingDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,187 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DataCenterDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,188 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = Ip Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,189 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = vnet Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,198 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = VlanDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,232 DEBUG [c.c.u.c.DBEncryptionUtil] (main:null) Error while decrypting: true
>
> The key is still the default password and I have decrypted all the ENC parameters from the db.properties file and they seem ok. What am I missing?
>
> thanks,
> Carlos
>
>
> On Jun 30, 2014, at 1:16 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> I found the comments in: https://issues.apache.org/jira/browse/CLOUDSTACK-3990 useful but how do I find out the database key so that I can set the pw.
>>
>> Also in looking at my previous backups for the host_details table it seems like the password entry changes on a regular basis.
>>
>> Is there something the keeps updating the db key and re-ecrypts the host passwords?
>>
>> On Jun 30, 2014, at 1:01 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
>>>
>>> thanks.
>>> Carlos
>>>
>>>
>>> 2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
>>> 2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>>> 2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>>> 2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>> 2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
>>> 2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
>>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
>>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
>>> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
>>> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
>>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
>>> at java.util.TimerThread.mainLoop(Timer.java:555)
>>> at java.util.TimerThread.run(Timer.java:505)
>>> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
>>> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
>>> at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
>>> ... 17 more
>>> 2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
>>> 2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
>>> 2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
>>> 2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
>>> 2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
>>>
>>>
>>>
>>> On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>>>
>>>> Hi Sudha,
>>>> Thanks for checking in. I was out for the weekend and just getting back to this now.
>>>>
>>>> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>>>>
>>>> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>>>>
>>>> thanks,
>>>> Carlos
>>>>
>>>>
>>>>
>>>> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>>>>
>>>>> Hi Carlos,
>>>>>
>>>>> Were you able to resolve the following? Was your upgrade successful?
>>>>>
>>>>> Thanks
>>>>> /sudha
>>>>>
>>>>> -----Original Message-----
>>>>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>>>>> Sent: Friday, June 27, 2014 8:55 PM
>>>>> To: CloudStack-Users
>>>>> Cc: dev@cloudstack.apache.org
>>>>> Subject: 4.4 upgrade issues
>>>>>
>>>>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>>>>
>>>>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>>>>
>>>>> -----
>>>>> Notes on the docs:
>>>>>
>>>>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>>>>
>>>>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>>>>
>>>>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>>>>
>>>>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>>>>
>>>>> --
>>>>> Issues:
>>>>>
>>>>> Saw the following in catalina.out, not sure if it is an issues:
>>>>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>>>>
>>>>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>>>>
>>>>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>>>>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>>>>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
I set encryption to none in db.properties and updated the passwords in host_details to unencrypted versions so I could make progress.
I don’t know what exactly the problem was but this is probably something that needs better testing. I’m pretty sure I had all the encryption stuff correct in the db.properties file but could not get it to work.
It would be nice if there was a specialized merging utility for the db.properties given the change in the organization of the file. I am guessing if the file had not been reorganized it would have been more obvious how to merge the 2 and I may have avoided this issue.
Now my hosts come up in an alert state and I’m not sure where to go from here. Please note I am not using bridge mode because I wanted to to a 4 nic bridge which bridge does not allow (only 2 nics). This was working fine in 4.1 so hopefully this is not a requirement for 4.4. I am not using security groups which was my understanding is what requires bridge networking:
The error in the log is this:
2014-06-30 14:06:50,073 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-1:ctx-35941dc7) Failed to configure brige firewall
2014-06-30 14:06:50,073 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-1:ctx-35941dc7) Check host 172.30.45.32 for CSP is installed or not and check network mode for bridge
2014-06-30 14:06:50,074 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: Response Received:
2014-06-30 14:06:50,075 DEBUG [c.c.a.t.Request] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: Processing: { Ans: , MgmtId: 233845174730253, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api
.SetupAnswer":{"_reconnect":true,"result":false,"details":"Failed to configure brige firewall","wait":0}}] }
2014-06-30 14:06:50,075 DEBUG [c.c.a.t.Request] (AgentTaskPool-2:ctx-b360d1bb) Seq 2-6232418934327345153: Received: { Ans: , MgmtId: 233845174730253, via: 2, Ver: v1, Flags: 110, { SetupAnswer } }
2014-06-30 14:06:50,076 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-1:ctx-35941dc7) Seq 2-6232418934327345153: No more commands found
2014-06-30 14:06:50,076 WARN [c.c.h.x.d.XcpServerDiscoverer] (AgentTaskPool-2:ctx-b360d1bb) Unable to setup agent 2 due to Failed to configure brige firewall
2014-06-30 14:06:50,079 INFO [c.c.u.e.CSExceptionErrorCode] (AgentTaskPool-2:ctx-b360d1bb) Could not find exception: com.cloud.exception.ConnectionException in error code list for exceptions
2014-06-30 14:06:50,079 WARN [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Monitor XcpServerDiscoverer says there is an error in the connect process for 2 due to Reinitialize agent after se
tup.
2014-06-30 14:06:50,079 INFO [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Host 2 is disconnecting with event AgentDisconnected
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) The next status of agent 2is Alert, current status is Connecting
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Deregistering link for 2 with state Alert
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Remove Agent : 2
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.DirectAgentAttache] (AgentTaskPool-2:ctx-b360d1bb) Processing disconnect 2(srvengxen02)
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.hypervisor.hyperv.discoverer.HypervServerDiscoverer
2014-06-30 14:06:50,085 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: org.apache.cloudstack.engine.orchestration.NetworkOrchestrator
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.deploy.DeploymentPlanningManagerImpl
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl
2014-06-30 14:06:50,086 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl
2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener
2014-06-30 14:06:50,091 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2014-06-30 14:06:50,093 DEBUG [c.c.n.NetworkUsageManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Disconnected called on 2 with status Alert
2014-06-30 14:06:50,093 DEBUG [c.c.a.m.AgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Sending Disconnect to listener: com.cloud.agent.manager.AgentManagerImpl$BehindOnPingListener
2014-06-30 14:06:50,093 DEBUG [c.c.h.Status] (AgentTaskPool-2:ctx-b360d1bb) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 2, name = srvengxen02]
2014-06-30 14:06:50,102 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Notifying other nodes of to disconnect
2014-06-30 14:06:50,108 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:ctx-59808af2) Copying /usr/share/cloudstack-management/webapps/client/WEB-INF/classes/scripts/vm/hypervisor/xenserver/xenserver60/../../../../network/domr//router_proxy.sh to /opt/cloud/bin on 172.30.45.32 with permission 0755
2014-06-30 14:06:50,108 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-2:ctx-59808af2) Unable to create destination path: /opt/cloud/bin on 172.30.45.32 but trying anyway
2014-06-30 14:06:50,110 WARN [c.c.r.ResourceManagerImpl] (AgentTaskPool-2:ctx-b360d1bb) Unable to connect due to
com.cloud.exception.ConnectionException: Reinitialize agent after setup.
at com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer.processConnect(XcpServerDiscoverer.java:656)
at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:514)
at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1427)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1765)
at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1891)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
On Jun 30, 2014, at 1:57 PM, Carlos Reátegui <cr...@gmail.com> wrote:
> Making a little progress but still stuck…
>
> I realized that when I did the upgrade it had asked me if to keep the old dp.properties or use the new one. The structure of the file seemed different enough and I did not recall using anything but the defaults so I went ahead and told it to use the new one. Seems this was not the right thing to do.
>
> I have updated the password/ecryption settings to match the old file but it is still not working. Now I am getting stuck here:
>
> 2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Found parameter routing unique name null
> 2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Use console proxy template : routing
> 2014-06-30 13:50:32,143 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HostPodDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,157 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DedicatedResourceDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,168 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HypervisorCapabilitiesDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 100 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,175 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = UserDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 5000 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 300 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,180 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = ServiceOfferingDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,187 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DataCenterDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,188 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = Ip Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,189 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = vnet Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,198 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = VlanDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
> 2014-06-30 13:50:32,232 DEBUG [c.c.u.c.DBEncryptionUtil] (main:null) Error while decrypting: true
>
> The key is still the default password and I have decrypted all the ENC parameters from the db.properties file and they seem ok. What am I missing?
>
> thanks,
> Carlos
>
>
> On Jun 30, 2014, at 1:16 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> I found the comments in: https://issues.apache.org/jira/browse/CLOUDSTACK-3990 useful but how do I find out the database key so that I can set the pw.
>>
>> Also in looking at my previous backups for the host_details table it seems like the password entry changes on a regular basis.
>>
>> Is there something the keeps updating the db key and re-ecrypts the host passwords?
>>
>> On Jun 30, 2014, at 1:01 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>>
>>> Hi All,
>>>
>>> I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
>>>
>>> thanks.
>>> Carlos
>>>
>>>
>>> 2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
>>> 2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>>> 2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>>> 2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>> 2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
>>> 2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
>>> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
>>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
>>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
>>> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
>>> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
>>> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
>>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
>>> at java.util.TimerThread.mainLoop(Timer.java:555)
>>> at java.util.TimerThread.run(Timer.java:505)
>>> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>>> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
>>> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
>>> at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
>>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
>>> ... 17 more
>>> 2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
>>> 2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
>>> 2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
>>> 2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
>>> 2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
>>>
>>>
>>>
>>> On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>>>
>>>> Hi Sudha,
>>>> Thanks for checking in. I was out for the weekend and just getting back to this now.
>>>>
>>>> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>>>>
>>>> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>>>>
>>>> thanks,
>>>> Carlos
>>>>
>>>>
>>>>
>>>> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>>>>
>>>>> Hi Carlos,
>>>>>
>>>>> Were you able to resolve the following? Was your upgrade successful?
>>>>>
>>>>> Thanks
>>>>> /sudha
>>>>>
>>>>> -----Original Message-----
>>>>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>>>>> Sent: Friday, June 27, 2014 8:55 PM
>>>>> To: CloudStack-Users
>>>>> Cc: dev@cloudstack.apache.org
>>>>> Subject: 4.4 upgrade issues
>>>>>
>>>>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>>>>
>>>>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>>>>
>>>>> -----
>>>>> Notes on the docs:
>>>>>
>>>>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>>>>
>>>>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>>>>
>>>>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>>>>
>>>>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>>>>
>>>>> --
>>>>> Issues:
>>>>>
>>>>> Saw the following in catalina.out, not sure if it is an issues:
>>>>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>>>>
>>>>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>>>>
>>>>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>>>>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>>>>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
Making a little progress but still stuck…
I realized that when I did the upgrade it had asked me if to keep the old dp.properties or use the new one. The structure of the file seemed different enough and I did not recall using anything but the defaults so I went ahead and told it to use the new one. Seems this was not the right thing to do.
I have updated the password/ecryption settings to match the old file but it is still not working. Now I am getting stuck here:
2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Found parameter routing unique name null
2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Use console proxy template : routing
2014-06-30 13:50:32,143 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HostPodDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,157 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DedicatedResourceDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,168 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HypervisorCapabilitiesDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 100 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,175 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = UserDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 5000 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 300 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,180 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = ServiceOfferingDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,187 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DataCenterDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,188 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = Ip Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,189 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = vnet Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,198 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = VlanDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,232 DEBUG [c.c.u.c.DBEncryptionUtil] (main:null) Error while decrypting: true
The key is still the default password and I have decrypted all the ENC parameters from the db.properties file and they seem ok. What am I missing?
thanks,
Carlos
On Jun 30, 2014, at 1:16 PM, Carlos Reátegui <cr...@gmail.com> wrote:
> I found the comments in: https://issues.apache.org/jira/browse/CLOUDSTACK-3990 useful but how do I find out the database key so that I can set the pw.
>
> Also in looking at my previous backups for the host_details table it seems like the password entry changes on a regular basis.
>
> Is there something the keeps updating the db key and re-ecrypts the host passwords?
>
> On Jun 30, 2014, at 1:01 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> Hi All,
>>
>> I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
>>
>> thanks.
>> Carlos
>>
>>
>> 2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
>> 2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>> 2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>> 2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> 2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
>> 2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
>> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
>> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
>> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
>> at java.util.TimerThread.mainLoop(Timer.java:555)
>> at java.util.TimerThread.run(Timer.java:505)
>> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
>> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
>> at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
>> ... 17 more
>> 2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
>> 2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
>> 2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
>> 2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
>> 2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
>>
>>
>>
>> On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>>
>>> Hi Sudha,
>>> Thanks for checking in. I was out for the weekend and just getting back to this now.
>>>
>>> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>>>
>>> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>>>
>>> thanks,
>>> Carlos
>>>
>>>
>>>
>>> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>>>
>>>> Hi Carlos,
>>>>
>>>> Were you able to resolve the following? Was your upgrade successful?
>>>>
>>>> Thanks
>>>> /sudha
>>>>
>>>> -----Original Message-----
>>>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>>>> Sent: Friday, June 27, 2014 8:55 PM
>>>> To: CloudStack-Users
>>>> Cc: dev@cloudstack.apache.org
>>>> Subject: 4.4 upgrade issues
>>>>
>>>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>>>
>>>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>>>
>>>> -----
>>>> Notes on the docs:
>>>>
>>>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>>>
>>>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>>>
>>>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>>>
>>>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>>>
>>>> --
>>>> Issues:
>>>>
>>>> Saw the following in catalina.out, not sure if it is an issues:
>>>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>>>
>>>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>>>
>>>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>>>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>>>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
Making a little progress but still stuck…
I realized that when I did the upgrade it had asked me if to keep the old dp.properties or use the new one. The structure of the file seemed different enough and I did not recall using anything but the defaults so I went ahead and told it to use the new one. Seems this was not the right thing to do.
I have updated the password/ecryption settings to match the old file but it is still not working. Now I am getting stuck here:
2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Found parameter routing unique name null
2014-06-30 13:50:32,139 DEBUG [c.c.s.d.VMTemplateDaoImpl] (main:null) Use console proxy template : routing
2014-06-30 13:50:32,143 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HostPodDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,157 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DedicatedResourceDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,168 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = HypervisorCapabilitiesDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 100 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,175 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = UserDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 5000 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 300 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,180 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = ServiceOfferingDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,187 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = DataCenterDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,188 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = Ip Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,189 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = vnet Alloc status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 50 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,198 INFO [c.c.u.d.GenericDaoBase] (main:null) Cache created: [ name = VlanDaoImpl status = STATUS_ALIVE eternal = false overflowToDisk = false maxEntriesLocalHeap = 30 maxEntriesLocalDisk = 0 memoryStoreEvictionPolicy = LRU timeToLiveSeconds = 3600 timeToIdleSeconds = 300 persistence = none diskExpiryThreadIntervalSeconds = 120 cacheEventListeners: net.sf.ehcache.statistics.LiveCacheStatisticsWrapper hitCount = 0 memoryStoreHitCount = 0 diskStoreHitCount = 0 missCountNotFound = 0 missCountExpired = 0 maxBytesLocalHeap = 0 overflowToOffHeap = false maxBytesLocalOffHeap = 0 maxBytesLocalDisk = 0 pinned = false ]
2014-06-30 13:50:32,232 DEBUG [c.c.u.c.DBEncryptionUtil] (main:null) Error while decrypting: true
The key is still the default password and I have decrypted all the ENC parameters from the db.properties file and they seem ok. What am I missing?
thanks,
Carlos
On Jun 30, 2014, at 1:16 PM, Carlos Reátegui <cr...@gmail.com> wrote:
> I found the comments in: https://issues.apache.org/jira/browse/CLOUDSTACK-3990 useful but how do I find out the database key so that I can set the pw.
>
> Also in looking at my previous backups for the host_details table it seems like the password entry changes on a regular basis.
>
> Is there something the keeps updating the db key and re-ecrypts the host passwords?
>
> On Jun 30, 2014, at 1:01 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> Hi All,
>>
>> I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
>>
>> thanks.
>> Carlos
>>
>>
>> 2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
>> 2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>> 2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>> 2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> 2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
>> 2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
>> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
>> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
>> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
>> at java.util.TimerThread.mainLoop(Timer.java:555)
>> at java.util.TimerThread.run(Timer.java:505)
>> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
>> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
>> at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
>> ... 17 more
>> 2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
>> 2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
>> 2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
>> 2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
>> 2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
>>
>>
>>
>> On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>>
>>> Hi Sudha,
>>> Thanks for checking in. I was out for the weekend and just getting back to this now.
>>>
>>> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>>>
>>> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>>>
>>> thanks,
>>> Carlos
>>>
>>>
>>>
>>> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>>>
>>>> Hi Carlos,
>>>>
>>>> Were you able to resolve the following? Was your upgrade successful?
>>>>
>>>> Thanks
>>>> /sudha
>>>>
>>>> -----Original Message-----
>>>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>>>> Sent: Friday, June 27, 2014 8:55 PM
>>>> To: CloudStack-Users
>>>> Cc: dev@cloudstack.apache.org
>>>> Subject: 4.4 upgrade issues
>>>>
>>>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>>>
>>>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>>>
>>>> -----
>>>> Notes on the docs:
>>>>
>>>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>>>
>>>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>>>
>>>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>>>
>>>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>>>
>>>> --
>>>> Issues:
>>>>
>>>> Saw the following in catalina.out, not sure if it is an issues:
>>>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>>>
>>>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>>>
>>>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>>>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>>>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>>>
>>>>
>>>>
>>>>
>>>
>>
>
Re: 4.4 upgrade issues
Posted by ilya musayev <il...@gmail.com>.
Carlos,
Thanks for trying this out,
If you want to understand how encryption and decryption works, look into
/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt
as an example
Specifically, these lines:
#get the key from /etc/cloudstack/management/key
msKey=$(cat /etc/cloudstack/management/key)
#get encrypted key - this regex seems a little too complex :)
encPassword=$(sed '/^\#/d' /etc/cloudstack/management/db.properties |
grep 'db.cloud.password' | tail -n 1 | cut -d "=" -f2- | sed
's/^[[:space:]]*//;s/[[:space:]]*$//'i | sed 's/^ENC(\(.*\))/\1/')
#decrypt encrypted key with key above
dbPassword=(`java -classpath $jasypt
org.jasypt.intf.cli.JasyptPBEStringDecryptionCLI decrypt.sh
input=$encPassword password=$msKey verbose=false`)
Regards,
ilya
On 6/30/14, 1:16 PM, Carlos Reátegui wrote:
> I found the comments in: https://issues.apache.org/jira/browse/CLOUDSTACK-3990 useful but how do I find out the database key so that I can set the pw.
>
> Also in looking at my previous backups for the host_details table it seems like the password entry changes on a regular basis.
>
> Is there something the keeps updating the db key and re-ecrypts the host passwords?
>
> On Jun 30, 2014, at 1:01 PM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> Hi All,
>>
>> I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
>>
>> thanks.
>> Carlos
>>
>>
>> 2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
>> 2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>> 2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
>> 2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> 2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
>> 2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
>> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
>> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
>> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
>> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
>> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
>> at java.util.TimerThread.mainLoop(Timer.java:555)
>> at java.util.TimerThread.run(Timer.java:505)
>> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
>> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
>> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
>> at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
>> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
>> ... 17 more
>> 2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
>> 2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
>> 2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
>> 2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
>> 2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
>>
>>
>>
>> On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>>
>>> Hi Sudha,
>>> Thanks for checking in. I was out for the weekend and just getting back to this now.
>>>
>>> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>>>
>>> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>>>
>>> thanks,
>>> Carlos
>>>
>>>
>>>
>>> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>>>
>>>> Hi Carlos,
>>>>
>>>> Were you able to resolve the following? Was your upgrade successful?
>>>>
>>>> Thanks
>>>> /sudha
>>>>
>>>> -----Original Message-----
>>>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>>>> Sent: Friday, June 27, 2014 8:55 PM
>>>> To: CloudStack-Users
>>>> Cc: dev@cloudstack.apache.org
>>>> Subject: 4.4 upgrade issues
>>>>
>>>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>>>
>>>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>>>
>>>> -----
>>>> Notes on the docs:
>>>>
>>>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>>>
>>>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>>>
>>>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>>>
>>>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>>>
>>>> --
>>>> Issues:
>>>>
>>>> Saw the following in catalina.out, not sure if it is an issues:
>>>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>>>
>>>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>>>
>>>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>>>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>>>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>>>
>>>>
>>>>
>>>>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
I found the comments in: https://issues.apache.org/jira/browse/CLOUDSTACK-3990 useful but how do I find out the database key so that I can set the pw.
Also in looking at my previous backups for the host_details table it seems like the password entry changes on a regular basis.
Is there something the keeps updating the db key and re-ecrypts the host passwords?
On Jun 30, 2014, at 1:01 PM, Carlos Reátegui <cr...@gmail.com> wrote:
> Hi All,
>
> I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
>
> thanks.
> Carlos
>
>
> 2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
> 2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
> 2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
> 2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
> 2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
> 2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
> at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
> ... 17 more
> 2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
> 2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
> 2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
> 2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
> 2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
>
>
>
> On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> Hi Sudha,
>> Thanks for checking in. I was out for the weekend and just getting back to this now.
>>
>> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>>
>> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>>
>> thanks,
>> Carlos
>>
>>
>>
>> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>>
>>> Hi Carlos,
>>>
>>> Were you able to resolve the following? Was your upgrade successful?
>>>
>>> Thanks
>>> /sudha
>>>
>>> -----Original Message-----
>>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>>> Sent: Friday, June 27, 2014 8:55 PM
>>> To: CloudStack-Users
>>> Cc: dev@cloudstack.apache.org
>>> Subject: 4.4 upgrade issues
>>>
>>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>>
>>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>>
>>> -----
>>> Notes on the docs:
>>>
>>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>>
>>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>>
>>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>>
>>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>>
>>> --
>>> Issues:
>>>
>>> Saw the following in catalina.out, not sure if it is an issues:
>>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>>
>>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>>
>>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>>
>>>
>>>
>>>
>>
>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
I found the comments in: https://issues.apache.org/jira/browse/CLOUDSTACK-3990 useful but how do I find out the database key so that I can set the pw.
Also in looking at my previous backups for the host_details table it seems like the password entry changes on a regular basis.
Is there something the keeps updating the db key and re-ecrypts the host passwords?
On Jun 30, 2014, at 1:01 PM, Carlos Reátegui <cr...@gmail.com> wrote:
> Hi All,
>
> I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
>
> thanks.
> Carlos
>
>
> 2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
> 2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
> 2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
> 2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
> 2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
> 2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
> com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
> at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
> at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
> at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
> at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
> at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
> at java.util.TimerThread.mainLoop(Timer.java:555)
> at java.util.TimerThread.run(Timer.java:505)
> Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
> at com.xensource.xenapi.Types.checkResponse(Types.java:322)
> at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
> at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
> at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
> ... 17 more
> 2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
> 2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
> 2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
> 2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
> 2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
>
>
>
> On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
>
>> Hi Sudha,
>> Thanks for checking in. I was out for the weekend and just getting back to this now.
>>
>> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>>
>> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>>
>> thanks,
>> Carlos
>>
>>
>>
>> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>>
>>> Hi Carlos,
>>>
>>> Were you able to resolve the following? Was your upgrade successful?
>>>
>>> Thanks
>>> /sudha
>>>
>>> -----Original Message-----
>>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>>> Sent: Friday, June 27, 2014 8:55 PM
>>> To: CloudStack-Users
>>> Cc: dev@cloudstack.apache.org
>>> Subject: 4.4 upgrade issues
>>>
>>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>>
>>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>>
>>> -----
>>> Notes on the docs:
>>>
>>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>>
>>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>>
>>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>>
>>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>>
>>> --
>>> Issues:
>>>
>>> Saw the following in catalina.out, not sure if it is an issues:
>>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>>
>>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>>
>>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>>
>>>
>>>
>>>
>>
>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
Hi All,
I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
thanks.
Carlos
2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
at com.xensource.xenapi.Types.checkResponse(Types.java:322)
at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
... 17 more
2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
> Hi Sudha,
> Thanks for checking in. I was out for the weekend and just getting back to this now.
>
> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>
> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>
> thanks,
> Carlos
>
>
>
> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>
>> Hi Carlos,
>>
>> Were you able to resolve the following? Was your upgrade successful?
>>
>> Thanks
>> /sudha
>>
>> -----Original Message-----
>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>> Sent: Friday, June 27, 2014 8:55 PM
>> To: CloudStack-Users
>> Cc: dev@cloudstack.apache.org
>> Subject: 4.4 upgrade issues
>>
>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>
>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>
>> -----
>> Notes on the docs:
>>
>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>
>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>
>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>
>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>
>> --
>> Issues:
>>
>> Saw the following in catalina.out, not sure if it is an issues:
>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>
>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>
>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>
>>
>>
>>
>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
Hi All,
I am having problems bringing my system back up. I have not checked the credentials of my hosts but the upgraded management server is unable to connect to them. Where is the password stored?
thanks.
Carlos
2014-06-30 12:55:59,277 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 1(srvengxen01)
2014-06-30 12:56:04,394 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
2014-06-30 12:56:04,428 DEBUG [c.c.n.l.LBHealthCheckManagerImpl] (LBHealthCheck-1:ctx-c6869648) LB HealthCheck Manager is running and getting the updates from LB providers and updating service status
2014-06-30 12:56:06,844 DEBUG [c.c.h.x.r.XenServerConnectionPool] (ClusteredAgentManager Timer:ctx-060c8ace) Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
2014-06-30 12:56:06,848 DEBUG [c.c.h.Status] (ClusteredAgentManager Timer:ctx-060c8ace) Transition:[Resource state = Enabled, Agent event = AgentDisconnected, Host id = 1, name = srvengxen01]
2014-06-30 12:56:06,862 WARN [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) can not load directly connected host 1(srvengxen01) due to
com.cloud.utils.exception.CloudRuntimeException: Unable to create master connection to host(172.30.45.31) , due to The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:168)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.CheckXenHostInfo(CitrixResourceBase.java:5722)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.configure(CitrixResourceBase.java:5705)
at com.cloud.resource.DiscovererBase.reloadResource(DiscovererBase.java:157)
at com.cloud.agent.manager.AgentManagerImpl.loadDirectlyConnectedHost(AgentManagerImpl.java:672)
at com.cloud.agent.manager.ClusteredAgentManagerImpl.scanDirectAgentToLoad(ClusteredAgentManagerImpl.java:218)
at com.cloud.agent.manager.ClusteredAgentManagerImpl.runDirectAgentScanTimerTask(ClusteredAgentManagerImpl.java:184)
at com.cloud.agent.manager.ClusteredAgentManagerImpl.access$100(ClusteredAgentManagerImpl.java:98)
at com.cloud.agent.manager.ClusteredAgentManagerImpl$DirectAgentScanTimerTask.runInContext(ClusteredAgentManagerImpl.java:234)
at org.apache.cloudstack.managed.context.ManagedContextTimerTask$1.runInContext(ManagedContextTimerTask.java:30)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.managed.context.ManagedContextTimerTask.run(ManagedContextTimerTask.java:27)
at java.util.TimerThread.mainLoop(Timer.java:555)
at java.util.TimerThread.run(Timer.java:505)
Caused by: The credentials given by the user are incorrect, so access has been denied, and you have not been issued a session handle.
at com.xensource.xenapi.Types.checkResponse(Types.java:322)
at com.xensource.xenapi.Connection.dispatch(Connection.java:350)
at com.xensource.xenapi.Session.loginWithPassword(Session.java:537)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.loginWithPassword(XenServerConnectionPool.java:321)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool.getConnect(XenServerConnectionPool.java:154)
... 17 more
2014-06-30 12:56:06,864 DEBUG [c.c.a.m.ClusteredAgentManagerImpl] (ClusteredAgentManager Timer:ctx-060c8ace) Loading directly connected host 2(srvengxen02)
2014-06-30 12:56:09,225 DEBUG [c.c.s.StatsCollector] (StatsCollector-1:ctx-8458e286) HostStatsCollector is running...
2014-06-30 12:56:09,226 DEBUG [c.c.s.StatsCollector] (StatsCollector-2:ctx-aa245eed) VmStatsCollector is running...
2014-06-30 12:56:09,227 DEBUG [c.c.s.StatsCollector] (StatsCollector-3:ctx-19894fa1) StorageCollector is running...
2014-06-30 12:56:09,230 DEBUG [c.c.s.StatsCollector] (StatsCollector-4:ctx-d66c71fb) AutoScaling Monitor is running...
On Jun 30, 2014, at 9:54 AM, Carlos Reátegui <cr...@gmail.com> wrote:
> Hi Sudha,
> Thanks for checking in. I was out for the weekend and just getting back to this now.
>
> My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
>
> Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
>
> thanks,
> Carlos
>
>
>
> On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
>
>> Hi Carlos,
>>
>> Were you able to resolve the following? Was your upgrade successful?
>>
>> Thanks
>> /sudha
>>
>> -----Original Message-----
>> From: Carlos Reátegui [mailto:creategui@gmail.com]
>> Sent: Friday, June 27, 2014 8:55 PM
>> To: CloudStack-Users
>> Cc: dev@cloudstack.apache.org
>> Subject: 4.4 upgrade issues
>>
>> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>>
>> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>>
>> -----
>> Notes on the docs:
>>
>> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>>
>> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>>
>> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>>
>> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>>
>> --
>> Issues:
>>
>> Saw the following in catalina.out, not sure if it is an issues:
>> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
>> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>>
>> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>>
>> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
>> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
>> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>>
>>
>>
>>
>
Re: 4.4 upgrade issues
Posted by Carlos Reátegui <cr...@gmail.com>.
Hi Sudha,
Thanks for checking in. I was out for the weekend and just getting back to this now.
My main question at this point is if it is ok for me to kill the system vms with the xe vm-shutdown command since the script provided by cloudstack does not work with ubuntu.
Also it would be great if someone could have a look at my logs to see if they look normal. I am seeing a lot of HA-Worker messages but I do not have an HA deployment (unless this is the thread that keeps the system vas running).
thanks,
Carlos
On Jun 29, 2014, at 11:51 PM, Sudha Ponnaganti <su...@citrix.com> wrote:
> Hi Carlos,
>
> Were you able to resolve the following? Was your upgrade successful?
>
> Thanks
> /sudha
>
> -----Original Message-----
> From: Carlos Reátegui [mailto:creategui@gmail.com]
> Sent: Friday, June 27, 2014 8:55 PM
> To: CloudStack-Users
> Cc: dev@cloudstack.apache.org
> Subject: 4.4 upgrade issues
>
> I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
>
> My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
>
> -----
> Notes on the docs:
>
> 8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
>
> 13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
>
> 14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
>
> Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
>
> --
> Issues:
>
> Saw the following in catalina.out, not sure if it is an issues:
> Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
> INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
>
> Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
>
> Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
> Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
> Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
>
>
>
>
RE: 4.4 upgrade issues
Posted by Sudha Ponnaganti <su...@citrix.com>.
Hi Carlos,
Were you able to resolve the following? Was your upgrade successful?
Thanks
/sudha
-----Original Message-----
From: Carlos Reátegui [mailto:creategui@gmail.com]
Sent: Friday, June 27, 2014 8:55 PM
To: CloudStack-Users
Cc: dev@cloudstack.apache.org
Subject: 4.4 upgrade issues
I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
-----
Notes on the docs:
8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
--
Issues:
Saw the following in catalina.out, not sure if it is an issues:
Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log
RE: 4.4 upgrade issues
Posted by Sudha Ponnaganti <su...@citrix.com>.
Hi Carlos,
Were you able to resolve the following? Was your upgrade successful?
Thanks
/sudha
-----Original Message-----
From: Carlos Reátegui [mailto:creategui@gmail.com]
Sent: Friday, June 27, 2014 8:55 PM
To: CloudStack-Users
Cc: dev@cloudstack.apache.org
Subject: 4.4 upgrade issues
I am trying out the upgrade instructions from http://docs.cloudstack.apache.org/projects/cloudstack-release-notes/en/4.3/rnotes.html#upgrade-from-4-1-x-to-4-3 but going to 4.4 built from source today.
My setup: XenServer 6.0.2 Hosts, Management Server on Ubuntu 12.04, Primary and Secondary on NFS, Basic Network, no security groups
-----
Notes on the docs:
8.4 - 8.6: This is only for hosts that use the cloudstack agent. Does not apply to KVM. In general this whole section does not do a good job of explaining what is on the MS vs the Hosts.
13: This fails on ubuntu because: cloudstack-sysvmadm sources /etc/rc.d/init.d/functions which does not exist on ubuntu/debian systems.
14: Copy vhf-util from where? Also the path /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver does not exist on the hosts so I am assuming this is on the MS, however the MS already has it since it is an upgrade and was put there by the original install. Or is this a new version that needs to be grabbed from somewhere?
Other: earlier versions like 4.1 worked with JDK 1.6 current releases require 1.7 but the Upgrade doc does not mention that.
--
Issues:
Saw the following in catalina.out, not sure if it is an issues:
Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/servlet-api-2.5-20081211.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class Jun 27, 2014 5:28:42 PM org.apache.catalina.loader.WebappClassLoader validateJarFile
INFO: validateJarFile(/usr/share/cloudstack-management/webapps/client/WEB-INF/lib/tomcat-embed-core-7.0.30.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class
Since the above script in step 13 did not work is it ok to do "xe vm-shutdown vm=." on each of the system vms? Will CloudStack notice they are ton and start new ones?
Here are my log files (please note I stopped the service prior to capturing these logs in case you are wondering):
Management server log: https://www.dropbox.com/s/7xhkutt8e724il1/management-server.log
Catalina log: https://www.dropbox.com/s/f45ypkbazhkogyj/catalina.2014-06-27.log