You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Nitin Mehta <Ni...@citrix.com> on 2013/11/08 01:16:58 UTC
Router requires upgrade. Unable to send command to router Error
Unable to deploy vms in the latest master because of the commit below. Is anyone noticing this on the latest master.
I checked the code and there was a commit made recently 3f5b8f which is where the exception points to.
WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 // dataCenterId:: 3 // podId:: 3 // clusterId:: null // message:: System Alert: Low Available Storage in cluster c3 pod p of availability zone z3
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add job-44 into job monitoring
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331) Detecting a change in xstoolsversion for r-6-VM
ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc ctx-d9a00f18) Failed to start instance VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
com.cloud.utils.exception.CloudRuntimeException: Router requires upgrade. Unable to send command to router:6
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3567)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute(VirtualNetworkApplianceManagerImpl.java:3003)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3848)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
Re: Router requires upgrade. Unable to send command to router Error
Posted by Will Stevens <ws...@cloudops.com>.
Thanks Hugo, that was the cause of my 'fsck' issue. NTP was broken on my
dev XenServer due to a misconfigured resolv.conf.
I can confirm that the recent System VM Templates are working.
Thank you...
On Mon, Nov 25, 2013 at 5:15 PM, Hugo Trippaers <hu...@trippaers.nl> wrote:
> Will,
>
> What is the error reported by the OS before the fsck is needed? I ran into
> a similar issue a while back and it was caused by a wrong date/time setting
> on the hypervisor. If the systemvm spins up with a time that is earlier
> than when the image was created the system will force an fsck because the
> last mounted time is in the future. To fix this is needed to set the time
> on the hypervisor correctly.
>
>
> Cheers,
>
> Hugo
>
> BTW the 4.2 images are ready in the jerkins job.
>
> On 25 nov. 2013, at 22:02, Will Stevens <ws...@cloudops.com> wrote:
>
> > I just used:
> >
> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/lastSuccessfulBuild/artifact/tools/appliance/dist/systemvmtemplate-2013-11-24-master-xen.vhd.bz2
> >
> > I ran into the same issue with this template as all the others I have
> been
> > trying. Basically, the vm comes up with a read only file system for some
> > reason. If I run 'fsck' on the vm from XenCenter when the system vms
> come
> > up, and then reboot, I can get the vm to come up correctly. Each time I
> > launch a new vm which requires a virtual router, I have to 'fsck' the VR
> in
> > XenCenter as soon as it tries to launch.
> >
> > This is working for me to launch an instance (but it is quite annoying).
> > Other than the 'fsck' issue, these templates seem to be working.
> >
> > ws
> >
> >
> > On Mon, Nov 25, 2013 at 3:29 PM, Hugo Trippaers <hu...@trippaers.nl>
> wrote:
> >
> >> The images should be downloadable from jenkins.buildacloud.org
> >>
> >> The 4.2 images are here:
> >> http://jenkins.buildacloud.org/view/4.2/job/build-systemvm-4.2/
> >>
> >> There was something wrong with this build, but i’m working on it. Expect
> >> the images soon.
> >>
> >> Cheers,
> >>
> >> Hugo
> >>
> >> On 25 nov. 2013, at 21:14, Travis Graham <tg...@tgraham.us> wrote:
> >>
> >>> Use the links from the Install Guide instead.
> >>>
> >>>
> >>
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/management-server-install-flow.html#prepare-system-vm-template
> >>>
> >>> Travis
> >>>
> >>> On Nov 25, 2013, at 3:01 PM, Will Stevens <ws...@cloudops.com>
> wrote:
> >>>
> >>>> In trying to troubleshoot I think I have found another issue. I went
> >> looking for a 'more official' source for the system templates. I found
> >> this:
> >>
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Release_Notes/upgrade-instructions.html#upgrade-from-3.0.x-to-4.0
> >>>>
> >>>> Which gives a system vm template of (for xen server):
> >>
> http://download.cloud.com/templates/4.2/systemvmtemplate-2013-07-12-master-xen.vhd.bz2
> >>>>
> >>>> Unfortunately, those system vm templates do not come up (check the
> >> attached files for images). Basically, the file system comes up as Read
> >> Only...
> >>>>
> >>>> I will go back to the buildacloud system templates to see if I can get
> >> them working...
> >>>>
> >>>> Would love to have someone confirm where we should be getting the
> >> System VM Templates from for 4.2+.
> >>>>
> >>>> Still trying to get System VM Templates to work on 4.3. If anyone has
> >> this working, please post how you get them working and where you got
> them
> >> from.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Will
> >>>>
> >>>>
> >>>> On Fri, Nov 22, 2013 at 7:43 AM, Will Stevens <ws...@cloudops.com>
> >> wrote:
> >>>> I will try this as a temporary solution. Thank you...
> >>>>
> >>>> Will
> >>>>
> >>>>
> >>>> On Fri, Nov 22, 2013 at 6:57 AM, Murali Reddy <
> Murali.Reddy@citrix.com>
> >> wrote:
> >>>>
> >>>> Don¹t understand problem well enough for clean fix, but I updated
> >>>> 'template_version' from 3.0 to 4.3 of the VR in the domain_router
> table
> >>>> that resolved the issue for me.
> >>>>
> >>>> On 22/11/13 3:50 PM, "Will Stevens" <ws...@cloudops.com> wrote:
> >>>>
> >>>>> Has anyone been able to resolve this issue? This is holding up my
> >> ability
> >>>>> to launch VMs and test the fixes to my plugin. I need to resolve
> this
> >>>>> issue to move forward...
> >>>>>
> >>>>> @Syed, are you still stuck on this as well?
> >>>>>
> >>>>> Cheers,
> >>>>>
> >>>>> Will
> >>>>>
> >>>>>
> >>>>> On Wed, Nov 20, 2013 at 5:18 PM, Syed Ahmed <sa...@cloudops.com>
> >> wrote:
> >>>>>
> >>>>>> OK here is how far I got debugging this. I think I am missing a
> small
> >>>>>> thing. I hope you guys can help.
> >>>>>>
> >>>>>> So my VM template has the correct version.
> >>>>>>
> >>>>>> root@eng-ns-dev-cs1: /export/secondary/template/tmpl/1/1 # strings
> >>>>>> f3fc75d9-0240-4c71-a3bf-fb65652e4763.vhd | grep Cloudstack
> >>>>>>
> >>>>>> Cloudstack Release* 4.2.0*Tue Nov 19 23:22:37 UTC 2013
> >>>>>>
> >>>>>>
> >>>>>> But in the database I see the following ( table domain_router )
> >>>>>>
> >>>>>> *************************** 4. row ***************************
> >>>>>> id: 11
> >>>>>> element_id: 4
> >>>>>> public_mac_address: 06:48:a8:00:00:68
> >>>>>> public_ip_address: 172.30.91.102
> >>>>>> public_netmask: 255.255.255.0
> >>>>>> guest_netmask: NULL
> >>>>>> guest_ip_address: NULL
> >>>>>> is_redundant_router: 0
> >>>>>> priority: 0
> >>>>>> is_priority_bumpup: 0
> >>>>>> redundant_state: UNKNOWN
> >>>>>> stop_pending: 0
> >>>>>> role: VIRTUAL_ROUTER
> >>>>>> template_version:*Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST
> >> 2012*
> >>>>>> scripts_version: 725d5e5901a62c68aed0dd3463023518
> >>>>>> vpc_id: NULL
> >>>>>> 4 rows in set (0.00 sec)
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> I guess this is populated from the VM that gets created. On the xen
> >> the
> >>>>>> vm
> >>>>>> is r-11. I see the following version on that VM
> >>>>>>
> >>>>>> root@r-11-VM:~# cat /etc/cloudstack-release
> >>>>>> Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012
> >>>>>>
> >>>>>>
> >>>>>> This means that Xen is not picking up the template present in the
> >>>>>> secondary storage. Does Xen cache the vhd files locally to avoid
> >> coming
> >>>>>> to
> >>>>>> the secondary storage? If so, how can I disable that?
> >>>>>>
> >>>>>> Also, I was looking at UpgradeRouterTemplateCmd API which basically
> >> goes
> >>>>>> through all the VRs and reboots them. It expects that when the
> reboot
> >> is
> >>>>>> completed, the router should have picked up the 4.2.0 version of the
> >>>>>> template ( see line 4072 in VirtualNetworkApplianceManagerImpl.java
> )
> >> I
> >>>>>> try to do the reboot manually but the template remains the same. Do
> >> you
> >>>>>> guys have any more suggestions?
> >>>>>>
> >>>>>> Thanks,
> >>>>>> -Syed
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Wed 20 Nov 2013 12:55:04 PM EST, Wei ZHOU wrote:
> >>>>>>
> >>>>>>>
> >>>>>>> FYI.
> >>>>>>>
> >>>>>>> I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working
> >>>>>>> after
> >>>>>>> running *cloudstack-sysvmadm to recreate.*
> >>>>>>>
> >>>>>>>
> >>>>>>> 2013/11/20 Syed Ahmed <sa...@cloudops.com>
> >>>>>>>
> >>>>>>>
> >>>>>>>> +1 Same error. The secondary storage VM and the Console proxy VM
> >> seem
> >>>>>>>> to
> >>>>>>>> be coming up alright. I see this error only when starting the
> >> virtual
> >>>>>>>> router which is preventing me from creating any instances.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> I am having the same problem. I got the latest system VMs from:
> >>>>>>>>>
> >> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
> >>>>>>>>> lastSuccessfulBuild/artifact/tools/appliance/dist/
> >>>>>>>>>
> >>>>>>>>> Are these the wrong System VM Templates? If so, where should I
> get
> >>>>>>>>> the
> >>>>>>>>> System VM Templates to make this work again?
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> Will
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
> >>>>>>>>> Alena.Prokharchyk@citrix.com> wrote:
> >>>>>>>>>
> >>>>>>>>> Nitin, I had the same problem, but I fixed it by uploading 4.2
> >> system
> >>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> templates to my secondary storage. Make sure you have the latest
> >>>>>>>>>> too.
> >>>>>>>>>>
> >>>>>>>>>> -alena.
> >>>>>>>>>>
> >>>>>>>>>> From: Nitin Mehta <Nitin.Mehta@citrix.com<mailto:
> >>>>>>>>>> Nitin.Mehta@citrix.com>
> >>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>> Reply-To:
> >>>>>>>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org
> >>>>>>>>>>> "
> >>>>>>>>>> <
> >>>>>>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> >>>>>>>>>> Date: Thursday, November 7, 2013 4:16 PM
> >>>>>>>>>> To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org
> >"
> >> <
> >>>>>>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> >>>>>>>>>> Subject: Router requires upgrade. Unable to send command to
> router
> >>>>>>>>>> Error
> >>>>>>>>>>
> >>>>>>>>>> Unable to deploy vms in the latest master because of the commit
> >>>>>>>>>> below.
> >>>>>>>>>> Is
> >>>>>>>>>> anyone noticing this on the latest master.
> >>>>>>>>>> I checked the code and there was a commit made recently 3f5b8f
> >>>>>>>>>> which is
> >>>>>>>>>> where the exception points to.
> >>>>>>>>>>
> >>>>>>>>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2
> >> //
> >>>>>>>>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message::
> >>>>>>>>>> System
> >>>>>>>>>> Alert: Low Available Storage in cluster c3 pod p of availability
> >>>>>>>>>> zone
> >>>>>>>>>> z3
> >>>>>>>>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc)
> >> Add
> >>>>>>>>>> job-44 into job monitoring
> >>>>>>>>>> WARN [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-26:ctx-3e786331)
> >>>>>>>>>> Detecting a change in xstoolsversion for r-6-VM
> >>>>>>>>>> ERROR [c.c.v.VirtualMachineManagerImpl]
> >> (Job-Executor-1:ctx-f118d6dc
> >>>>>>>>>> ctx-d9a00f18) Failed to start instance
> >>>>>>>>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
> >>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
> >>>>>>>>>> upgrade.
> >>>>>>>>>> Unable to send command to router:6
> >>>>>>>>>> at
> >>>>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>>>>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManager
> >>>>>>>>>> Impl.java:3567)
> >>>>>>>>>> at
> >>>>>>>>>>
> >>>>>>>>>>
> >> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute
> >>>>>>>>>> (
> >>>>>>>>>> VirtualNetworkApplianceManagerImpl.java:3003)
> >>>>>>>>>> at
> >>>>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>>>>>> Impl.applyRules(
> >>>>>>>>>> VirtualNetworkApplianceManagerImpl.java:3848)
> >>>>>>>>>> at
> >>>>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>>>>>>
> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
> >>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>>>>>>> at
> >>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
> >>>>>>>>>> NativeMethodAccessorImpl.java:39)
> >>>>>>>>>> at
> >>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> >>>>>>>>>> DelegatingMethodAccessorImpl.java:25)
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>
> >>
>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Hugo Trippaers <hu...@trippaers.nl>.
Will,
What is the error reported by the OS before the fsck is needed? I ran into a similar issue a while back and it was caused by a wrong date/time setting on the hypervisor. If the systemvm spins up with a time that is earlier than when the image was created the system will force an fsck because the last mounted time is in the future. To fix this is needed to set the time on the hypervisor correctly.
Cheers,
Hugo
BTW the 4.2 images are ready in the jerkins job.
On 25 nov. 2013, at 22:02, Will Stevens <ws...@cloudops.com> wrote:
> I just used:
> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/lastSuccessfulBuild/artifact/tools/appliance/dist/systemvmtemplate-2013-11-24-master-xen.vhd.bz2
>
> I ran into the same issue with this template as all the others I have been
> trying. Basically, the vm comes up with a read only file system for some
> reason. If I run 'fsck' on the vm from XenCenter when the system vms come
> up, and then reboot, I can get the vm to come up correctly. Each time I
> launch a new vm which requires a virtual router, I have to 'fsck' the VR in
> XenCenter as soon as it tries to launch.
>
> This is working for me to launch an instance (but it is quite annoying).
> Other than the 'fsck' issue, these templates seem to be working.
>
> ws
>
>
> On Mon, Nov 25, 2013 at 3:29 PM, Hugo Trippaers <hu...@trippaers.nl> wrote:
>
>> The images should be downloadable from jenkins.buildacloud.org
>>
>> The 4.2 images are here:
>> http://jenkins.buildacloud.org/view/4.2/job/build-systemvm-4.2/
>>
>> There was something wrong with this build, but i’m working on it. Expect
>> the images soon.
>>
>> Cheers,
>>
>> Hugo
>>
>> On 25 nov. 2013, at 21:14, Travis Graham <tg...@tgraham.us> wrote:
>>
>>> Use the links from the Install Guide instead.
>>>
>>>
>> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/management-server-install-flow.html#prepare-system-vm-template
>>>
>>> Travis
>>>
>>> On Nov 25, 2013, at 3:01 PM, Will Stevens <ws...@cloudops.com> wrote:
>>>
>>>> In trying to troubleshoot I think I have found another issue. I went
>> looking for a 'more official' source for the system templates. I found
>> this:
>> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Release_Notes/upgrade-instructions.html#upgrade-from-3.0.x-to-4.0
>>>>
>>>> Which gives a system vm template of (for xen server):
>> http://download.cloud.com/templates/4.2/systemvmtemplate-2013-07-12-master-xen.vhd.bz2
>>>>
>>>> Unfortunately, those system vm templates do not come up (check the
>> attached files for images). Basically, the file system comes up as Read
>> Only...
>>>>
>>>> I will go back to the buildacloud system templates to see if I can get
>> them working...
>>>>
>>>> Would love to have someone confirm where we should be getting the
>> System VM Templates from for 4.2+.
>>>>
>>>> Still trying to get System VM Templates to work on 4.3. If anyone has
>> this working, please post how you get them working and where you got them
>> from.
>>>>
>>>> Thanks,
>>>>
>>>> Will
>>>>
>>>>
>>>> On Fri, Nov 22, 2013 at 7:43 AM, Will Stevens <ws...@cloudops.com>
>> wrote:
>>>> I will try this as a temporary solution. Thank you...
>>>>
>>>> Will
>>>>
>>>>
>>>> On Fri, Nov 22, 2013 at 6:57 AM, Murali Reddy <Mu...@citrix.com>
>> wrote:
>>>>
>>>> Don¹t understand problem well enough for clean fix, but I updated
>>>> 'template_version' from 3.0 to 4.3 of the VR in the domain_router table
>>>> that resolved the issue for me.
>>>>
>>>> On 22/11/13 3:50 PM, "Will Stevens" <ws...@cloudops.com> wrote:
>>>>
>>>>> Has anyone been able to resolve this issue? This is holding up my
>> ability
>>>>> to launch VMs and test the fixes to my plugin. I need to resolve this
>>>>> issue to move forward...
>>>>>
>>>>> @Syed, are you still stuck on this as well?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Will
>>>>>
>>>>>
>>>>> On Wed, Nov 20, 2013 at 5:18 PM, Syed Ahmed <sa...@cloudops.com>
>> wrote:
>>>>>
>>>>>> OK here is how far I got debugging this. I think I am missing a small
>>>>>> thing. I hope you guys can help.
>>>>>>
>>>>>> So my VM template has the correct version.
>>>>>>
>>>>>> root@eng-ns-dev-cs1: /export/secondary/template/tmpl/1/1 # strings
>>>>>> f3fc75d9-0240-4c71-a3bf-fb65652e4763.vhd | grep Cloudstack
>>>>>>
>>>>>> Cloudstack Release* 4.2.0*Tue Nov 19 23:22:37 UTC 2013
>>>>>>
>>>>>>
>>>>>> But in the database I see the following ( table domain_router )
>>>>>>
>>>>>> *************************** 4. row ***************************
>>>>>> id: 11
>>>>>> element_id: 4
>>>>>> public_mac_address: 06:48:a8:00:00:68
>>>>>> public_ip_address: 172.30.91.102
>>>>>> public_netmask: 255.255.255.0
>>>>>> guest_netmask: NULL
>>>>>> guest_ip_address: NULL
>>>>>> is_redundant_router: 0
>>>>>> priority: 0
>>>>>> is_priority_bumpup: 0
>>>>>> redundant_state: UNKNOWN
>>>>>> stop_pending: 0
>>>>>> role: VIRTUAL_ROUTER
>>>>>> template_version:*Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST
>> 2012*
>>>>>> scripts_version: 725d5e5901a62c68aed0dd3463023518
>>>>>> vpc_id: NULL
>>>>>> 4 rows in set (0.00 sec)
>>>>>>
>>>>>>
>>>>>>
>>>>>> I guess this is populated from the VM that gets created. On the xen
>> the
>>>>>> vm
>>>>>> is r-11. I see the following version on that VM
>>>>>>
>>>>>> root@r-11-VM:~# cat /etc/cloudstack-release
>>>>>> Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012
>>>>>>
>>>>>>
>>>>>> This means that Xen is not picking up the template present in the
>>>>>> secondary storage. Does Xen cache the vhd files locally to avoid
>> coming
>>>>>> to
>>>>>> the secondary storage? If so, how can I disable that?
>>>>>>
>>>>>> Also, I was looking at UpgradeRouterTemplateCmd API which basically
>> goes
>>>>>> through all the VRs and reboots them. It expects that when the reboot
>> is
>>>>>> completed, the router should have picked up the 4.2.0 version of the
>>>>>> template ( see line 4072 in VirtualNetworkApplianceManagerImpl.java )
>> I
>>>>>> try to do the reboot manually but the template remains the same. Do
>> you
>>>>>> guys have any more suggestions?
>>>>>>
>>>>>> Thanks,
>>>>>> -Syed
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed 20 Nov 2013 12:55:04 PM EST, Wei ZHOU wrote:
>>>>>>
>>>>>>>
>>>>>>> FYI.
>>>>>>>
>>>>>>> I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working
>>>>>>> after
>>>>>>> running *cloudstack-sysvmadm to recreate.*
>>>>>>>
>>>>>>>
>>>>>>> 2013/11/20 Syed Ahmed <sa...@cloudops.com>
>>>>>>>
>>>>>>>
>>>>>>>> +1 Same error. The secondary storage VM and the Console proxy VM
>> seem
>>>>>>>> to
>>>>>>>> be coming up alright. I see this error only when starting the
>> virtual
>>>>>>>> router which is preventing me from creating any instances.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>> I am having the same problem. I got the latest system VMs from:
>>>>>>>>>
>> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
>>>>>>>>> lastSuccessfulBuild/artifact/tools/appliance/dist/
>>>>>>>>>
>>>>>>>>> Are these the wrong System VM Templates? If so, where should I get
>>>>>>>>> the
>>>>>>>>> System VM Templates to make this work again?
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Will
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
>>>>>>>>> Alena.Prokharchyk@citrix.com> wrote:
>>>>>>>>>
>>>>>>>>> Nitin, I had the same problem, but I fixed it by uploading 4.2
>> system
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> templates to my secondary storage. Make sure you have the latest
>>>>>>>>>> too.
>>>>>>>>>>
>>>>>>>>>> -alena.
>>>>>>>>>>
>>>>>>>>>> From: Nitin Mehta <Nitin.Mehta@citrix.com<mailto:
>>>>>>>>>> Nitin.Mehta@citrix.com>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>> Reply-To:
>>>>>>>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org
>>>>>>>>>>> "
>>>>>>>>>> <
>>>>>>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>>>>>>>>> Date: Thursday, November 7, 2013 4:16 PM
>>>>>>>>>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
>> <
>>>>>>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>>>>>>>>> Subject: Router requires upgrade. Unable to send command to router
>>>>>>>>>> Error
>>>>>>>>>>
>>>>>>>>>> Unable to deploy vms in the latest master because of the commit
>>>>>>>>>> below.
>>>>>>>>>> Is
>>>>>>>>>> anyone noticing this on the latest master.
>>>>>>>>>> I checked the code and there was a commit made recently 3f5b8f
>>>>>>>>>> which is
>>>>>>>>>> where the exception points to.
>>>>>>>>>>
>>>>>>>>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2
>> //
>>>>>>>>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message::
>>>>>>>>>> System
>>>>>>>>>> Alert: Low Available Storage in cluster c3 pod p of availability
>>>>>>>>>> zone
>>>>>>>>>> z3
>>>>>>>>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc)
>> Add
>>>>>>>>>> job-44 into job monitoring
>>>>>>>>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
>>>>>>>>>> Detecting a change in xstoolsversion for r-6-VM
>>>>>>>>>> ERROR [c.c.v.VirtualMachineManagerImpl]
>> (Job-Executor-1:ctx-f118d6dc
>>>>>>>>>> ctx-d9a00f18) Failed to start instance
>>>>>>>>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
>>>>>>>>>> upgrade.
>>>>>>>>>> Unable to send command to router:6
>>>>>>>>>> at
>>>>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>>>>>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManager
>>>>>>>>>> Impl.java:3567)
>>>>>>>>>> at
>>>>>>>>>>
>>>>>>>>>>
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute
>>>>>>>>>> (
>>>>>>>>>> VirtualNetworkApplianceManagerImpl.java:3003)
>>>>>>>>>> at
>>>>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>>>>>>> Impl.applyRules(
>>>>>>>>>> VirtualNetworkApplianceManagerImpl.java:3848)
>>>>>>>>>> at
>>>>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>>>>>>> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
>>>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>>>> at
>>>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>>>>>>> NativeMethodAccessorImpl.java:39)
>>>>>>>>>> at
>>>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>>>>>>> DelegatingMethodAccessorImpl.java:25)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>
>>
>>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Will Stevens <ws...@cloudops.com>.
I just used:
http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/lastSuccessfulBuild/artifact/tools/appliance/dist/systemvmtemplate-2013-11-24-master-xen.vhd.bz2
I ran into the same issue with this template as all the others I have been
trying. Basically, the vm comes up with a read only file system for some
reason. If I run 'fsck' on the vm from XenCenter when the system vms come
up, and then reboot, I can get the vm to come up correctly. Each time I
launch a new vm which requires a virtual router, I have to 'fsck' the VR in
XenCenter as soon as it tries to launch.
This is working for me to launch an instance (but it is quite annoying).
Other than the 'fsck' issue, these templates seem to be working.
ws
On Mon, Nov 25, 2013 at 3:29 PM, Hugo Trippaers <hu...@trippaers.nl> wrote:
> The images should be downloadable from jenkins.buildacloud.org
>
> The 4.2 images are here:
> http://jenkins.buildacloud.org/view/4.2/job/build-systemvm-4.2/
>
> There was something wrong with this build, but i’m working on it. Expect
> the images soon.
>
> Cheers,
>
> Hugo
>
> On 25 nov. 2013, at 21:14, Travis Graham <tg...@tgraham.us> wrote:
>
> > Use the links from the Install Guide instead.
> >
> >
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/management-server-install-flow.html#prepare-system-vm-template
> >
> > Travis
> >
> > On Nov 25, 2013, at 3:01 PM, Will Stevens <ws...@cloudops.com> wrote:
> >
> >> In trying to troubleshoot I think I have found another issue. I went
> looking for a 'more official' source for the system templates. I found
> this:
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Release_Notes/upgrade-instructions.html#upgrade-from-3.0.x-to-4.0
> >>
> >> Which gives a system vm template of (for xen server):
> http://download.cloud.com/templates/4.2/systemvmtemplate-2013-07-12-master-xen.vhd.bz2
> >>
> >> Unfortunately, those system vm templates do not come up (check the
> attached files for images). Basically, the file system comes up as Read
> Only...
> >>
> >> I will go back to the buildacloud system templates to see if I can get
> them working...
> >>
> >> Would love to have someone confirm where we should be getting the
> System VM Templates from for 4.2+.
> >>
> >> Still trying to get System VM Templates to work on 4.3. If anyone has
> this working, please post how you get them working and where you got them
> from.
> >>
> >> Thanks,
> >>
> >> Will
> >>
> >>
> >> On Fri, Nov 22, 2013 at 7:43 AM, Will Stevens <ws...@cloudops.com>
> wrote:
> >> I will try this as a temporary solution. Thank you...
> >>
> >> Will
> >>
> >>
> >> On Fri, Nov 22, 2013 at 6:57 AM, Murali Reddy <Mu...@citrix.com>
> wrote:
> >>
> >> Don¹t understand problem well enough for clean fix, but I updated
> >> 'template_version' from 3.0 to 4.3 of the VR in the domain_router table
> >> that resolved the issue for me.
> >>
> >> On 22/11/13 3:50 PM, "Will Stevens" <ws...@cloudops.com> wrote:
> >>
> >>> Has anyone been able to resolve this issue? This is holding up my
> ability
> >>> to launch VMs and test the fixes to my plugin. I need to resolve this
> >>> issue to move forward...
> >>>
> >>> @Syed, are you still stuck on this as well?
> >>>
> >>> Cheers,
> >>>
> >>> Will
> >>>
> >>>
> >>> On Wed, Nov 20, 2013 at 5:18 PM, Syed Ahmed <sa...@cloudops.com>
> wrote:
> >>>
> >>>> OK here is how far I got debugging this. I think I am missing a small
> >>>> thing. I hope you guys can help.
> >>>>
> >>>> So my VM template has the correct version.
> >>>>
> >>>> root@eng-ns-dev-cs1: /export/secondary/template/tmpl/1/1 # strings
> >>>> f3fc75d9-0240-4c71-a3bf-fb65652e4763.vhd | grep Cloudstack
> >>>>
> >>>> Cloudstack Release* 4.2.0*Tue Nov 19 23:22:37 UTC 2013
> >>>>
> >>>>
> >>>> But in the database I see the following ( table domain_router )
> >>>>
> >>>> *************************** 4. row ***************************
> >>>> id: 11
> >>>> element_id: 4
> >>>> public_mac_address: 06:48:a8:00:00:68
> >>>> public_ip_address: 172.30.91.102
> >>>> public_netmask: 255.255.255.0
> >>>> guest_netmask: NULL
> >>>> guest_ip_address: NULL
> >>>> is_redundant_router: 0
> >>>> priority: 0
> >>>> is_priority_bumpup: 0
> >>>> redundant_state: UNKNOWN
> >>>> stop_pending: 0
> >>>> role: VIRTUAL_ROUTER
> >>>> template_version:*Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST
> 2012*
> >>>> scripts_version: 725d5e5901a62c68aed0dd3463023518
> >>>> vpc_id: NULL
> >>>> 4 rows in set (0.00 sec)
> >>>>
> >>>>
> >>>>
> >>>> I guess this is populated from the VM that gets created. On the xen
> the
> >>>> vm
> >>>> is r-11. I see the following version on that VM
> >>>>
> >>>> root@r-11-VM:~# cat /etc/cloudstack-release
> >>>> Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012
> >>>>
> >>>>
> >>>> This means that Xen is not picking up the template present in the
> >>>> secondary storage. Does Xen cache the vhd files locally to avoid
> coming
> >>>> to
> >>>> the secondary storage? If so, how can I disable that?
> >>>>
> >>>> Also, I was looking at UpgradeRouterTemplateCmd API which basically
> goes
> >>>> through all the VRs and reboots them. It expects that when the reboot
> is
> >>>> completed, the router should have picked up the 4.2.0 version of the
> >>>> template ( see line 4072 in VirtualNetworkApplianceManagerImpl.java )
> I
> >>>> try to do the reboot manually but the template remains the same. Do
> you
> >>>> guys have any more suggestions?
> >>>>
> >>>> Thanks,
> >>>> -Syed
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On Wed 20 Nov 2013 12:55:04 PM EST, Wei ZHOU wrote:
> >>>>
> >>>>>
> >>>>> FYI.
> >>>>>
> >>>>> I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working
> >>>>> after
> >>>>> running *cloudstack-sysvmadm to recreate.*
> >>>>>
> >>>>>
> >>>>> 2013/11/20 Syed Ahmed <sa...@cloudops.com>
> >>>>>
> >>>>>
> >>>>>> +1 Same error. The secondary storage VM and the Console proxy VM
> seem
> >>>>>> to
> >>>>>> be coming up alright. I see this error only when starting the
> virtual
> >>>>>> router which is preventing me from creating any instances.
> >>>>>>
> >>>>>>
> >>>>>> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
> >>>>>>
> >>>>>>
> >>>>>>> I am having the same problem. I got the latest system VMs from:
> >>>>>>>
> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
> >>>>>>> lastSuccessfulBuild/artifact/tools/appliance/dist/
> >>>>>>>
> >>>>>>> Are these the wrong System VM Templates? If so, where should I get
> >>>>>>> the
> >>>>>>> System VM Templates to make this work again?
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Will
> >>>>>>>
> >>>>>>>
> >>>>>>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
> >>>>>>> Alena.Prokharchyk@citrix.com> wrote:
> >>>>>>>
> >>>>>>> Nitin, I had the same problem, but I fixed it by uploading 4.2
> system
> >>>>>>>
> >>>>>>>>
> >>>>>>>> templates to my secondary storage. Make sure you have the latest
> >>>>>>>> too.
> >>>>>>>>
> >>>>>>>> -alena.
> >>>>>>>>
> >>>>>>>> From: Nitin Mehta <Nitin.Mehta@citrix.com<mailto:
> >>>>>>>> Nitin.Mehta@citrix.com>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>> Reply-To:
> >>>>>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org
> >>>>>>>>> "
> >>>>>>>> <
> >>>>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> >>>>>>>> Date: Thursday, November 7, 2013 4:16 PM
> >>>>>>>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
> <
> >>>>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> >>>>>>>> Subject: Router requires upgrade. Unable to send command to router
> >>>>>>>> Error
> >>>>>>>>
> >>>>>>>> Unable to deploy vms in the latest master because of the commit
> >>>>>>>> below.
> >>>>>>>> Is
> >>>>>>>> anyone noticing this on the latest master.
> >>>>>>>> I checked the code and there was a commit made recently 3f5b8f
> >>>>>>>> which is
> >>>>>>>> where the exception points to.
> >>>>>>>>
> >>>>>>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2
> //
> >>>>>>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message::
> >>>>>>>> System
> >>>>>>>> Alert: Low Available Storage in cluster c3 pod p of availability
> >>>>>>>> zone
> >>>>>>>> z3
> >>>>>>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc)
> Add
> >>>>>>>> job-44 into job monitoring
> >>>>>>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
> >>>>>>>> Detecting a change in xstoolsversion for r-6-VM
> >>>>>>>> ERROR [c.c.v.VirtualMachineManagerImpl]
> (Job-Executor-1:ctx-f118d6dc
> >>>>>>>> ctx-d9a00f18) Failed to start instance
> >>>>>>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
> >>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
> >>>>>>>> upgrade.
> >>>>>>>> Unable to send command to router:6
> >>>>>>>> at
> >>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManager
> >>>>>>>> Impl.java:3567)
> >>>>>>>> at
> >>>>>>>>
> >>>>>>>>
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute
> >>>>>>>> (
> >>>>>>>> VirtualNetworkApplianceManagerImpl.java:3003)
> >>>>>>>> at
> >>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>>>> Impl.applyRules(
> >>>>>>>> VirtualNetworkApplianceManagerImpl.java:3848)
> >>>>>>>> at
> >>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>>>> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
> >>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>>>>> at
> >>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
> >>>>>>>> NativeMethodAccessorImpl.java:39)
> >>>>>>>> at
> >>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> >>>>>>>> DelegatingMethodAccessorImpl.java:25)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>
> >>
> >>
> >>
> >>
> >
>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Hugo Trippaers <hu...@trippaers.nl>.
The images should be downloadable from jenkins.buildacloud.org
The 4.2 images are here: http://jenkins.buildacloud.org/view/4.2/job/build-systemvm-4.2/
There was something wrong with this build, but i’m working on it. Expect the images soon.
Cheers,
Hugo
On 25 nov. 2013, at 21:14, Travis Graham <tg...@tgraham.us> wrote:
> Use the links from the Install Guide instead.
>
> http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/management-server-install-flow.html#prepare-system-vm-template
>
> Travis
>
> On Nov 25, 2013, at 3:01 PM, Will Stevens <ws...@cloudops.com> wrote:
>
>> In trying to troubleshoot I think I have found another issue. I went looking for a 'more official' source for the system templates. I found this: http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Release_Notes/upgrade-instructions.html#upgrade-from-3.0.x-to-4.0
>>
>> Which gives a system vm template of (for xen server): http://download.cloud.com/templates/4.2/systemvmtemplate-2013-07-12-master-xen.vhd.bz2
>>
>> Unfortunately, those system vm templates do not come up (check the attached files for images). Basically, the file system comes up as Read Only...
>>
>> I will go back to the buildacloud system templates to see if I can get them working...
>>
>> Would love to have someone confirm where we should be getting the System VM Templates from for 4.2+.
>>
>> Still trying to get System VM Templates to work on 4.3. If anyone has this working, please post how you get them working and where you got them from.
>>
>> Thanks,
>>
>> Will
>>
>>
>> On Fri, Nov 22, 2013 at 7:43 AM, Will Stevens <ws...@cloudops.com> wrote:
>> I will try this as a temporary solution. Thank you...
>>
>> Will
>>
>>
>> On Fri, Nov 22, 2013 at 6:57 AM, Murali Reddy <Mu...@citrix.com> wrote:
>>
>> Don¹t understand problem well enough for clean fix, but I updated
>> 'template_version' from 3.0 to 4.3 of the VR in the domain_router table
>> that resolved the issue for me.
>>
>> On 22/11/13 3:50 PM, "Will Stevens" <ws...@cloudops.com> wrote:
>>
>>> Has anyone been able to resolve this issue? This is holding up my ability
>>> to launch VMs and test the fixes to my plugin. I need to resolve this
>>> issue to move forward...
>>>
>>> @Syed, are you still stuck on this as well?
>>>
>>> Cheers,
>>>
>>> Will
>>>
>>>
>>> On Wed, Nov 20, 2013 at 5:18 PM, Syed Ahmed <sa...@cloudops.com> wrote:
>>>
>>>> OK here is how far I got debugging this. I think I am missing a small
>>>> thing. I hope you guys can help.
>>>>
>>>> So my VM template has the correct version.
>>>>
>>>> root@eng-ns-dev-cs1: /export/secondary/template/tmpl/1/1 # strings
>>>> f3fc75d9-0240-4c71-a3bf-fb65652e4763.vhd | grep Cloudstack
>>>>
>>>> Cloudstack Release* 4.2.0*Tue Nov 19 23:22:37 UTC 2013
>>>>
>>>>
>>>> But in the database I see the following ( table domain_router )
>>>>
>>>> *************************** 4. row ***************************
>>>> id: 11
>>>> element_id: 4
>>>> public_mac_address: 06:48:a8:00:00:68
>>>> public_ip_address: 172.30.91.102
>>>> public_netmask: 255.255.255.0
>>>> guest_netmask: NULL
>>>> guest_ip_address: NULL
>>>> is_redundant_router: 0
>>>> priority: 0
>>>> is_priority_bumpup: 0
>>>> redundant_state: UNKNOWN
>>>> stop_pending: 0
>>>> role: VIRTUAL_ROUTER
>>>> template_version:*Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012*
>>>> scripts_version: 725d5e5901a62c68aed0dd3463023518
>>>> vpc_id: NULL
>>>> 4 rows in set (0.00 sec)
>>>>
>>>>
>>>>
>>>> I guess this is populated from the VM that gets created. On the xen the
>>>> vm
>>>> is r-11. I see the following version on that VM
>>>>
>>>> root@r-11-VM:~# cat /etc/cloudstack-release
>>>> Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012
>>>>
>>>>
>>>> This means that Xen is not picking up the template present in the
>>>> secondary storage. Does Xen cache the vhd files locally to avoid coming
>>>> to
>>>> the secondary storage? If so, how can I disable that?
>>>>
>>>> Also, I was looking at UpgradeRouterTemplateCmd API which basically goes
>>>> through all the VRs and reboots them. It expects that when the reboot is
>>>> completed, the router should have picked up the 4.2.0 version of the
>>>> template ( see line 4072 in VirtualNetworkApplianceManagerImpl.java ) I
>>>> try to do the reboot manually but the template remains the same. Do you
>>>> guys have any more suggestions?
>>>>
>>>> Thanks,
>>>> -Syed
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed 20 Nov 2013 12:55:04 PM EST, Wei ZHOU wrote:
>>>>
>>>>>
>>>>> FYI.
>>>>>
>>>>> I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working
>>>>> after
>>>>> running *cloudstack-sysvmadm to recreate.*
>>>>>
>>>>>
>>>>> 2013/11/20 Syed Ahmed <sa...@cloudops.com>
>>>>>
>>>>>
>>>>>> +1 Same error. The secondary storage VM and the Console proxy VM seem
>>>>>> to
>>>>>> be coming up alright. I see this error only when starting the virtual
>>>>>> router which is preventing me from creating any instances.
>>>>>>
>>>>>>
>>>>>> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
>>>>>>
>>>>>>
>>>>>>> I am having the same problem. I got the latest system VMs from:
>>>>>>> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
>>>>>>> lastSuccessfulBuild/artifact/tools/appliance/dist/
>>>>>>>
>>>>>>> Are these the wrong System VM Templates? If so, where should I get
>>>>>>> the
>>>>>>> System VM Templates to make this work again?
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Will
>>>>>>>
>>>>>>>
>>>>>>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
>>>>>>> Alena.Prokharchyk@citrix.com> wrote:
>>>>>>>
>>>>>>> Nitin, I had the same problem, but I fixed it by uploading 4.2 system
>>>>>>>
>>>>>>>>
>>>>>>>> templates to my secondary storage. Make sure you have the latest
>>>>>>>> too.
>>>>>>>>
>>>>>>>> -alena.
>>>>>>>>
>>>>>>>> From: Nitin Mehta <Nitin.Mehta@citrix.com<mailto:
>>>>>>>> Nitin.Mehta@citrix.com>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>> Reply-To:
>>>>>>>> "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org
>>>>>>>>> "
>>>>>>>> <
>>>>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>>>>>>> Date: Thursday, November 7, 2013 4:16 PM
>>>>>>>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
>>>>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>>>>>>> Subject: Router requires upgrade. Unable to send command to router
>>>>>>>> Error
>>>>>>>>
>>>>>>>> Unable to deploy vms in the latest master because of the commit
>>>>>>>> below.
>>>>>>>> Is
>>>>>>>> anyone noticing this on the latest master.
>>>>>>>> I checked the code and there was a commit made recently 3f5b8f
>>>>>>>> which is
>>>>>>>> where the exception points to.
>>>>>>>>
>>>>>>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 //
>>>>>>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message::
>>>>>>>> System
>>>>>>>> Alert: Low Available Storage in cluster c3 pod p of availability
>>>>>>>> zone
>>>>>>>> z3
>>>>>>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add
>>>>>>>> job-44 into job monitoring
>>>>>>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
>>>>>>>> Detecting a change in xstoolsversion for r-6-VM
>>>>>>>> ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc
>>>>>>>> ctx-d9a00f18) Failed to start instance
>>>>>>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
>>>>>>>> upgrade.
>>>>>>>> Unable to send command to router:6
>>>>>>>> at
>>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>>>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManager
>>>>>>>> Impl.java:3567)
>>>>>>>> at
>>>>>>>>
>>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute
>>>>>>>> (
>>>>>>>> VirtualNetworkApplianceManagerImpl.java:3003)
>>>>>>>> at
>>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>>>>> Impl.applyRules(
>>>>>>>> VirtualNetworkApplianceManagerImpl.java:3848)
>>>>>>>> at
>>>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>>>>> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
>>>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>>>> at
>>>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>>>>> NativeMethodAccessorImpl.java:39)
>>>>>>>> at
>>>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>>>>> DelegatingMethodAccessorImpl.java:25)
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>>
>>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Travis Graham <tg...@tgraham.us>.
Use the links from the Install Guide instead.
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Installation_Guide/management-server-install-flow.html#prepare-system-vm-template
Travis
On Nov 25, 2013, at 3:01 PM, Will Stevens <ws...@cloudops.com> wrote:
> In trying to troubleshoot I think I have found another issue. I went looking for a 'more official' source for the system templates. I found this: http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Release_Notes/upgrade-instructions.html#upgrade-from-3.0.x-to-4.0
>
> Which gives a system vm template of (for xen server): http://download.cloud.com/templates/4.2/systemvmtemplate-2013-07-12-master-xen.vhd.bz2
>
> Unfortunately, those system vm templates do not come up (check the attached files for images). Basically, the file system comes up as Read Only...
>
> I will go back to the buildacloud system templates to see if I can get them working...
>
> Would love to have someone confirm where we should be getting the System VM Templates from for 4.2+.
>
> Still trying to get System VM Templates to work on 4.3. If anyone has this working, please post how you get them working and where you got them from.
>
> Thanks,
>
> Will
>
>
> On Fri, Nov 22, 2013 at 7:43 AM, Will Stevens <ws...@cloudops.com> wrote:
> I will try this as a temporary solution. Thank you...
>
> Will
>
>
> On Fri, Nov 22, 2013 at 6:57 AM, Murali Reddy <Mu...@citrix.com> wrote:
>
> Don¹t understand problem well enough for clean fix, but I updated
> 'template_version' from 3.0 to 4.3 of the VR in the domain_router table
> that resolved the issue for me.
>
> On 22/11/13 3:50 PM, "Will Stevens" <ws...@cloudops.com> wrote:
>
> >Has anyone been able to resolve this issue? This is holding up my ability
> >to launch VMs and test the fixes to my plugin. I need to resolve this
> >issue to move forward...
> >
> >@Syed, are you still stuck on this as well?
> >
> >Cheers,
> >
> >Will
> >
> >
> >On Wed, Nov 20, 2013 at 5:18 PM, Syed Ahmed <sa...@cloudops.com> wrote:
> >
> >> OK here is how far I got debugging this. I think I am missing a small
> >> thing. I hope you guys can help.
> >>
> >> So my VM template has the correct version.
> >>
> >> root@eng-ns-dev-cs1: /export/secondary/template/tmpl/1/1 # strings
> >> f3fc75d9-0240-4c71-a3bf-fb65652e4763.vhd | grep Cloudstack
> >>
> >> Cloudstack Release* 4.2.0*Tue Nov 19 23:22:37 UTC 2013
> >>
> >>
> >> But in the database I see the following ( table domain_router )
> >>
> >> *************************** 4. row ***************************
> >> id: 11
> >> element_id: 4
> >> public_mac_address: 06:48:a8:00:00:68
> >> public_ip_address: 172.30.91.102
> >> public_netmask: 255.255.255.0
> >> guest_netmask: NULL
> >> guest_ip_address: NULL
> >> is_redundant_router: 0
> >> priority: 0
> >> is_priority_bumpup: 0
> >> redundant_state: UNKNOWN
> >> stop_pending: 0
> >> role: VIRTUAL_ROUTER
> >> template_version:*Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012*
> >> scripts_version: 725d5e5901a62c68aed0dd3463023518
> >> vpc_id: NULL
> >> 4 rows in set (0.00 sec)
> >>
> >>
> >>
> >> I guess this is populated from the VM that gets created. On the xen the
> >>vm
> >> is r-11. I see the following version on that VM
> >>
> >> root@r-11-VM:~# cat /etc/cloudstack-release
> >> Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012
> >>
> >>
> >> This means that Xen is not picking up the template present in the
> >> secondary storage. Does Xen cache the vhd files locally to avoid coming
> >>to
> >> the secondary storage? If so, how can I disable that?
> >>
> >> Also, I was looking at UpgradeRouterTemplateCmd API which basically goes
> >> through all the VRs and reboots them. It expects that when the reboot is
> >> completed, the router should have picked up the 4.2.0 version of the
> >> template ( see line 4072 in VirtualNetworkApplianceManagerImpl.java ) I
> >> try to do the reboot manually but the template remains the same. Do you
> >> guys have any more suggestions?
> >>
> >> Thanks,
> >> -Syed
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed 20 Nov 2013 12:55:04 PM EST, Wei ZHOU wrote:
> >>
> >>>
> >>> FYI.
> >>>
> >>> I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working
> >>>after
> >>> running *cloudstack-sysvmadm to recreate.*
> >>>
> >>>
> >>> 2013/11/20 Syed Ahmed <sa...@cloudops.com>
> >>>
> >>>
> >>>> +1 Same error. The secondary storage VM and the Console proxy VM seem
> >>>>to
> >>>> be coming up alright. I see this error only when starting the virtual
> >>>> router which is preventing me from creating any instances.
> >>>>
> >>>>
> >>>> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
> >>>>
> >>>>
> >>>>> I am having the same problem. I got the latest system VMs from:
> >>>>> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
> >>>>> lastSuccessfulBuild/artifact/tools/appliance/dist/
> >>>>>
> >>>>> Are these the wrong System VM Templates? If so, where should I get
> >>>>>the
> >>>>> System VM Templates to make this work again?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Will
> >>>>>
> >>>>>
> >>>>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
> >>>>> Alena.Prokharchyk@citrix.com> wrote:
> >>>>>
> >>>>> Nitin, I had the same problem, but I fixed it by uploading 4.2 system
> >>>>>
> >>>>>>
> >>>>>> templates to my secondary storage. Make sure you have the latest
> >>>>>>too.
> >>>>>>
> >>>>>> -alena.
> >>>>>>
> >>>>>> From: Nitin Mehta <Nitin.Mehta@citrix.com<mailto:
> >>>>>> Nitin.Mehta@citrix.com>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> Reply-To:
> >>>>>>"dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org
> >>>>>> >"
> >>>>>> <
> >>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> >>>>>> Date: Thursday, November 7, 2013 4:16 PM
> >>>>>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
> >>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> >>>>>> Subject: Router requires upgrade. Unable to send command to router
> >>>>>> Error
> >>>>>>
> >>>>>> Unable to deploy vms in the latest master because of the commit
> >>>>>>below.
> >>>>>> Is
> >>>>>> anyone noticing this on the latest master.
> >>>>>> I checked the code and there was a commit made recently 3f5b8f
> >>>>>>which is
> >>>>>> where the exception points to.
> >>>>>>
> >>>>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 //
> >>>>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message::
> >>>>>>System
> >>>>>> Alert: Low Available Storage in cluster c3 pod p of availability
> >>>>>>zone
> >>>>>> z3
> >>>>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add
> >>>>>> job-44 into job monitoring
> >>>>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
> >>>>>> Detecting a change in xstoolsversion for r-6-VM
> >>>>>> ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc
> >>>>>> ctx-d9a00f18) Failed to start instance
> >>>>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
> >>>>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
> >>>>>> upgrade.
> >>>>>> Unable to send command to router:6
> >>>>>> at
> >>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManager
> >>>>>> Impl.java:3567)
> >>>>>> at
> >>>>>>
> >>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute
> >>>>>>(
> >>>>>> VirtualNetworkApplianceManagerImpl.java:3003)
> >>>>>> at
> >>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>> Impl.applyRules(
> >>>>>> VirtualNetworkApplianceManagerImpl.java:3848)
> >>>>>> at
> >>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>>> at
> >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
> >>>>>> NativeMethodAccessorImpl.java:39)
> >>>>>> at
> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> >>>>>> DelegatingMethodAccessorImpl.java:25)
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >
>
>
>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Will Stevens <ws...@cloudops.com>.
In trying to troubleshoot I think I have found another issue. I went
looking for a 'more official' source for the system templates. I found
this:
http://cloudstack.apache.org/docs/en-US/Apache_CloudStack/4.2.0/html/Release_Notes/upgrade-instructions.html#upgrade-from-3.0.x-to-4.0
Which gives a system vm template of (for xen server):
http://download.cloud.com/templates/4.2/systemvmtemplate-2013-07-12-master-xen.vhd.bz2
Unfortunately, those system vm templates do not come up (check the attached
files for images). Basically, the file system comes up as Read Only...
I will go back to the buildacloud system templates to see if I can get them
working...
Would love to have someone confirm where we should be getting the System VM
Templates from for 4.2+.
Still trying to get System VM Templates to work on 4.3. If anyone has this
working, please post how you get them working and where you got them from.
Thanks,
Will
On Fri, Nov 22, 2013 at 7:43 AM, Will Stevens <ws...@cloudops.com> wrote:
> I will try this as a temporary solution. Thank you...
>
> Will
>
>
> On Fri, Nov 22, 2013 at 6:57 AM, Murali Reddy <Mu...@citrix.com>wrote:
>
>>
>> Don¹t understand problem well enough for clean fix, but I updated
>> 'template_version' from 3.0 to 4.3 of the VR in the domain_router table
>> that resolved the issue for me.
>>
>> On 22/11/13 3:50 PM, "Will Stevens" <ws...@cloudops.com> wrote:
>>
>> >Has anyone been able to resolve this issue? This is holding up my
>> ability
>> >to launch VMs and test the fixes to my plugin. I need to resolve this
>> >issue to move forward...
>> >
>> >@Syed, are you still stuck on this as well?
>> >
>> >Cheers,
>> >
>> >Will
>> >
>> >
>> >On Wed, Nov 20, 2013 at 5:18 PM, Syed Ahmed <sa...@cloudops.com> wrote:
>> >
>> >> OK here is how far I got debugging this. I think I am missing a small
>> >> thing. I hope you guys can help.
>> >>
>> >> So my VM template has the correct version.
>> >>
>> >> root@eng-ns-dev-cs1: /export/secondary/template/tmpl/1/1 # strings
>> >> f3fc75d9-0240-4c71-a3bf-fb65652e4763.vhd | grep Cloudstack
>> >>
>> >> Cloudstack Release* 4.2.0*Tue Nov 19 23:22:37 UTC 2013
>> >>
>> >>
>> >> But in the database I see the following ( table domain_router )
>> >>
>> >> *************************** 4. row ***************************
>> >> id: 11
>> >> element_id: 4
>> >> public_mac_address: 06:48:a8:00:00:68
>> >> public_ip_address: 172.30.91.102
>> >> public_netmask: 255.255.255.0
>> >> guest_netmask: NULL
>> >> guest_ip_address: NULL
>> >> is_redundant_router: 0
>> >> priority: 0
>> >> is_priority_bumpup: 0
>> >> redundant_state: UNKNOWN
>> >> stop_pending: 0
>> >> role: VIRTUAL_ROUTER
>> >> template_version:*Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST
>> 2012*
>> >> scripts_version: 725d5e5901a62c68aed0dd3463023518
>> >> vpc_id: NULL
>> >> 4 rows in set (0.00 sec)
>> >>
>> >>
>> >>
>> >> I guess this is populated from the VM that gets created. On the xen the
>> >>vm
>> >> is r-11. I see the following version on that VM
>> >>
>> >> root@r-11-VM:~# cat /etc/cloudstack-release
>> >> Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012
>> >>
>> >>
>> >> This means that Xen is not picking up the template present in the
>> >> secondary storage. Does Xen cache the vhd files locally to avoid coming
>> >>to
>> >> the secondary storage? If so, how can I disable that?
>> >>
>> >> Also, I was looking at UpgradeRouterTemplateCmd API which basically
>> goes
>> >> through all the VRs and reboots them. It expects that when the reboot
>> is
>> >> completed, the router should have picked up the 4.2.0 version of the
>> >> template ( see line 4072 in VirtualNetworkApplianceManagerImpl.java ) I
>> >> try to do the reboot manually but the template remains the same. Do you
>> >> guys have any more suggestions?
>> >>
>> >> Thanks,
>> >> -Syed
>> >>
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> On Wed 20 Nov 2013 12:55:04 PM EST, Wei ZHOU wrote:
>> >>
>> >>>
>> >>> FYI.
>> >>>
>> >>> I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working
>> >>>after
>> >>> running *cloudstack-sysvmadm to recreate.*
>> >>>
>> >>>
>> >>> 2013/11/20 Syed Ahmed <sa...@cloudops.com>
>> >>>
>> >>>
>> >>>> +1 Same error. The secondary storage VM and the Console proxy VM seem
>> >>>>to
>> >>>> be coming up alright. I see this error only when starting the virtual
>> >>>> router which is preventing me from creating any instances.
>> >>>>
>> >>>>
>> >>>> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
>> >>>>
>> >>>>
>> >>>>> I am having the same problem. I got the latest system VMs from:
>> >>>>>
>> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
>> >>>>> lastSuccessfulBuild/artifact/tools/appliance/dist/
>> >>>>>
>> >>>>> Are these the wrong System VM Templates? If so, where should I get
>> >>>>>the
>> >>>>> System VM Templates to make this work again?
>> >>>>>
>> >>>>> Thanks,
>> >>>>>
>> >>>>> Will
>> >>>>>
>> >>>>>
>> >>>>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
>> >>>>> Alena.Prokharchyk@citrix.com> wrote:
>> >>>>>
>> >>>>> Nitin, I had the same problem, but I fixed it by uploading 4.2
>> system
>> >>>>>
>> >>>>>>
>> >>>>>> templates to my secondary storage. Make sure you have the latest
>> >>>>>>too.
>> >>>>>>
>> >>>>>> -alena.
>> >>>>>>
>> >>>>>> From: Nitin Mehta <Nitin.Mehta@citrix.com<mailto:
>> >>>>>> Nitin.Mehta@citrix.com>
>> >>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>> Reply-To:
>> >>>>>>"dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org
>> >>>>>> >"
>> >>>>>> <
>> >>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>> >>>>>> Date: Thursday, November 7, 2013 4:16 PM
>> >>>>>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
>> <
>> >>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>> >>>>>> Subject: Router requires upgrade. Unable to send command to router
>> >>>>>> Error
>> >>>>>>
>> >>>>>> Unable to deploy vms in the latest master because of the commit
>> >>>>>>below.
>> >>>>>> Is
>> >>>>>> anyone noticing this on the latest master.
>> >>>>>> I checked the code and there was a commit made recently 3f5b8f
>> >>>>>>which is
>> >>>>>> where the exception points to.
>> >>>>>>
>> >>>>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 //
>> >>>>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message::
>> >>>>>>System
>> >>>>>> Alert: Low Available Storage in cluster c3 pod p of availability
>> >>>>>>zone
>> >>>>>> z3
>> >>>>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc)
>> Add
>> >>>>>> job-44 into job monitoring
>> >>>>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
>> >>>>>> Detecting a change in xstoolsversion for r-6-VM
>> >>>>>> ERROR [c.c.v.VirtualMachineManagerImpl]
>> (Job-Executor-1:ctx-f118d6dc
>> >>>>>> ctx-d9a00f18) Failed to start instance
>> >>>>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
>> >>>>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
>> >>>>>> upgrade.
>> >>>>>> Unable to send command to router:6
>> >>>>>> at
>> >>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>> >>>>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManager
>> >>>>>> Impl.java:3567)
>> >>>>>> at
>> >>>>>>
>>
>> >>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute
>> >>>>>>(
>> >>>>>> VirtualNetworkApplianceManagerImpl.java:3003)
>> >>>>>> at
>> >>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>> >>>>>> Impl.applyRules(
>> >>>>>> VirtualNetworkApplianceManagerImpl.java:3848)
>> >>>>>> at
>> >>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>> >>>>>> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
>> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> >>>>>> at
>> >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>> >>>>>> NativeMethodAccessorImpl.java:39)
>> >>>>>> at
>> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>> >>>>>> DelegatingMethodAccessorImpl.java:25)
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>
>> >>
>> >
>>
>>
>>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Will Stevens <ws...@cloudops.com>.
I will try this as a temporary solution. Thank you...
Will
On Fri, Nov 22, 2013 at 6:57 AM, Murali Reddy <Mu...@citrix.com>wrote:
>
> Don¹t understand problem well enough for clean fix, but I updated
> 'template_version' from 3.0 to 4.3 of the VR in the domain_router table
> that resolved the issue for me.
>
> On 22/11/13 3:50 PM, "Will Stevens" <ws...@cloudops.com> wrote:
>
> >Has anyone been able to resolve this issue? This is holding up my ability
> >to launch VMs and test the fixes to my plugin. I need to resolve this
> >issue to move forward...
> >
> >@Syed, are you still stuck on this as well?
> >
> >Cheers,
> >
> >Will
> >
> >
> >On Wed, Nov 20, 2013 at 5:18 PM, Syed Ahmed <sa...@cloudops.com> wrote:
> >
> >> OK here is how far I got debugging this. I think I am missing a small
> >> thing. I hope you guys can help.
> >>
> >> So my VM template has the correct version.
> >>
> >> root@eng-ns-dev-cs1: /export/secondary/template/tmpl/1/1 # strings
> >> f3fc75d9-0240-4c71-a3bf-fb65652e4763.vhd | grep Cloudstack
> >>
> >> Cloudstack Release* 4.2.0*Tue Nov 19 23:22:37 UTC 2013
> >>
> >>
> >> But in the database I see the following ( table domain_router )
> >>
> >> *************************** 4. row ***************************
> >> id: 11
> >> element_id: 4
> >> public_mac_address: 06:48:a8:00:00:68
> >> public_ip_address: 172.30.91.102
> >> public_netmask: 255.255.255.0
> >> guest_netmask: NULL
> >> guest_ip_address: NULL
> >> is_redundant_router: 0
> >> priority: 0
> >> is_priority_bumpup: 0
> >> redundant_state: UNKNOWN
> >> stop_pending: 0
> >> role: VIRTUAL_ROUTER
> >> template_version:*Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012*
> >> scripts_version: 725d5e5901a62c68aed0dd3463023518
> >> vpc_id: NULL
> >> 4 rows in set (0.00 sec)
> >>
> >>
> >>
> >> I guess this is populated from the VM that gets created. On the xen the
> >>vm
> >> is r-11. I see the following version on that VM
> >>
> >> root@r-11-VM:~# cat /etc/cloudstack-release
> >> Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012
> >>
> >>
> >> This means that Xen is not picking up the template present in the
> >> secondary storage. Does Xen cache the vhd files locally to avoid coming
> >>to
> >> the secondary storage? If so, how can I disable that?
> >>
> >> Also, I was looking at UpgradeRouterTemplateCmd API which basically goes
> >> through all the VRs and reboots them. It expects that when the reboot is
> >> completed, the router should have picked up the 4.2.0 version of the
> >> template ( see line 4072 in VirtualNetworkApplianceManagerImpl.java ) I
> >> try to do the reboot manually but the template remains the same. Do you
> >> guys have any more suggestions?
> >>
> >> Thanks,
> >> -Syed
> >>
> >>
> >>
> >>
> >>
> >>
> >> On Wed 20 Nov 2013 12:55:04 PM EST, Wei ZHOU wrote:
> >>
> >>>
> >>> FYI.
> >>>
> >>> I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working
> >>>after
> >>> running *cloudstack-sysvmadm to recreate.*
> >>>
> >>>
> >>> 2013/11/20 Syed Ahmed <sa...@cloudops.com>
> >>>
> >>>
> >>>> +1 Same error. The secondary storage VM and the Console proxy VM seem
> >>>>to
> >>>> be coming up alright. I see this error only when starting the virtual
> >>>> router which is preventing me from creating any instances.
> >>>>
> >>>>
> >>>> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
> >>>>
> >>>>
> >>>>> I am having the same problem. I got the latest system VMs from:
> >>>>>
> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
> >>>>> lastSuccessfulBuild/artifact/tools/appliance/dist/
> >>>>>
> >>>>> Are these the wrong System VM Templates? If so, where should I get
> >>>>>the
> >>>>> System VM Templates to make this work again?
> >>>>>
> >>>>> Thanks,
> >>>>>
> >>>>> Will
> >>>>>
> >>>>>
> >>>>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
> >>>>> Alena.Prokharchyk@citrix.com> wrote:
> >>>>>
> >>>>> Nitin, I had the same problem, but I fixed it by uploading 4.2 system
> >>>>>
> >>>>>>
> >>>>>> templates to my secondary storage. Make sure you have the latest
> >>>>>>too.
> >>>>>>
> >>>>>> -alena.
> >>>>>>
> >>>>>> From: Nitin Mehta <Nitin.Mehta@citrix.com<mailto:
> >>>>>> Nitin.Mehta@citrix.com>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>> Reply-To:
> >>>>>>"dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org
> >>>>>> >"
> >>>>>> <
> >>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> >>>>>> Date: Thursday, November 7, 2013 4:16 PM
> >>>>>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
> >>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> >>>>>> Subject: Router requires upgrade. Unable to send command to router
> >>>>>> Error
> >>>>>>
> >>>>>> Unable to deploy vms in the latest master because of the commit
> >>>>>>below.
> >>>>>> Is
> >>>>>> anyone noticing this on the latest master.
> >>>>>> I checked the code and there was a commit made recently 3f5b8f
> >>>>>>which is
> >>>>>> where the exception points to.
> >>>>>>
> >>>>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 //
> >>>>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message::
> >>>>>>System
> >>>>>> Alert: Low Available Storage in cluster c3 pod p of availability
> >>>>>>zone
> >>>>>> z3
> >>>>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add
> >>>>>> job-44 into job monitoring
> >>>>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
> >>>>>> Detecting a change in xstoolsversion for r-6-VM
> >>>>>> ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc
> >>>>>> ctx-d9a00f18) Failed to start instance
> >>>>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
> >>>>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
> >>>>>> upgrade.
> >>>>>> Unable to send command to router:6
> >>>>>> at
> >>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManager
> >>>>>> Impl.java:3567)
> >>>>>> at
> >>>>>>
> >>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute
> >>>>>>(
> >>>>>> VirtualNetworkApplianceManagerImpl.java:3003)
> >>>>>> at
> >>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>> Impl.applyRules(
> >>>>>> VirtualNetworkApplianceManagerImpl.java:3848)
> >>>>>> at
> >>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
> >>>>>> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
> >>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> >>>>>> at
> >>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
> >>>>>> NativeMethodAccessorImpl.java:39)
> >>>>>> at
> >>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
> >>>>>> DelegatingMethodAccessorImpl.java:25)
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>
> >>>>>
> >>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >>
> >
>
>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Murali Reddy <Mu...@citrix.com>.
Don¹t understand problem well enough for clean fix, but I updated
'template_version' from 3.0 to 4.3 of the VR in the domain_router table
that resolved the issue for me.
On 22/11/13 3:50 PM, "Will Stevens" <ws...@cloudops.com> wrote:
>Has anyone been able to resolve this issue? This is holding up my ability
>to launch VMs and test the fixes to my plugin. I need to resolve this
>issue to move forward...
>
>@Syed, are you still stuck on this as well?
>
>Cheers,
>
>Will
>
>
>On Wed, Nov 20, 2013 at 5:18 PM, Syed Ahmed <sa...@cloudops.com> wrote:
>
>> OK here is how far I got debugging this. I think I am missing a small
>> thing. I hope you guys can help.
>>
>> So my VM template has the correct version.
>>
>> root@eng-ns-dev-cs1: /export/secondary/template/tmpl/1/1 # strings
>> f3fc75d9-0240-4c71-a3bf-fb65652e4763.vhd | grep Cloudstack
>>
>> Cloudstack Release* 4.2.0*Tue Nov 19 23:22:37 UTC 2013
>>
>>
>> But in the database I see the following ( table domain_router )
>>
>> *************************** 4. row ***************************
>> id: 11
>> element_id: 4
>> public_mac_address: 06:48:a8:00:00:68
>> public_ip_address: 172.30.91.102
>> public_netmask: 255.255.255.0
>> guest_netmask: NULL
>> guest_ip_address: NULL
>> is_redundant_router: 0
>> priority: 0
>> is_priority_bumpup: 0
>> redundant_state: UNKNOWN
>> stop_pending: 0
>> role: VIRTUAL_ROUTER
>> template_version:*Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012*
>> scripts_version: 725d5e5901a62c68aed0dd3463023518
>> vpc_id: NULL
>> 4 rows in set (0.00 sec)
>>
>>
>>
>> I guess this is populated from the VM that gets created. On the xen the
>>vm
>> is r-11. I see the following version on that VM
>>
>> root@r-11-VM:~# cat /etc/cloudstack-release
>> Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012
>>
>>
>> This means that Xen is not picking up the template present in the
>> secondary storage. Does Xen cache the vhd files locally to avoid coming
>>to
>> the secondary storage? If so, how can I disable that?
>>
>> Also, I was looking at UpgradeRouterTemplateCmd API which basically goes
>> through all the VRs and reboots them. It expects that when the reboot is
>> completed, the router should have picked up the 4.2.0 version of the
>> template ( see line 4072 in VirtualNetworkApplianceManagerImpl.java ) I
>> try to do the reboot manually but the template remains the same. Do you
>> guys have any more suggestions?
>>
>> Thanks,
>> -Syed
>>
>>
>>
>>
>>
>>
>> On Wed 20 Nov 2013 12:55:04 PM EST, Wei ZHOU wrote:
>>
>>>
>>> FYI.
>>>
>>> I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working
>>>after
>>> running *cloudstack-sysvmadm to recreate.*
>>>
>>>
>>> 2013/11/20 Syed Ahmed <sa...@cloudops.com>
>>>
>>>
>>>> +1 Same error. The secondary storage VM and the Console proxy VM seem
>>>>to
>>>> be coming up alright. I see this error only when starting the virtual
>>>> router which is preventing me from creating any instances.
>>>>
>>>>
>>>> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
>>>>
>>>>
>>>>> I am having the same problem. I got the latest system VMs from:
>>>>> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
>>>>> lastSuccessfulBuild/artifact/tools/appliance/dist/
>>>>>
>>>>> Are these the wrong System VM Templates? If so, where should I get
>>>>>the
>>>>> System VM Templates to make this work again?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Will
>>>>>
>>>>>
>>>>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
>>>>> Alena.Prokharchyk@citrix.com> wrote:
>>>>>
>>>>> Nitin, I had the same problem, but I fixed it by uploading 4.2 system
>>>>>
>>>>>>
>>>>>> templates to my secondary storage. Make sure you have the latest
>>>>>>too.
>>>>>>
>>>>>> -alena.
>>>>>>
>>>>>> From: Nitin Mehta <Nitin.Mehta@citrix.com<mailto:
>>>>>> Nitin.Mehta@citrix.com>
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>> Reply-To:
>>>>>>"dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org
>>>>>> >"
>>>>>> <
>>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>>>>> Date: Thursday, November 7, 2013 4:16 PM
>>>>>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
>>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>>>>> Subject: Router requires upgrade. Unable to send command to router
>>>>>> Error
>>>>>>
>>>>>> Unable to deploy vms in the latest master because of the commit
>>>>>>below.
>>>>>> Is
>>>>>> anyone noticing this on the latest master.
>>>>>> I checked the code and there was a commit made recently 3f5b8f
>>>>>>which is
>>>>>> where the exception points to.
>>>>>>
>>>>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 //
>>>>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message::
>>>>>>System
>>>>>> Alert: Low Available Storage in cluster c3 pod p of availability
>>>>>>zone
>>>>>> z3
>>>>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add
>>>>>> job-44 into job monitoring
>>>>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
>>>>>> Detecting a change in xstoolsversion for r-6-VM
>>>>>> ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc
>>>>>> ctx-d9a00f18) Failed to start instance
>>>>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
>>>>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
>>>>>> upgrade.
>>>>>> Unable to send command to router:6
>>>>>> at
>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManager
>>>>>> Impl.java:3567)
>>>>>> at
>>>>>>
>>>>>>com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute
>>>>>>(
>>>>>> VirtualNetworkApplianceManagerImpl.java:3003)
>>>>>> at
>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>>> Impl.applyRules(
>>>>>> VirtualNetworkApplianceManagerImpl.java:3848)
>>>>>> at
>>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>>> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
>>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>>> at
>>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>>> NativeMethodAccessorImpl.java:39)
>>>>>> at
>>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>>> DelegatingMethodAccessorImpl.java:25)
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Will Stevens <ws...@cloudops.com>.
Has anyone been able to resolve this issue? This is holding up my ability
to launch VMs and test the fixes to my plugin. I need to resolve this
issue to move forward...
@Syed, are you still stuck on this as well?
Cheers,
Will
On Wed, Nov 20, 2013 at 5:18 PM, Syed Ahmed <sa...@cloudops.com> wrote:
> OK here is how far I got debugging this. I think I am missing a small
> thing. I hope you guys can help.
>
> So my VM template has the correct version.
>
> root@eng-ns-dev-cs1: /export/secondary/template/tmpl/1/1 # strings
> f3fc75d9-0240-4c71-a3bf-fb65652e4763.vhd | grep Cloudstack
>
> Cloudstack Release* 4.2.0*Tue Nov 19 23:22:37 UTC 2013
>
>
> But in the database I see the following ( table domain_router )
>
> *************************** 4. row ***************************
> id: 11
> element_id: 4
> public_mac_address: 06:48:a8:00:00:68
> public_ip_address: 172.30.91.102
> public_netmask: 255.255.255.0
> guest_netmask: NULL
> guest_ip_address: NULL
> is_redundant_router: 0
> priority: 0
> is_priority_bumpup: 0
> redundant_state: UNKNOWN
> stop_pending: 0
> role: VIRTUAL_ROUTER
> template_version:*Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012*
> scripts_version: 725d5e5901a62c68aed0dd3463023518
> vpc_id: NULL
> 4 rows in set (0.00 sec)
>
>
>
> I guess this is populated from the VM that gets created. On the xen the vm
> is r-11. I see the following version on that VM
>
> root@r-11-VM:~# cat /etc/cloudstack-release
> Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012
>
>
> This means that Xen is not picking up the template present in the
> secondary storage. Does Xen cache the vhd files locally to avoid coming to
> the secondary storage? If so, how can I disable that?
>
> Also, I was looking at UpgradeRouterTemplateCmd API which basically goes
> through all the VRs and reboots them. It expects that when the reboot is
> completed, the router should have picked up the 4.2.0 version of the
> template ( see line 4072 in VirtualNetworkApplianceManagerImpl.java ) I
> try to do the reboot manually but the template remains the same. Do you
> guys have any more suggestions?
>
> Thanks,
> -Syed
>
>
>
>
>
>
> On Wed 20 Nov 2013 12:55:04 PM EST, Wei ZHOU wrote:
>
>>
>> FYI.
>>
>> I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working after
>> running *cloudstack-sysvmadm to recreate.*
>>
>>
>> 2013/11/20 Syed Ahmed <sa...@cloudops.com>
>>
>>
>>> +1 Same error. The secondary storage VM and the Console proxy VM seem to
>>> be coming up alright. I see this error only when starting the virtual
>>> router which is preventing me from creating any instances.
>>>
>>>
>>> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
>>>
>>>
>>>> I am having the same problem. I got the latest system VMs from:
>>>> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
>>>> lastSuccessfulBuild/artifact/tools/appliance/dist/
>>>>
>>>> Are these the wrong System VM Templates? If so, where should I get the
>>>> System VM Templates to make this work again?
>>>>
>>>> Thanks,
>>>>
>>>> Will
>>>>
>>>>
>>>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
>>>> Alena.Prokharchyk@citrix.com> wrote:
>>>>
>>>> Nitin, I had the same problem, but I fixed it by uploading 4.2 system
>>>>
>>>>>
>>>>> templates to my secondary storage. Make sure you have the latest too.
>>>>>
>>>>> -alena.
>>>>>
>>>>> From: Nitin Mehta <Nitin.Mehta@citrix.com<mailto:
>>>>> Nitin.Mehta@citrix.com>
>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>> Reply-To: "dev@cloudstack.apache.org<mailto:dev@cloudstack.apache.org
>>>>> >"
>>>>> <
>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>>>> Date: Thursday, November 7, 2013 4:16 PM
>>>>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
>>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>>>> Subject: Router requires upgrade. Unable to send command to router
>>>>> Error
>>>>>
>>>>> Unable to deploy vms in the latest master because of the commit below.
>>>>> Is
>>>>> anyone noticing this on the latest master.
>>>>> I checked the code and there was a commit made recently 3f5b8f which is
>>>>> where the exception points to.
>>>>>
>>>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 //
>>>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message:: System
>>>>> Alert: Low Available Storage in cluster c3 pod p of availability zone
>>>>> z3
>>>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add
>>>>> job-44 into job monitoring
>>>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
>>>>> Detecting a change in xstoolsversion for r-6-VM
>>>>> ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc
>>>>> ctx-d9a00f18) Failed to start instance
>>>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
>>>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
>>>>> upgrade.
>>>>> Unable to send command to router:6
>>>>> at
>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManager
>>>>> Impl.java:3567)
>>>>> at
>>>>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute(
>>>>> VirtualNetworkApplianceManagerImpl.java:3003)
>>>>> at
>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>> Impl.applyRules(
>>>>> VirtualNetworkApplianceManagerImpl.java:3848)
>>>>> at
>>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>>> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
>>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> at
>>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>>>> NativeMethodAccessorImpl.java:39)
>>>>> at
>>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>>> DelegatingMethodAccessorImpl.java:25)
>>>>>
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>
>>
>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Syed Ahmed <sa...@cloudops.com>.
OK here is how far I got debugging this. I think I am missing a small
thing. I hope you guys can help.
So my VM template has the correct version.
root@eng-ns-dev-cs1: /export/secondary/template/tmpl/1/1 # strings f3fc75d9-0240-4c71-a3bf-fb65652e4763.vhd | grep Cloudstack
Cloudstack Release* 4.2.0*Tue Nov 19 23:22:37 UTC 2013
But in the database I see the following ( table domain_router )
*************************** 4. row ***************************
id: 11
element_id: 4
public_mac_address: 06:48:a8:00:00:68
public_ip_address: 172.30.91.102
public_netmask: 255.255.255.0
guest_netmask: NULL
guest_ip_address: NULL
is_redundant_router: 0
priority: 0
is_priority_bumpup: 0
redundant_state: UNKNOWN
stop_pending: 0
role: VIRTUAL_ROUTER
template_version:*Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012*
scripts_version: 725d5e5901a62c68aed0dd3463023518
vpc_id: NULL
4 rows in set (0.00 sec)
I guess this is populated from the VM that gets created. On the xen the
vm is r-11. I see the following version on that VM
root@r-11-VM:~# cat /etc/cloudstack-release
Cloudstack Release 3.0 Mon Feb 6 15:10:04 PST 2012
This means that Xen is not picking up the template present in the
secondary storage. Does Xen cache the vhd files locally to avoid coming
to the secondary storage? If so, how can I disable that?
Also, I was looking at UpgradeRouterTemplateCmd API which basically goes
through all the VRs and reboots them. It expects that when the reboot is
completed, the router should have picked up the 4.2.0 version of the
template ( see line 4072 in VirtualNetworkApplianceManagerImpl.java ) I
try to do the reboot manually but the template remains the same. Do you
guys have any more suggestions?
Thanks,
-Syed
On Wed 20 Nov 2013 12:55:04 PM EST, Wei ZHOU wrote:
>
> FYI.
>
> I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working after
> running *cloudstack-sysvmadm to recreate.*
>
>
> 2013/11/20 Syed Ahmed <sa...@cloudops.com>
>
>>
>> +1 Same error. The secondary storage VM and the Console proxy VM seem to
>> be coming up alright. I see this error only when starting the virtual
>> router which is preventing me from creating any instances.
>>
>>
>> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
>>
>>>
>>> I am having the same problem. I got the latest system VMs from:
>>> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
>>> lastSuccessfulBuild/artifact/tools/appliance/dist/
>>>
>>> Are these the wrong System VM Templates? If so, where should I get the
>>> System VM Templates to make this work again?
>>>
>>> Thanks,
>>>
>>> Will
>>>
>>>
>>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
>>> Alena.Prokharchyk@citrix.com> wrote:
>>>
>>> Nitin, I had the same problem, but I fixed it by uploading 4.2 system
>>>>
>>>> templates to my secondary storage. Make sure you have the latest too.
>>>>
>>>> -alena.
>>>>
>>>> From: Nitin Mehta
>>>> <Ni...@citrix.com>
>>>>>
>>>>>
>>>>
>>>> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
>>>> <
>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>>> Date: Thursday, November 7, 2013 4:16 PM
>>>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
>>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>>> Subject: Router requires upgrade. Unable to send command to router
>>>> Error
>>>>
>>>> Unable to deploy vms in the latest master because of the commit
>>>> below. Is
>>>> anyone noticing this on the latest master.
>>>> I checked the code and there was a commit made recently 3f5b8f which is
>>>> where the exception points to.
>>>>
>>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 //
>>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message:: System
>>>> Alert: Low Available Storage in cluster c3 pod p of availability
>>>> zone z3
>>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add
>>>> job-44 into job monitoring
>>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
>>>> Detecting a change in xstoolsversion for r-6-VM
>>>> ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc
>>>> ctx-d9a00f18) Failed to start instance
>>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
>>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
>>>> upgrade.
>>>> Unable to send command to router:6
>>>> at
>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3567)
>>>> at
>>>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute(
>>>> VirtualNetworkApplianceManagerImpl.java:3003)
>>>> at
>>>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(
>>>> VirtualNetworkApplianceManagerImpl.java:3848)
>>>> at
>>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>>> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>> at
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>>> NativeMethodAccessorImpl.java:39)
>>>> at
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>>> DelegatingMethodAccessorImpl.java:25)
>>>>
>>>>
>>>>
>>>
>>>
>>
>>
>>
>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Wei ZHOU <us...@gmail.com>.
FYI.
I upgraded from 2.2.14 to 4.2.1. The CPVM, SSVM and VRs are working after
running *cloudstack-sysvmadm to recreate.*
2013/11/20 Syed Ahmed <sa...@cloudops.com>
> +1 Same error. The secondary storage VM and the Console proxy VM seem to
> be coming up alright. I see this error only when starting the virtual
> router which is preventing me from creating any instances.
>
>
> On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
>
>> I am having the same problem. I got the latest system VMs from:
>> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/
>> lastSuccessfulBuild/artifact/tools/appliance/dist/
>>
>> Are these the wrong System VM Templates? If so, where should I get the
>> System VM Templates to make this work again?
>>
>> Thanks,
>>
>> Will
>>
>>
>> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
>> Alena.Prokharchyk@citrix.com> wrote:
>>
>> Nitin, I had the same problem, but I fixed it by uploading 4.2 system
>>> templates to my secondary storage. Make sure you have the latest too.
>>>
>>> -alena.
>>>
>>> From: Nitin Mehta <Ni...@citrix.com>
>>> >
>>> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>"
>>> <
>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>> Date: Thursday, November 7, 2013 4:16 PM
>>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
>>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>>> Subject: Router requires upgrade. Unable to send command to router Error
>>>
>>> Unable to deploy vms in the latest master because of the commit below. Is
>>> anyone noticing this on the latest master.
>>> I checked the code and there was a commit made recently 3f5b8f which is
>>> where the exception points to.
>>>
>>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 //
>>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message:: System
>>> Alert: Low Available Storage in cluster c3 pod p of availability zone z3
>>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add
>>> job-44 into job monitoring
>>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
>>> Detecting a change in xstoolsversion for r-6-VM
>>> ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc
>>> ctx-d9a00f18) Failed to start instance
>>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
>>> com.cloud.utils.exception.CloudRuntimeException: Router requires
>>> upgrade.
>>> Unable to send command to router:6
>>> at
>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>> Impl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3567)
>>> at
>>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute(
>>> VirtualNetworkApplianceManagerImpl.java:3003)
>>> at
>>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(
>>> VirtualNetworkApplianceManagerImpl.java:3848)
>>> at
>>> com.cloud.network.router.VirtualNetworkApplianceManager
>>> Impl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(
>>> NativeMethodAccessorImpl.java:39)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(
>>> DelegatingMethodAccessorImpl.java:25)
>>>
>>>
>>>
>>
>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Syed Ahmed <sa...@cloudops.com>.
+1 Same error. The secondary storage VM and the Console proxy VM seem
to be coming up alright. I see this error only when starting the
virtual router which is preventing me from creating any instances.
On Wed 20 Nov 2013 11:14:47 AM EST, Will Stevens wrote:
> I am having the same problem. I got the latest system VMs from:
> http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/lastSuccessfulBuild/artifact/tools/appliance/dist/
>
> Are these the wrong System VM Templates? If so, where should I get the
> System VM Templates to make this work again?
>
> Thanks,
>
> Will
>
>
> On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
> Alena.Prokharchyk@citrix.com> wrote:
>
>> Nitin, I had the same problem, but I fixed it by uploading 4.2 system
>> templates to my secondary storage. Make sure you have the latest too.
>>
>> -alena.
>>
>> From: Nitin Mehta <Ni...@citrix.com>>
>> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>> Date: Thursday, November 7, 2013 4:16 PM
>> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
>> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
>> Subject: Router requires upgrade. Unable to send command to router Error
>>
>> Unable to deploy vms in the latest master because of the commit below. Is
>> anyone noticing this on the latest master.
>> I checked the code and there was a commit made recently 3f5b8f which is
>> where the exception points to.
>>
>> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 //
>> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message:: System
>> Alert: Low Available Storage in cluster c3 pod p of availability zone z3
>> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add
>> job-44 into job monitoring
>> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
>> Detecting a change in xstoolsversion for r-6-VM
>> ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc
>> ctx-d9a00f18) Failed to start instance
>> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
>> com.cloud.utils.exception.CloudRuntimeException: Router requires upgrade.
>> Unable to send command to router:6
>> at
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3567)
>> at
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute(VirtualNetworkApplianceManagerImpl.java:3003)
>> at
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3848)
>> at
>> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>
>>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Will Stevens <ws...@cloudops.com>.
I am having the same problem. I got the latest system VMs from:
http://jenkins.buildacloud.org/view/master/job/build-systemvm-master/lastSuccessfulBuild/artifact/tools/appliance/dist/
Are these the wrong System VM Templates? If so, where should I get the
System VM Templates to make this work again?
Thanks,
Will
On Thu, Nov 7, 2013 at 7:42 PM, Alena Prokharchyk <
Alena.Prokharchyk@citrix.com> wrote:
> Nitin, I had the same problem, but I fixed it by uploading 4.2 system
> templates to my secondary storage. Make sure you have the latest too.
>
> -alena.
>
> From: Nitin Mehta <Ni...@citrix.com>>
> Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> Date: Thursday, November 7, 2013 4:16 PM
> To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <
> dev@cloudstack.apache.org<ma...@cloudstack.apache.org>>
> Subject: Router requires upgrade. Unable to send command to router Error
>
> Unable to deploy vms in the latest master because of the commit below. Is
> anyone noticing this on the latest master.
> I checked the code and there was a commit made recently 3f5b8f which is
> where the exception points to.
>
> WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 //
> dataCenterId:: 3 // podId:: 3 // clusterId:: null // message:: System
> Alert: Low Available Storage in cluster c3 pod p of availability zone z3
> INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add
> job-44 into job monitoring
> WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331)
> Detecting a change in xstoolsversion for r-6-VM
> ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc
> ctx-d9a00f18) Failed to start instance
> VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
> com.cloud.utils.exception.CloudRuntimeException: Router requires upgrade.
> Unable to send command to router:6
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3567)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute(VirtualNetworkApplianceManagerImpl.java:3003)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3848)
> at
> com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>
>
Re: Router requires upgrade. Unable to send command to router Error
Posted by Alena Prokharchyk <Al...@citrix.com>.
Nitin, I had the same problem, but I fixed it by uploading 4.2 system templates to my secondary storage. Make sure you have the latest too.
-alena.
From: Nitin Mehta <Ni...@citrix.com>>
Reply-To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Date: Thursday, November 7, 2013 4:16 PM
To: "dev@cloudstack.apache.org<ma...@cloudstack.apache.org>" <de...@cloudstack.apache.org>>
Subject: Router requires upgrade. Unable to send command to router Error
Unable to deploy vms in the latest master because of the commit below. Is anyone noticing this on the latest master.
I checked the code and there was a commit made recently 3f5b8f which is where the exception points to.
WARN [o.a.c.alerts] (CapacityChecker:ctx-4f1ef01f) alertType:: 2 // dataCenterId:: 3 // podId:: 3 // clusterId:: null // message:: System Alert: Low Available Storage in cluster c3 pod p of availability zone z3
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-1:ctx-f118d6dc) Add job-44 into job monitoring
WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-26:ctx-3e786331) Detecting a change in xstoolsversion for r-6-VM
ERROR [c.c.v.VirtualMachineManagerImpl] (Job-Executor-1:ctx-f118d6dc ctx-d9a00f18) Failed to start instance VM[User|VM-4d5d5db2-e5ba-4bbd-b1dc-e749ac42a74c]
com.cloud.utils.exception.CloudRuntimeException: Router requires upgrade. Unable to send command to router:6
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.sendCommandsToRouter(VirtualNetworkApplianceManagerImpl.java:3567)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl$7.execute(VirtualNetworkApplianceManagerImpl.java:3003)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3848)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyDhcpEntry(VirtualNetworkApplianceManagerImpl.java:2995)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)