You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Ian Young <iy...@ratespecial.com> on 2014/05/01 00:00:15 UTC
Re: failed to start virtual router
I notice my dashboard says "Management server node 127.0.0.1 is up." It
used to have an actual address, not localhost. Could this be causing
problems and if so, how can I set it back?
On Wed, Apr 30, 2014 at 12:40 PM, Ian Young <iy...@ratespecial.com> wrote:
> Yes, I replaced the new files with the rpmsave ones, which allowed the
> agent to start. However, most of the functions in the management console
> fail.
>
>
> On Wed, Apr 30, 2014 at 12:34 PM, stevenliang <st...@yesup.com>wrote:
>
>> Do you have the file db.properties.rpmsave on management server and
>> agent.properties.rpmsave on agents? If so, and the date is correct, you can
>> use it rather than db.properties and agent.properties.
>> And then restart management and agent services.
>>
>>
>> On 30/04/14 03:26 PM, Ian Young wrote:
>>
>>> Yes, I restored the DB from the backup. When I try to start the router
>>> it
>>> says:
>>>
>>> Resource [Host:1] is unreachable: Host 1: Unable to start instance due to
>>> Unable to start VM[DomainRouter|r-63-VM] due to error in finalizeStart,
>>> not
>>> retrying
>>>
>>> The management server log says:
>>>
>>> 2014-04-30 12:20:52,485 ERROR [cloud.async.AsyncJobManagerImpl]
>>> (Job-Executor-7:job-520 = [ 9d7c898c-b5d0-4bd0-a711-563a91d7acc9 ])
>>> Unexpected exception while executing
>>> org.apache.cloudstack.api.command.admin.router.StartRouterCmd
>>> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>> unreachable: Host 1: Unable to start instance due to Unable to start
>>> VM[DomainRouter|r-63-VM] due to error in finalizeStart, not retrying
>>>
>>>
>>> On Wed, Apr 30, 2014 at 12:02 PM, stevenliang <st...@yesup.com>
>>> wrote:
>>>
>>> I think you had backed up database, when you upgraded.
>>>> When you downgraded CS, you also need to restore DB.
>>>>
>>>>
>>>> On 30/04/14 02:58 PM, Ian Young wrote:
>>>>
>>>> I think my problem stems from a partially downloaded system VM
>>>>> template.
>>>>> I
>>>>> just noticed systemvm-kvm-4.3 is stuck at 27% downloaded. It must have
>>>>> been interrupted during the upgrade to 4.3. At the moment I've rolled
>>>>> back
>>>>> to 4.2.1 with a somewhat usable management interface, although the
>>>>> system
>>>>> VMs won't start. I suspect there is something in the database that is
>>>>> causing it to try to use the 4.3 template. How can I delete the
>>>>> template
>>>>> and make sure the management server is using the older one?
>>>>>
>>>>>
>>>>> On Tue, Apr 29, 2014 at 8:23 PM, Ian Young <iy...@ratespecial.com>
>>>>> wrote:
>>>>>
>>>>> Ok, so I've figured out a way to identify volumes in the filesystem.
>>>>> For
>>>>>
>>>>>> instance, /var/storage/primary/4d324e1a-e3a6-4da8-9c4d-44ad723482ad
>>>>>> is
>>>>>> the
>>>>>> root volume for an instance I want to back up. Is this in qcow2
>>>>>> format
>>>>>> or
>>>>>> something else? I'm using KVM.
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 29, 2014 at 7:38 PM, Ian Young <iy...@ratespecial.com>
>>>>>> wrote:
>>>>>>
>>>>>> Now I can't start cloudstack-agent. The agent.log says:
>>>>>>
>>>>>>> Unable to start agent: Failed to get private nic name
>>>>>>>
>>>>>>> I know this is because the network bridge is no longer set up
>>>>>>> correctly.
>>>>>>> I used to have a cloud0 and a cloudbr0 interface. Now I only have
>>>>>>> cloudbr0. I haven't changed my network configuration. Somehow it's
>>>>>>> been
>>>>>>> changed by CloudStack during the upgrade/downgrade. This is getting
>>>>>>> worse
>>>>>>> and worse the more I try to recover my data. Is there any way to
>>>>>>> back
>>>>>>> up
>>>>>>> the instances' volumes via the command line? I can't tell which is
>>>>>>> which
>>>>>>> because the filenames are all hashes. I really need to get these
>>>>>>> instances
>>>>>>> up and running--there are several months worth of work at stake here.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 29, 2014 at 6:13 PM, ma y <br...@gmail.com> wrote:
>>>>>>>
>>>>>>> I got the same problem, and how to downgrade CS 4.3.0 to 4.2.1
>>>>>>> safely?
>>>>>>>
>>>>>>>>
>>>>>>>> 2014-04-30 8:45 GMT+08:00 Ian Young <iy...@ratespecial.com>:
>>>>>>>>
>>>>>>>> Ok, my Cloudstack installation is now so broken that I think it's
>>>>>>>> probably
>>>>>>>>
>>>>>>>> best to backup all my instances and templates, wipe the databases,
>>>>>>>>> and
>>>>>>>>> start from scratch. However, I can't take snapshots or download
>>>>>>>>>
>>>>>>>>> volumes
>>>>>>>>
>>>>>>>> anymore. What's causing these errors?
>>>>>>>>>
>>>>>>>>> 2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m.
>>>>>>>>> AncientDataMotionStrategy]
>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed:
>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send
>>>>>>>>>
>>>>>>>>> command,
>>>>>>>>
>>>>>>>> due to Agent:1, com.cloud.exception.OperationTimedoutException:
>>>>>>>>>
>>>>>>>>> Commands
>>>>>>>>
>>>>>>>> 841744457 to Host 1 timed out after 21600
>>>>>>>>> 2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m.
>>>>>>>>> AncientDataMotionStrategy]
>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed
>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send
>>>>>>>>>
>>>>>>>>> command,
>>>>>>>>
>>>>>>>> due to Agent:1, com.cloud.exception.OperationTimedoutException:
>>>>>>>>>
>>>>>>>>> Commands
>>>>>>>>
>>>>>>>> 841744457 to Host 1 timed out after 21600
>>>>>>>>> 2014-04-29 17:40:51,269 WARN [o.a.c.s.d.
>>>>>>>>> ObjectInDataStoreManagerImpl]
>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data object
>>>>>>>>> (VOLUME,
>>>>>>>>> org.apache.cloudstack.storage.datastore.
>>>>>>>>> PrimaryDataStoreImpl@7bbbd901
>>>>>>>>> ),
>>>>>>>>>
>>>>>>>>> no
>>>>>>>>
>>>>>>>> need to delete from object in store ref table
>>>>>>>>> 2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Unexpected exception while executing
>>>>>>>>> org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd
>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to copy
>>>>>>>>> the
>>>>>>>>>
>>>>>>>>> volume
>>>>>>>>
>>>>>>>> from the source primary storage pool to secondary storage.
>>>>>>>>> 2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Complete async job-501, jobStatus:
>>>>>>>>>
>>>>>>>>> FAILED,
>>>>>>>>
>>>>>>>> resultCode: 530, result:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/
>>>>>>>>>
>>>>>>>> null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>>>>>>>>
>>>>>>>> to copy the volume from the source primary storage pool to
>>>>>>>>> secondary
>>>>>>>>> storage."}
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Apr 29, 2014 at 4:15 PM, Ian Young <iyoung@ratespecial.com
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I downgraded to 4.2.1 again but cloudstack-management won't start
>>>>>>>>> because
>>>>>>>>> the database is version 4.3. Is it safe to restore the database
>>>>>>>>> backup I
>>>>>>>>> made prior to this whole process? In the meantime I have destroyed
>>>>>>>>> and
>>>>>>>>> created system VMs, so I'm not sure it's a good idea.
>>>>>>>>>
>>>>>>>>>> On Apr 29, 2014 3:09 PM, "Ian Young" <iy...@ratespecial.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> @stevenliang: I take it back--you can't set the VM size when you
>>>>>>>>>> register
>>>>>>>>>> the template.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz <
>>>>>>>>>>> motty.cruz@gmail.com>
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>> yes, you would have to shutdown the router, then click on "Change
>>>>>>>>>>
>>>>>>>>>>> Service
>>>>>>>>>>>
>>>>>>>>>> Offering"
>>>>>>>>>>
>>>>>>>>>>> restart the VR.
>>>>>>>>>>>>
>>>>>>>>>>>> To Ian,
>>>>>>>>>>>>
>>>>>>>>>>>> I suspect you forgot the last step: "
>>>>>>>>>>>> cloudstack-setup-management"
>>>>>>>>>>>>
>>>>>>>>>>>> that would fix your issue, I think,
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> ---
>>>>>>>>>>>> I downgraded to 4.2.1 and then upgraded to 4.3. Now the
>>>>>>>>>>>> cloudstack-management service can't start because it can't
>>>>>>>>>>>> connect
>>>>>>>>>>>>
>>>>>>>>>>>> to
>>>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>
>>>>>>>>> database.
>>>>>>>>>>
>>>>>>>>>>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2] (main:null)
>>>>>>>>>>>>
>>>>>>>>>>>> Unable
>>>>>>>>>>>
>>>>>>>>>> to
>>>>>>>>>
>>>>>>>>>> get a new db connection
>>>>>>>>>>>> Caused by: java.sql.SQLException: Access denied for user
>>>>>>>>>>>> 'cloud'@
>>>>>>>>>>>> 'localhost'
>>>>>>>>>>>> (using password: YES)
>>>>>>>>>>>>
>>>>>>>>>>>> Where are the credentials stored?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang <
>>>>>>>>>>>>
>>>>>>>>>>>> stevenliang@yesup.com>
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> oh, then change service offering for vr?
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 29/04/14 05:53 PM, motty cruz wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> for my VR, I created a new
>>>>>>>>>>>>>
>>>>>>>>>>>>>> "System Offering For Software Router"
>>>>>>>>>>>>>> CPU in (MHz) 1.00GHz
>>>>>>>>>>>>>> Memory (in MB) 1.00GB
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> this are my current offerings, I'm sure the more RAM and CPU
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> better
>>>>>>>>>>>>>
>>>>>>>>>>>> performance.
>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang <
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thank you again, motty.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I didn't notice this earlier.
>>>>>>>>>>>>>>> BTW, how did you make your vr had 1GB CPU and 512MB RAM?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 29/04/14 05:33 PM, motty cruz wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Stevellang,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I not sure if you saw this in the forums earlier :
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> http://mail-archives.apache.
>>>>>>>>>>>>>>>> org/mod_mbox/cloudstack-users/
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 201404.mbox/%
>>>>>>>>
>>>>>>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc=
>>>>>>>>>
>>>>>>>>>> c2rfEDRoY9mA@mail.gmail.com%3E
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I don't know if the bug was fixed yet,
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I will try upgrade in the next couple of days on a testing
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> will
>>>>>>>>>>
>>>>>>>>>>> report back if the bug was fixed.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thank you, motty.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I am also running kvm. Since that time I failed upgrade,
>>>>>>>>>>>>>>>> I am
>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> using
>>>>>>>>>>
>>>>>>>>>>> 4.2.1. I'll try as your advice.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 29/04/14 05:19 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Stevenllang,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I had the similar issue with VR, I notice it was because
>>>>>>>>>>>>>>>>> I
>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> default system specs on the VR, for instance by default
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 500MHz
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> on
>>>>>>>>>
>>>>>>>>> CPU
>>>>>>>>>>
>>>>>>>>>>> and
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 128MB on RAM, if you upgrade to at least 1GB on CPU and
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 512MB of
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> RAM
>>>>>>>>>
>>>>>>>>>> your
>>>>>>>>>>>>>
>>>>>>>>>>>>>> VR will survive the upgrade from 4.2.1 to 4.3.1.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am running KVM, when I upgrade from 4.2.1 to 4.3 my VMs
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> were
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> not
>>>>>>>>>
>>>>>>>>> able
>>>>>>>>>>
>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>> access outside world, even if I created a new router.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wish you the best,
>>>>>>>>>>>>>>>>>> -motty
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> stevenliang@yesup.com>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Yes, I had two zones(one is basic, another is advanced
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> mode).
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> After I upgraded from 4.2.1 to 4.3, the vrouter lost.
>>>>>>>>>
>>>>>>>>>> So I rolled back to 4.2.1, the vrouter came back.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 29/04/14 04:54 PM, Ian Young wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Did rolling back to 4.2 fix the problem?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 1:22 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I met your situation before. Finally I rolled back to
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 4.2
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 29/04/14 04:18 PM, Ian Young wrote:
>>>>>>>>>
>>>>>>>>>> I destroyed the old virtual router and was able to
>>>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>
>>>>>>>>>> one
>>>>>>>>>>>>>
>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> adding a new instance. However, this new router
>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> failed
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>> start,
>>>>>>>>>>>>>
>>>>>>>>>>>>>> citing the same error. After that, the expungement
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> delay
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> elapsed
>>>>>>>>>
>>>>>>>>>> and
>>>>>>>>>>>>>
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>> virtual router was expunged, so now I have none.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2014 at 8:52 PM, Ian Young <
>>>>>>>>>>>>>>>>>>>>>> iyoung@ratespecial.com>
>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I upgraded from 4.2.1 to 4.3.0 tonight,
>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> instructions
>>>>>>>>>
>>>>>>>>>> here:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> http://docs.cloudstack.apache.
>>>>>>>>>>>>>>>>>>>>>> org/projects/cloudstack-
>>>>>>>>>>>>>>>>>>>>>> release-notes/en/latest/
>>>>>>>>>>>>>>>>>>>>>> rnotes.html#upgrade-from-4-2-
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> x-to-4-3
>>>>>>>>>
>>>>>>>>> At the last step, I tried to restart the system VMs.
>>>>>>>>>>
>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> virtual
>>>>>>>>>
>>>>>>>>>> router
>>>>>>>>>>>>>
>>>>>>>>>>>>>> failed to start. Here is the message that was
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> displayed in
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>
>>>>>>>>>> web
>>>>>>>>>>>>>
>>>>>>>>>>>>>> UI:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable to
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>
>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> Unable to start VM[DomainRouter|r-4-VM] due to error
>>>>>>>>>>>>>>>>>>>>>>> in
>>>>>>>>>>>>>>>>>>>>>>> finalizeStart,
>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>> retrying
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I tried running the script to restart the VMs but
>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>
>>>>>>>>>> failed
>>>>>>>>>>>>>
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>> start the console proxy:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> [root@virthost1 ~]$ cloudstack-sysvmadm -d
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 192.168.100.6
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> -u
>>>>>>>>>
>>>>>>>>> cloud
>>>>>>>>>>
>>>>>>>>>>> -p
>>>>>>>>>>>>>>>>>>>>>>> -a
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 secondary storage vm(s)...
>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting secondary storage vm(s)
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 console proxy vm(s)...
>>>>>>>>>>>>>>>>>>>>>>> ERROR: Failed to start console proxy vm with id 2
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting console proxy vm(s) .
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 0 running routing vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Is there a way to wipe the system VMs out and start
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> over?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>
>
Re: failed to start virtual router
Posted by Ian Young <iy...@ratespecial.com>.
I downgraded to 4.2.1, restored the database backup I made during the
initial upgrade to 4.3, copied the rpmsave files over
/etc/cloudstack/agent/agent.properties and
/etc/cloudstack/managment/db.properties, and started the agent and
management services. The service_ip field in the mshost table changed to
127.0.0.1 again, so I changed it to the actual IP address of the server.
Then I ran
cloud-install-sys-tmplt -m /var/storage/secondary -u
http://download.cloud.com/templates/4.2/systemvmtemplate-2013-06-12-master-kvm.qcow2.bz2-h
kvm -F
which said it successfully installed the system VM template to
/var/storage/secondary/template/tmpl/1/3/. Next, I restarted the
management service. When I logged in, the management console said there
was an SSVM, console proxy, and router present but they were all stopped.
I tried starting the SSVM but it failed to start. Here is an excerpt from
management.log:
http://pastebin.com/uW4NPZbC
Is there some sort of general troubleshooting script that could identify
major configuration problems? I feel like this system is mostly functional
but one or two misconfigured things are causing it to break. I'd hate to
have to wipe it clean and start over, losing all my VMs in the process.
On Wed, Apr 30, 2014 at 6:00 PM, Ian Young <iy...@ratespecial.com> wrote:
> I read this article about upgrading the system VMs:
>
>
> https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+4.2+(KVM)+System+Vm+Upgrade
>
> However, there's just an empty set in the template_host_ref table. Is
> this no longer used in 4.3?
>
>
> On Wed, Apr 30, 2014 at 5:41 PM, Ian Young <iy...@ratespecial.com> wrote:
>
>> I've tried upgrading to 4.3 again. After poking around some more in the
>> database, I've discovered that the KVM system VM template was only 27%
>> downloaded. I think this is why the virtual router was unable to
>> start--the template was incomplete. Is there a way to force it to resume
>> downloading?
>>
>>
>> On Wed, Apr 30, 2014 at 3:03 PM, Ian Young <iy...@ratespecial.com>wrote:
>>
>>> The address in Infrastructure > Hosts > (management server) is set to
>>> the correct IP address, not 127.0.0.1. Why are the logs referring to
>>> 127.0.0.1?
>>>
>>>
>>> On Wed, Apr 30, 2014 at 3:00 PM, Ian Young <iy...@ratespecial.com>wrote:
>>>
>>>> I notice my dashboard says "Management server node 127.0.0.1 is up."
>>>> It used to have an actual address, not localhost. Could this be causing
>>>> problems and if so, how can I set it back?
>>>>
>>>>
>>>> On Wed, Apr 30, 2014 at 12:40 PM, Ian Young <iy...@ratespecial.com>wrote:
>>>>
>>>>> Yes, I replaced the new files with the rpmsave ones, which allowed the
>>>>> agent to start. However, most of the functions in the management console
>>>>> fail.
>>>>>
>>>>>
>>>>> On Wed, Apr 30, 2014 at 12:34 PM, stevenliang <st...@yesup.com>wrote:
>>>>>
>>>>>> Do you have the file db.properties.rpmsave on management server and
>>>>>> agent.properties.rpmsave on agents? If so, and the date is correct, you can
>>>>>> use it rather than db.properties and agent.properties.
>>>>>> And then restart management and agent services.
>>>>>>
>>>>>>
>>>>>> On 30/04/14 03:26 PM, Ian Young wrote:
>>>>>>
>>>>>>> Yes, I restored the DB from the backup. When I try to start the
>>>>>>> router it
>>>>>>> says:
>>>>>>>
>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable to start instance
>>>>>>> due to
>>>>>>> Unable to start VM[DomainRouter|r-63-VM] due to error in
>>>>>>> finalizeStart, not
>>>>>>> retrying
>>>>>>>
>>>>>>> The management server log says:
>>>>>>>
>>>>>>> 2014-04-30 12:20:52,485 ERROR [cloud.async.AsyncJobManagerImpl]
>>>>>>> (Job-Executor-7:job-520 = [ 9d7c898c-b5d0-4bd0-a711-563a91d7acc9 ])
>>>>>>> Unexpected exception while executing
>>>>>>> org.apache.cloudstack.api.command.admin.router.StartRouterCmd
>>>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>>>>>> unreachable: Host 1: Unable to start instance due to Unable to start
>>>>>>> VM[DomainRouter|r-63-VM] due to error in finalizeStart, not retrying
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Apr 30, 2014 at 12:02 PM, stevenliang <st...@yesup.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> I think you had backed up database, when you upgraded.
>>>>>>>> When you downgraded CS, you also need to restore DB.
>>>>>>>>
>>>>>>>>
>>>>>>>> On 30/04/14 02:58 PM, Ian Young wrote:
>>>>>>>>
>>>>>>>> I think my problem stems from a partially downloaded system VM
>>>>>>>>> template.
>>>>>>>>> I
>>>>>>>>> just noticed systemvm-kvm-4.3 is stuck at 27% downloaded. It must
>>>>>>>>> have
>>>>>>>>> been interrupted during the upgrade to 4.3. At the moment I've
>>>>>>>>> rolled
>>>>>>>>> back
>>>>>>>>> to 4.2.1 with a somewhat usable management interface, although the
>>>>>>>>> system
>>>>>>>>> VMs won't start. I suspect there is something in the database
>>>>>>>>> that is
>>>>>>>>> causing it to try to use the 4.3 template. How can I delete the
>>>>>>>>> template
>>>>>>>>> and make sure the management server is using the older one?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Apr 29, 2014 at 8:23 PM, Ian Young <iyoung@ratespecial.com
>>>>>>>>> >
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Ok, so I've figured out a way to identify volumes in the
>>>>>>>>> filesystem. For
>>>>>>>>>
>>>>>>>>>> instance, /var/storage/primary/4d324e1a-e3a6-4da8-9c4d-44ad723482ad
>>>>>>>>>> is
>>>>>>>>>> the
>>>>>>>>>> root volume for an instance I want to back up. Is this in qcow2
>>>>>>>>>> format
>>>>>>>>>> or
>>>>>>>>>> something else? I'm using KVM.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 29, 2014 at 7:38 PM, Ian Young <
>>>>>>>>>> iyoung@ratespecial.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> Now I can't start cloudstack-agent. The agent.log says:
>>>>>>>>>>
>>>>>>>>>>> Unable to start agent: Failed to get private nic name
>>>>>>>>>>>
>>>>>>>>>>> I know this is because the network bridge is no longer set up
>>>>>>>>>>> correctly.
>>>>>>>>>>> I used to have a cloud0 and a cloudbr0 interface. Now I only
>>>>>>>>>>> have
>>>>>>>>>>> cloudbr0. I haven't changed my network configuration. Somehow
>>>>>>>>>>> it's
>>>>>>>>>>> been
>>>>>>>>>>> changed by CloudStack during the upgrade/downgrade. This is
>>>>>>>>>>> getting
>>>>>>>>>>> worse
>>>>>>>>>>> and worse the more I try to recover my data. Is there any way
>>>>>>>>>>> to back
>>>>>>>>>>> up
>>>>>>>>>>> the instances' volumes via the command line? I can't tell which
>>>>>>>>>>> is
>>>>>>>>>>> which
>>>>>>>>>>> because the filenames are all hashes. I really need to get these
>>>>>>>>>>> instances
>>>>>>>>>>> up and running--there are several months worth of work at stake
>>>>>>>>>>> here.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 29, 2014 at 6:13 PM, ma y <br...@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I got the same problem, and how to downgrade CS 4.3.0 to 4.2.1
>>>>>>>>>>> safely?
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> 2014-04-30 8:45 GMT+08:00 Ian Young <iy...@ratespecial.com>:
>>>>>>>>>>>>
>>>>>>>>>>>> Ok, my Cloudstack installation is now so broken that I think
>>>>>>>>>>>> it's
>>>>>>>>>>>> probably
>>>>>>>>>>>>
>>>>>>>>>>>> best to backup all my instances and templates, wipe the
>>>>>>>>>>>>> databases, and
>>>>>>>>>>>>> start from scratch. However, I can't take snapshots or
>>>>>>>>>>>>> download
>>>>>>>>>>>>>
>>>>>>>>>>>>> volumes
>>>>>>>>>>>>
>>>>>>>>>>>> anymore. What's causing these errors?
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m.
>>>>>>>>>>>>> AncientDataMotionStrategy]
>>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed:
>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to
>>>>>>>>>>>>> send
>>>>>>>>>>>>>
>>>>>>>>>>>>> command,
>>>>>>>>>>>>
>>>>>>>>>>>> due to Agent:1, com.cloud.exception.
>>>>>>>>>>>>> OperationTimedoutException:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Commands
>>>>>>>>>>>>
>>>>>>>>>>>> 841744457 to Host 1 timed out after 21600
>>>>>>>>>>>>> 2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m.
>>>>>>>>>>>>> AncientDataMotionStrategy]
>>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed
>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to
>>>>>>>>>>>>> send
>>>>>>>>>>>>>
>>>>>>>>>>>>> command,
>>>>>>>>>>>>
>>>>>>>>>>>> due to Agent:1, com.cloud.exception.
>>>>>>>>>>>>> OperationTimedoutException:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Commands
>>>>>>>>>>>>
>>>>>>>>>>>> 841744457 to Host 1 timed out after 21600
>>>>>>>>>>>>> 2014-04-29 17:40:51,269 WARN [o.a.c.s.d.
>>>>>>>>>>>>> ObjectInDataStoreManagerImpl]
>>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data
>>>>>>>>>>>>> object
>>>>>>>>>>>>> (VOLUME,
>>>>>>>>>>>>> org.apache.cloudstack.storage.datastore.
>>>>>>>>>>>>> PrimaryDataStoreImpl@7bbbd901
>>>>>>>>>>>>> ),
>>>>>>>>>>>>>
>>>>>>>>>>>>> no
>>>>>>>>>>>>
>>>>>>>>>>>> need to delete from object in store ref table
>>>>>>>>>>>>> 2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Unexpected exception while
>>>>>>>>>>>>> executing
>>>>>>>>>>>>> org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd
>>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to
>>>>>>>>>>>>> copy the
>>>>>>>>>>>>>
>>>>>>>>>>>>> volume
>>>>>>>>>>>>
>>>>>>>>>>>> from the source primary storage pool to secondary storage.
>>>>>>>>>>>>> 2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i.
>>>>>>>>>>>>> AsyncJobManagerImpl]
>>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Complete async job-501,
>>>>>>>>>>>>> jobStatus:
>>>>>>>>>>>>>
>>>>>>>>>>>>> FAILED,
>>>>>>>>>>>>
>>>>>>>>>>>> resultCode: 530, result:
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/
>>>>>>>>>>>>>
>>>>>>>>>>>> null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>>>>>>>>>>>>
>>>>>>>>>>>> to copy the volume from the source primary storage pool to
>>>>>>>>>>>>> secondary
>>>>>>>>>>>>> storage."}
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 4:15 PM, Ian Young <
>>>>>>>>>>>>> iyoung@ratespecial.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> I downgraded to 4.2.1 again but cloudstack-management won't
>>>>>>>>>>>>> start
>>>>>>>>>>>>> because
>>>>>>>>>>>>> the database is version 4.3. Is it safe to restore the database
>>>>>>>>>>>>> backup I
>>>>>>>>>>>>> made prior to this whole process? In the meantime I have
>>>>>>>>>>>>> destroyed
>>>>>>>>>>>>> and
>>>>>>>>>>>>> created system VMs, so I'm not sure it's a good idea.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Apr 29, 2014 3:09 PM, "Ian Young" <iy...@ratespecial.com>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> @stevenliang: I take it back--you can't set the VM size
>>>>>>>>>>>>>> when you
>>>>>>>>>>>>>> register
>>>>>>>>>>>>>> the template.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz <
>>>>>>>>>>>>>>> motty.cruz@gmail.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>> yes, you would have to shutdown the router, then click on
>>>>>>>>>>>>>> "Change
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Service
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Offering"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> restart the VR.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> To Ian,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I suspect you forgot the last step: "
>>>>>>>>>>>>>>>> cloudstack-setup-management"
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> that would fix your issue, I think,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>> I downgraded to 4.2.1 and then upgraded to 4.3. Now the
>>>>>>>>>>>>>>>> cloudstack-management service can't start because it can't
>>>>>>>>>>>>>>>> connect
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> the
>>>>>>>>>>>>>
>>>>>>>>>>>>> database.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2]
>>>>>>>>>>>>>>>> (main:null)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Unable
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>
>>>>>>>>>>>>>> get a new db connection
>>>>>>>>>>>>>>>> Caused by: java.sql.SQLException: Access denied for user
>>>>>>>>>>>>>>>> 'cloud'@
>>>>>>>>>>>>>>>> 'localhost'
>>>>>>>>>>>>>>>> (using password: YES)
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Where are the credentials stored?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> stevenliang@yesup.com>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> oh, then change service offering for vr?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 29/04/14 05:53 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> for my VR, I created a new
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> "System Offering For Software Router"
>>>>>>>>>>>>>>>>>> CPU in (MHz) 1.00GHz
>>>>>>>>>>>>>>>>>> Memory (in MB) 1.00GB
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> this are my current offerings, I'm sure the more RAM and
>>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you again, motty.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I didn't notice this earlier.
>>>>>>>>>>>>>>>>>>> BTW, how did you make your vr had 1GB CPU and 512MB RAM?
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 29/04/14 05:33 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Stevellang,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I not sure if you saw this in the forums earlier :
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> http://mail-archives.apache.
>>>>>>>>>>>>>>>>>>>> org/mod_mbox/cloudstack-users/
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 201404.mbox/%
>>>>>>>>>>>>
>>>>>>>>>>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc=
>>>>>>>>>>>>>
>>>>>>>>>>>>>> c2rfEDRoY9mA@mail.gmail.com%3E
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I don't know if the bug was fixed yet,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I will try upgrade in the next couple of days on a testing
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> report back if the bug was fixed.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thank you, motty.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I am also running kvm. Since that time I failed
>>>>>>>>>>>>>>>>>>>> upgrade, I am
>>>>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 4.2.1. I'll try as your advice.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 29/04/14 05:19 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Stevenllang,
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I had the similar issue with VR, I notice it was
>>>>>>>>>>>>>>>>>>>>> because I
>>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>
>>>>>>>>>>>>>> default system specs on the VR, for instance by default
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 500MHz
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>>
>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 128MB on RAM, if you upgrade to at least 1GB on CPU and
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 512MB of
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> RAM
>>>>>>>>>>>>>
>>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> VR will survive the upgrade from 4.2.1 to 4.3.1.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> I am running KVM, when I upgrade from 4.2.1 to 4.3 my
>>>>>>>>>>>>>>>>>>>>>> VMs
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> were
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>
>>>>>>>>>>>>> able
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>> access outside world, even if I created a new router.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> wish you the best,
>>>>>>>>>>>>>>>>>>>>>> -motty
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Yes, I had two zones(one is basic, another is
>>>>>>>>>>>>>>>>>>>>>> advanced
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> mode).
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> After I upgraded from 4.2.1 to 4.3, the vrouter lost.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> So I rolled back to 4.2.1, the vrouter came back.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On 29/04/14 04:54 PM, Ian Young wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> Did rolling back to 4.2 fix the problem?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 1:22 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I met your situation before. Finally I rolled
>>>>>>>>>>>>>>>>>>>>>>>> back to
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 4.2
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 29/04/14 04:18 PM, Ian Young wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> I destroyed the old virtual router and was able to
>>>>>>>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>>
>>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> adding a new instance. However, this new
>>>>>>>>>>>>>>>>>>>>>>>>> router also
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> failed
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> start,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> citing the same error. After that, the expungement
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> delay
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> elapsed
>>>>>>>>>>>>>
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>> virtual router was expunged, so now I have none.
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2014 at 8:52 PM, Ian Young <
>>>>>>>>>>>>>>>>>>>>>>>>>> iyoung@ratespecial.com>
>>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I upgraded from 4.2.1 to 4.3.0 tonight,
>>>>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> instructions
>>>>>>>>>>>>>
>>>>>>>>>>>>>> here:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> http://docs.cloudstack.apache.
>>>>>>>>>>>>>>>>>>>>>>>>>> org/projects/cloudstack-
>>>>>>>>>>>>>>>>>>>>>>>>>> release-notes/en/latest/
>>>>>>>>>>>>>>>>>>>>>>>>>> rnotes.html#upgrade-from-4-2-
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> x-to-4-3
>>>>>>>>>>>>>
>>>>>>>>>>>>> At the last step, I tried to restart the system VMs.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> virtual
>>>>>>>>>>>>>
>>>>>>>>>>>>>> router
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> failed to start. Here is the message that was
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> displayed in
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>
>>>>>>>>>>>>>> web
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> UI:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable
>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>>>
>>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> Unable to start VM[DomainRouter|r-4-VM] due to
>>>>>>>>>>>>>>>>>>>>>>>>>>> error in
>>>>>>>>>>>>>>>>>>>>>>>>>>> finalizeStart,
>>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>>> retrying
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> I tried running the script to restart the VMs
>>>>>>>>>>>>>>>>>>>>>>>>>>> but this
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>>
>>>>>>>>>>>>>> failed
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>> start the console proxy:
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> [root@virthost1 ~]$ cloudstack-sysvmadm -d
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> 192.168.100.6
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> -u
>>>>>>>>>>>>>
>>>>>>>>>>>>> cloud
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> -p
>>>>>>>>>>>>>>>>>>>>>>>>>>> -a
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 secondary storage
>>>>>>>>>>>>>>>>>>>>>>>>>>> vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting secondary storage
>>>>>>>>>>>>>>>>>>>>>>>>>>> vm(s)
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 console proxy vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: Failed to start console proxy vm with id 2
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting console proxy vm(s) .
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 0 running routing vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> Is there a way to wipe the system VMs out and
>>>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>> over?
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: failed to start virtual router
Posted by Ian Young <iy...@ratespecial.com>.
I read this article about upgrading the system VMs:
https://cwiki.apache.org/confluence/display/CLOUDSTACK/CloudStack+4.2+(KVM)+System+Vm+Upgrade
However, there's just an empty set in the template_host_ref table. Is this
no longer used in 4.3?
On Wed, Apr 30, 2014 at 5:41 PM, Ian Young <iy...@ratespecial.com> wrote:
> I've tried upgrading to 4.3 again. After poking around some more in the
> database, I've discovered that the KVM system VM template was only 27%
> downloaded. I think this is why the virtual router was unable to
> start--the template was incomplete. Is there a way to force it to resume
> downloading?
>
>
> On Wed, Apr 30, 2014 at 3:03 PM, Ian Young <iy...@ratespecial.com> wrote:
>
>> The address in Infrastructure > Hosts > (management server) is set to the
>> correct IP address, not 127.0.0.1. Why are the logs referring to 127.0.0.1?
>>
>>
>> On Wed, Apr 30, 2014 at 3:00 PM, Ian Young <iy...@ratespecial.com>wrote:
>>
>>> I notice my dashboard says "Management server node 127.0.0.1 is up." It
>>> used to have an actual address, not localhost. Could this be causing
>>> problems and if so, how can I set it back?
>>>
>>>
>>> On Wed, Apr 30, 2014 at 12:40 PM, Ian Young <iy...@ratespecial.com>wrote:
>>>
>>>> Yes, I replaced the new files with the rpmsave ones, which allowed the
>>>> agent to start. However, most of the functions in the management console
>>>> fail.
>>>>
>>>>
>>>> On Wed, Apr 30, 2014 at 12:34 PM, stevenliang <st...@yesup.com>wrote:
>>>>
>>>>> Do you have the file db.properties.rpmsave on management server and
>>>>> agent.properties.rpmsave on agents? If so, and the date is correct, you can
>>>>> use it rather than db.properties and agent.properties.
>>>>> And then restart management and agent services.
>>>>>
>>>>>
>>>>> On 30/04/14 03:26 PM, Ian Young wrote:
>>>>>
>>>>>> Yes, I restored the DB from the backup. When I try to start the
>>>>>> router it
>>>>>> says:
>>>>>>
>>>>>> Resource [Host:1] is unreachable: Host 1: Unable to start instance
>>>>>> due to
>>>>>> Unable to start VM[DomainRouter|r-63-VM] due to error in
>>>>>> finalizeStart, not
>>>>>> retrying
>>>>>>
>>>>>> The management server log says:
>>>>>>
>>>>>> 2014-04-30 12:20:52,485 ERROR [cloud.async.AsyncJobManagerImpl]
>>>>>> (Job-Executor-7:job-520 = [ 9d7c898c-b5d0-4bd0-a711-563a91d7acc9 ])
>>>>>> Unexpected exception while executing
>>>>>> org.apache.cloudstack.api.command.admin.router.StartRouterCmd
>>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>>>>> unreachable: Host 1: Unable to start instance due to Unable to start
>>>>>> VM[DomainRouter|r-63-VM] due to error in finalizeStart, not retrying
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 30, 2014 at 12:02 PM, stevenliang <st...@yesup.com>
>>>>>> wrote:
>>>>>>
>>>>>> I think you had backed up database, when you upgraded.
>>>>>>> When you downgraded CS, you also need to restore DB.
>>>>>>>
>>>>>>>
>>>>>>> On 30/04/14 02:58 PM, Ian Young wrote:
>>>>>>>
>>>>>>> I think my problem stems from a partially downloaded system VM
>>>>>>>> template.
>>>>>>>> I
>>>>>>>> just noticed systemvm-kvm-4.3 is stuck at 27% downloaded. It must
>>>>>>>> have
>>>>>>>> been interrupted during the upgrade to 4.3. At the moment I've
>>>>>>>> rolled
>>>>>>>> back
>>>>>>>> to 4.2.1 with a somewhat usable management interface, although the
>>>>>>>> system
>>>>>>>> VMs won't start. I suspect there is something in the database that
>>>>>>>> is
>>>>>>>> causing it to try to use the 4.3 template. How can I delete the
>>>>>>>> template
>>>>>>>> and make sure the management server is using the older one?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 29, 2014 at 8:23 PM, Ian Young <iy...@ratespecial.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Ok, so I've figured out a way to identify volumes in the
>>>>>>>> filesystem. For
>>>>>>>>
>>>>>>>>> instance, /var/storage/primary/4d324e1a-e3a6-4da8-9c4d-44ad723482ad
>>>>>>>>> is
>>>>>>>>> the
>>>>>>>>> root volume for an instance I want to back up. Is this in qcow2
>>>>>>>>> format
>>>>>>>>> or
>>>>>>>>> something else? I'm using KVM.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Apr 29, 2014 at 7:38 PM, Ian Young <iyoung@ratespecial.com
>>>>>>>>> >
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Now I can't start cloudstack-agent. The agent.log says:
>>>>>>>>>
>>>>>>>>>> Unable to start agent: Failed to get private nic name
>>>>>>>>>>
>>>>>>>>>> I know this is because the network bridge is no longer set up
>>>>>>>>>> correctly.
>>>>>>>>>> I used to have a cloud0 and a cloudbr0 interface. Now I only
>>>>>>>>>> have
>>>>>>>>>> cloudbr0. I haven't changed my network configuration. Somehow
>>>>>>>>>> it's
>>>>>>>>>> been
>>>>>>>>>> changed by CloudStack during the upgrade/downgrade. This is
>>>>>>>>>> getting
>>>>>>>>>> worse
>>>>>>>>>> and worse the more I try to recover my data. Is there any way to
>>>>>>>>>> back
>>>>>>>>>> up
>>>>>>>>>> the instances' volumes via the command line? I can't tell which
>>>>>>>>>> is
>>>>>>>>>> which
>>>>>>>>>> because the filenames are all hashes. I really need to get these
>>>>>>>>>> instances
>>>>>>>>>> up and running--there are several months worth of work at stake
>>>>>>>>>> here.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 29, 2014 at 6:13 PM, ma y <br...@gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I got the same problem, and how to downgrade CS 4.3.0 to 4.2.1
>>>>>>>>>> safely?
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> 2014-04-30 8:45 GMT+08:00 Ian Young <iy...@ratespecial.com>:
>>>>>>>>>>>
>>>>>>>>>>> Ok, my Cloudstack installation is now so broken that I think
>>>>>>>>>>> it's
>>>>>>>>>>> probably
>>>>>>>>>>>
>>>>>>>>>>> best to backup all my instances and templates, wipe the
>>>>>>>>>>>> databases, and
>>>>>>>>>>>> start from scratch. However, I can't take snapshots or download
>>>>>>>>>>>>
>>>>>>>>>>>> volumes
>>>>>>>>>>>
>>>>>>>>>>> anymore. What's causing these errors?
>>>>>>>>>>>>
>>>>>>>>>>>> 2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m.
>>>>>>>>>>>> AncientDataMotionStrategy]
>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed:
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send
>>>>>>>>>>>>
>>>>>>>>>>>> command,
>>>>>>>>>>>
>>>>>>>>>>> due to Agent:1, com.cloud.exception.OperationTimedoutException:
>>>>>>>>>>>>
>>>>>>>>>>>> Commands
>>>>>>>>>>>
>>>>>>>>>>> 841744457 to Host 1 timed out after 21600
>>>>>>>>>>>> 2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m.
>>>>>>>>>>>> AncientDataMotionStrategy]
>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send
>>>>>>>>>>>>
>>>>>>>>>>>> command,
>>>>>>>>>>>
>>>>>>>>>>> due to Agent:1, com.cloud.exception.OperationTimedoutException:
>>>>>>>>>>>>
>>>>>>>>>>>> Commands
>>>>>>>>>>>
>>>>>>>>>>> 841744457 to Host 1 timed out after 21600
>>>>>>>>>>>> 2014-04-29 17:40:51,269 WARN [o.a.c.s.d.
>>>>>>>>>>>> ObjectInDataStoreManagerImpl]
>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data
>>>>>>>>>>>> object
>>>>>>>>>>>> (VOLUME,
>>>>>>>>>>>> org.apache.cloudstack.storage.datastore.
>>>>>>>>>>>> PrimaryDataStoreImpl@7bbbd901
>>>>>>>>>>>> ),
>>>>>>>>>>>>
>>>>>>>>>>>> no
>>>>>>>>>>>
>>>>>>>>>>> need to delete from object in store ref table
>>>>>>>>>>>> 2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Unexpected exception while
>>>>>>>>>>>> executing
>>>>>>>>>>>> org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd
>>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to
>>>>>>>>>>>> copy the
>>>>>>>>>>>>
>>>>>>>>>>>> volume
>>>>>>>>>>>
>>>>>>>>>>> from the source primary storage pool to secondary storage.
>>>>>>>>>>>> 2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Complete async job-501,
>>>>>>>>>>>> jobStatus:
>>>>>>>>>>>>
>>>>>>>>>>>> FAILED,
>>>>>>>>>>>
>>>>>>>>>>> resultCode: 530, result:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/
>>>>>>>>>>>>
>>>>>>>>>>> null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>>>>>>>>>>>
>>>>>>>>>>> to copy the volume from the source primary storage pool to
>>>>>>>>>>>> secondary
>>>>>>>>>>>> storage."}
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 29, 2014 at 4:15 PM, Ian Young <
>>>>>>>>>>>> iyoung@ratespecial.com>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> I downgraded to 4.2.1 again but cloudstack-management won't
>>>>>>>>>>>> start
>>>>>>>>>>>> because
>>>>>>>>>>>> the database is version 4.3. Is it safe to restore the database
>>>>>>>>>>>> backup I
>>>>>>>>>>>> made prior to this whole process? In the meantime I have
>>>>>>>>>>>> destroyed
>>>>>>>>>>>> and
>>>>>>>>>>>> created system VMs, so I'm not sure it's a good idea.
>>>>>>>>>>>>
>>>>>>>>>>>>> On Apr 29, 2014 3:09 PM, "Ian Young" <iy...@ratespecial.com>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> @stevenliang: I take it back--you can't set the VM size when
>>>>>>>>>>>>> you
>>>>>>>>>>>>> register
>>>>>>>>>>>>> the template.
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz <
>>>>>>>>>>>>>> motty.cruz@gmail.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>> yes, you would have to shutdown the router, then click on
>>>>>>>>>>>>> "Change
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Service
>>>>>>>>>>>>>>
>>>>>>>>>>>>> Offering"
>>>>>>>>>>>>>
>>>>>>>>>>>>>> restart the VR.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> To Ian,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I suspect you forgot the last step: "
>>>>>>>>>>>>>>> cloudstack-setup-management"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> that would fix your issue, I think,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>> I downgraded to 4.2.1 and then upgraded to 4.3. Now the
>>>>>>>>>>>>>>> cloudstack-management service can't start because it can't
>>>>>>>>>>>>>>> connect
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>
>>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>> database.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2]
>>>>>>>>>>>>>>> (main:null)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Unable
>>>>>>>>>>>>>>
>>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>>> get a new db connection
>>>>>>>>>>>>>>> Caused by: java.sql.SQLException: Access denied for user
>>>>>>>>>>>>>>> 'cloud'@
>>>>>>>>>>>>>>> 'localhost'
>>>>>>>>>>>>>>> (using password: YES)
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Where are the credentials stored?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> stevenliang@yesup.com>
>>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> oh, then change service offering for vr?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 29/04/14 05:53 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> for my VR, I created a new
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> "System Offering For Software Router"
>>>>>>>>>>>>>>>>> CPU in (MHz) 1.00GHz
>>>>>>>>>>>>>>>>> Memory (in MB) 1.00GB
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> this are my current offerings, I'm sure the more RAM and
>>>>>>>>>>>>>>>>> CPU
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> performance.
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you again, motty.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I didn't notice this earlier.
>>>>>>>>>>>>>>>>>> BTW, how did you make your vr had 1GB CPU and 512MB RAM?
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 29/04/14 05:33 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Stevellang,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I not sure if you saw this in the forums earlier :
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> http://mail-archives.apache.
>>>>>>>>>>>>>>>>>>> org/mod_mbox/cloudstack-users/
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> 201404.mbox/%
>>>>>>>>>>>
>>>>>>>>>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc=
>>>>>>>>>>>>
>>>>>>>>>>>>> c2rfEDRoY9mA@mail.gmail.com%3E
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I don't know if the bug was fixed yet,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I will try upgrade in the next couple of days on a testing
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>>
>>>>>>>>>>>>>> report back if the bug was fixed.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thank you, motty.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am also running kvm. Since that time I failed
>>>>>>>>>>>>>>>>>>> upgrade, I am
>>>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>>
>>>>>>>>>>>>>> 4.2.1. I'll try as your advice.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 29/04/14 05:19 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Stevenllang,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I had the similar issue with VR, I notice it was
>>>>>>>>>>>>>>>>>>>> because I
>>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>>> default system specs on the VR, for instance by default
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 500MHz
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>>
>>>>>>>>>>>> CPU
>>>>>>>>>>>>>
>>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 128MB on RAM, if you upgrade to at least 1GB on CPU and
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 512MB of
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> RAM
>>>>>>>>>>>>
>>>>>>>>>>>>> your
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> VR will survive the upgrade from 4.2.1 to 4.3.1.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> I am running KVM, when I upgrade from 4.2.1 to 4.3 my
>>>>>>>>>>>>>>>>>>>>> VMs
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> were
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>
>>>>>>>>>>>> able
>>>>>>>>>>>>>
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>> access outside world, even if I created a new router.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> wish you the best,
>>>>>>>>>>>>>>>>>>>>> -motty
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Yes, I had two zones(one is basic, another is
>>>>>>>>>>>>>>>>>>>>> advanced
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> mode).
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> After I upgraded from 4.2.1 to 4.3, the vrouter lost.
>>>>>>>>>>>>
>>>>>>>>>>>>> So I rolled back to 4.2.1, the vrouter came back.
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On 29/04/14 04:54 PM, Ian Young wrote:
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> Did rolling back to 4.2 fix the problem?
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 1:22 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I met your situation before. Finally I rolled back
>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> 4.2
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 29/04/14 04:18 PM, Ian Young wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> I destroyed the old virtual router and was able to
>>>>>>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>>
>>>>>>>>>>>>> one
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> adding a new instance. However, this new
>>>>>>>>>>>>>>>>>>>>>>>> router also
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> failed
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>
>>>>>>>>>>>>>> start,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> citing the same error. After that, the expungement
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> delay
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> elapsed
>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>> virtual router was expunged, so now I have none.
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2014 at 8:52 PM, Ian Young <
>>>>>>>>>>>>>>>>>>>>>>>>> iyoung@ratespecial.com>
>>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I upgraded from 4.2.1 to 4.3.0 tonight,
>>>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> instructions
>>>>>>>>>>>>
>>>>>>>>>>>>> here:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> http://docs.cloudstack.apache.
>>>>>>>>>>>>>>>>>>>>>>>>> org/projects/cloudstack-
>>>>>>>>>>>>>>>>>>>>>>>>> release-notes/en/latest/
>>>>>>>>>>>>>>>>>>>>>>>>> rnotes.html#upgrade-from-4-2-
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> x-to-4-3
>>>>>>>>>>>>
>>>>>>>>>>>> At the last step, I tried to restart the system VMs.
>>>>>>>>>>>>>
>>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> virtual
>>>>>>>>>>>>
>>>>>>>>>>>>> router
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> failed to start. Here is the message that was
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> displayed in
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>
>>>>>>>>>>>>> web
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> UI:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>>
>>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> Unable to start VM[DomainRouter|r-4-VM] due to
>>>>>>>>>>>>>>>>>>>>>>>>>> error in
>>>>>>>>>>>>>>>>>>>>>>>>>> finalizeStart,
>>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>>> retrying
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> I tried running the script to restart the VMs but
>>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>>
>>>>>>>>>>>>> failed
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>>> start the console proxy:
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> [root@virthost1 ~]$ cloudstack-sysvmadm -d
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> 192.168.100.6
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> -u
>>>>>>>>>>>>
>>>>>>>>>>>> cloud
>>>>>>>>>>>>>
>>>>>>>>>>>>>> -p
>>>>>>>>>>>>>>>>>>>>>>>>>> -a
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 secondary storage vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting secondary storage vm(s)
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 console proxy vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: Failed to start console proxy vm with id 2
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting console proxy vm(s) .
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 0 running routing vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> Is there a way to wipe the system VMs out and
>>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>> over?
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>
>>>>
>>>
>>
>
Re: failed to start virtual router
Posted by Ian Young <iy...@ratespecial.com>.
I've tried upgrading to 4.3 again. After poking around some more in the
database, I've discovered that the KVM system VM template was only 27%
downloaded. I think this is why the virtual router was unable to
start--the template was incomplete. Is there a way to force it to resume
downloading?
On Wed, Apr 30, 2014 at 3:03 PM, Ian Young <iy...@ratespecial.com> wrote:
> The address in Infrastructure > Hosts > (management server) is set to the
> correct IP address, not 127.0.0.1. Why are the logs referring to 127.0.0.1?
>
>
> On Wed, Apr 30, 2014 at 3:00 PM, Ian Young <iy...@ratespecial.com> wrote:
>
>> I notice my dashboard says "Management server node 127.0.0.1 is up." It
>> used to have an actual address, not localhost. Could this be causing
>> problems and if so, how can I set it back?
>>
>>
>> On Wed, Apr 30, 2014 at 12:40 PM, Ian Young <iy...@ratespecial.com>wrote:
>>
>>> Yes, I replaced the new files with the rpmsave ones, which allowed the
>>> agent to start. However, most of the functions in the management console
>>> fail.
>>>
>>>
>>> On Wed, Apr 30, 2014 at 12:34 PM, stevenliang <st...@yesup.com>wrote:
>>>
>>>> Do you have the file db.properties.rpmsave on management server and
>>>> agent.properties.rpmsave on agents? If so, and the date is correct, you can
>>>> use it rather than db.properties and agent.properties.
>>>> And then restart management and agent services.
>>>>
>>>>
>>>> On 30/04/14 03:26 PM, Ian Young wrote:
>>>>
>>>>> Yes, I restored the DB from the backup. When I try to start the
>>>>> router it
>>>>> says:
>>>>>
>>>>> Resource [Host:1] is unreachable: Host 1: Unable to start instance due
>>>>> to
>>>>> Unable to start VM[DomainRouter|r-63-VM] due to error in
>>>>> finalizeStart, not
>>>>> retrying
>>>>>
>>>>> The management server log says:
>>>>>
>>>>> 2014-04-30 12:20:52,485 ERROR [cloud.async.AsyncJobManagerImpl]
>>>>> (Job-Executor-7:job-520 = [ 9d7c898c-b5d0-4bd0-a711-563a91d7acc9 ])
>>>>> Unexpected exception while executing
>>>>> org.apache.cloudstack.api.command.admin.router.StartRouterCmd
>>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>>>> unreachable: Host 1: Unable to start instance due to Unable to start
>>>>> VM[DomainRouter|r-63-VM] due to error in finalizeStart, not retrying
>>>>>
>>>>>
>>>>> On Wed, Apr 30, 2014 at 12:02 PM, stevenliang <st...@yesup.com>
>>>>> wrote:
>>>>>
>>>>> I think you had backed up database, when you upgraded.
>>>>>> When you downgraded CS, you also need to restore DB.
>>>>>>
>>>>>>
>>>>>> On 30/04/14 02:58 PM, Ian Young wrote:
>>>>>>
>>>>>> I think my problem stems from a partially downloaded system VM
>>>>>>> template.
>>>>>>> I
>>>>>>> just noticed systemvm-kvm-4.3 is stuck at 27% downloaded. It must
>>>>>>> have
>>>>>>> been interrupted during the upgrade to 4.3. At the moment I've
>>>>>>> rolled
>>>>>>> back
>>>>>>> to 4.2.1 with a somewhat usable management interface, although the
>>>>>>> system
>>>>>>> VMs won't start. I suspect there is something in the database that
>>>>>>> is
>>>>>>> causing it to try to use the 4.3 template. How can I delete the
>>>>>>> template
>>>>>>> and make sure the management server is using the older one?
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 29, 2014 at 8:23 PM, Ian Young <iy...@ratespecial.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Ok, so I've figured out a way to identify volumes in the
>>>>>>> filesystem. For
>>>>>>>
>>>>>>>> instance, /var/storage/primary/4d324e1a-e3a6-4da8-9c4d-44ad723482ad
>>>>>>>> is
>>>>>>>> the
>>>>>>>> root volume for an instance I want to back up. Is this in qcow2
>>>>>>>> format
>>>>>>>> or
>>>>>>>> something else? I'm using KVM.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 29, 2014 at 7:38 PM, Ian Young <iy...@ratespecial.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Now I can't start cloudstack-agent. The agent.log says:
>>>>>>>>
>>>>>>>>> Unable to start agent: Failed to get private nic name
>>>>>>>>>
>>>>>>>>> I know this is because the network bridge is no longer set up
>>>>>>>>> correctly.
>>>>>>>>> I used to have a cloud0 and a cloudbr0 interface. Now I only
>>>>>>>>> have
>>>>>>>>> cloudbr0. I haven't changed my network configuration. Somehow
>>>>>>>>> it's
>>>>>>>>> been
>>>>>>>>> changed by CloudStack during the upgrade/downgrade. This is
>>>>>>>>> getting
>>>>>>>>> worse
>>>>>>>>> and worse the more I try to recover my data. Is there any way to
>>>>>>>>> back
>>>>>>>>> up
>>>>>>>>> the instances' volumes via the command line? I can't tell which is
>>>>>>>>> which
>>>>>>>>> because the filenames are all hashes. I really need to get these
>>>>>>>>> instances
>>>>>>>>> up and running--there are several months worth of work at stake
>>>>>>>>> here.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Apr 29, 2014 at 6:13 PM, ma y <br...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I got the same problem, and how to downgrade CS 4.3.0 to 4.2.1
>>>>>>>>> safely?
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 2014-04-30 8:45 GMT+08:00 Ian Young <iy...@ratespecial.com>:
>>>>>>>>>>
>>>>>>>>>> Ok, my Cloudstack installation is now so broken that I think
>>>>>>>>>> it's
>>>>>>>>>> probably
>>>>>>>>>>
>>>>>>>>>> best to backup all my instances and templates, wipe the
>>>>>>>>>>> databases, and
>>>>>>>>>>> start from scratch. However, I can't take snapshots or download
>>>>>>>>>>>
>>>>>>>>>>> volumes
>>>>>>>>>>
>>>>>>>>>> anymore. What's causing these errors?
>>>>>>>>>>>
>>>>>>>>>>> 2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m.
>>>>>>>>>>> AncientDataMotionStrategy]
>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed:
>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send
>>>>>>>>>>>
>>>>>>>>>>> command,
>>>>>>>>>>
>>>>>>>>>> due to Agent:1, com.cloud.exception.OperationTimedoutException:
>>>>>>>>>>>
>>>>>>>>>>> Commands
>>>>>>>>>>
>>>>>>>>>> 841744457 to Host 1 timed out after 21600
>>>>>>>>>>> 2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m.
>>>>>>>>>>> AncientDataMotionStrategy]
>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed
>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send
>>>>>>>>>>>
>>>>>>>>>>> command,
>>>>>>>>>>
>>>>>>>>>> due to Agent:1, com.cloud.exception.OperationTimedoutException:
>>>>>>>>>>>
>>>>>>>>>>> Commands
>>>>>>>>>>
>>>>>>>>>> 841744457 to Host 1 timed out after 21600
>>>>>>>>>>> 2014-04-29 17:40:51,269 WARN [o.a.c.s.d.
>>>>>>>>>>> ObjectInDataStoreManagerImpl]
>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data
>>>>>>>>>>> object
>>>>>>>>>>> (VOLUME,
>>>>>>>>>>> org.apache.cloudstack.storage.datastore.
>>>>>>>>>>> PrimaryDataStoreImpl@7bbbd901
>>>>>>>>>>> ),
>>>>>>>>>>>
>>>>>>>>>>> no
>>>>>>>>>>
>>>>>>>>>> need to delete from object in store ref table
>>>>>>>>>>> 2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Unexpected exception while
>>>>>>>>>>> executing
>>>>>>>>>>> org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd
>>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to copy
>>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>> volume
>>>>>>>>>>
>>>>>>>>>> from the source primary storage pool to secondary storage.
>>>>>>>>>>> 2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Complete async job-501, jobStatus:
>>>>>>>>>>>
>>>>>>>>>>> FAILED,
>>>>>>>>>>
>>>>>>>>>> resultCode: 530, result:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/
>>>>>>>>>>>
>>>>>>>>>> null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>>>>>>>>>>
>>>>>>>>>> to copy the volume from the source primary storage pool to
>>>>>>>>>>> secondary
>>>>>>>>>>> storage."}
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Tue, Apr 29, 2014 at 4:15 PM, Ian Young <
>>>>>>>>>>> iyoung@ratespecial.com>
>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I downgraded to 4.2.1 again but cloudstack-management won't start
>>>>>>>>>>> because
>>>>>>>>>>> the database is version 4.3. Is it safe to restore the database
>>>>>>>>>>> backup I
>>>>>>>>>>> made prior to this whole process? In the meantime I have
>>>>>>>>>>> destroyed
>>>>>>>>>>> and
>>>>>>>>>>> created system VMs, so I'm not sure it's a good idea.
>>>>>>>>>>>
>>>>>>>>>>>> On Apr 29, 2014 3:09 PM, "Ian Young" <iy...@ratespecial.com>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> @stevenliang: I take it back--you can't set the VM size when
>>>>>>>>>>>> you
>>>>>>>>>>>> register
>>>>>>>>>>>> the template.
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz <
>>>>>>>>>>>>> motty.cruz@gmail.com>
>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>> yes, you would have to shutdown the router, then click on
>>>>>>>>>>>> "Change
>>>>>>>>>>>>
>>>>>>>>>>>>> Service
>>>>>>>>>>>>>
>>>>>>>>>>>> Offering"
>>>>>>>>>>>>
>>>>>>>>>>>>> restart the VR.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> To Ian,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I suspect you forgot the last step: "
>>>>>>>>>>>>>> cloudstack-setup-management"
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> that would fix your issue, I think,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>> I downgraded to 4.2.1 and then upgraded to 4.3. Now the
>>>>>>>>>>>>>> cloudstack-management service can't start because it can't
>>>>>>>>>>>>>> connect
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> to
>>>>>>>>>>>>>
>>>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>> database.
>>>>>>>>>>>>
>>>>>>>>>>>>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2]
>>>>>>>>>>>>>> (main:null)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Unable
>>>>>>>>>>>>>
>>>>>>>>>>>> to
>>>>>>>>>>>
>>>>>>>>>>>> get a new db connection
>>>>>>>>>>>>>> Caused by: java.sql.SQLException: Access denied for user
>>>>>>>>>>>>>> 'cloud'@
>>>>>>>>>>>>>> 'localhost'
>>>>>>>>>>>>>> (using password: YES)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Where are the credentials stored?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang <
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> stevenliang@yesup.com>
>>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> oh, then change service offering for vr?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 29/04/14 05:53 PM, motty cruz wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> for my VR, I created a new
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> "System Offering For Software Router"
>>>>>>>>>>>>>>>> CPU in (MHz) 1.00GHz
>>>>>>>>>>>>>>>> Memory (in MB) 1.00GB
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> this are my current offerings, I'm sure the more RAM and CPU
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> performance.
>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang <
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you again, motty.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I didn't notice this earlier.
>>>>>>>>>>>>>>>>> BTW, how did you make your vr had 1GB CPU and 512MB RAM?
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On 29/04/14 05:33 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Stevellang,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I not sure if you saw this in the forums earlier :
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> http://mail-archives.apache.
>>>>>>>>>>>>>>>>>> org/mod_mbox/cloudstack-users/
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> 201404.mbox/%
>>>>>>>>>>
>>>>>>>>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc=
>>>>>>>>>>>
>>>>>>>>>>>> c2rfEDRoY9mA@mail.gmail.com%3E
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I don't know if the bug was fixed yet,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I will try upgrade in the next couple of days on a testing
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> will
>>>>>>>>>>>>
>>>>>>>>>>>>> report back if the bug was fixed.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang <
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thank you, motty.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I am also running kvm. Since that time I failed
>>>>>>>>>>>>>>>>>> upgrade, I am
>>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> using
>>>>>>>>>>>>
>>>>>>>>>>>>> 4.2.1. I'll try as your advice.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 29/04/14 05:19 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Stevenllang,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I had the similar issue with VR, I notice it was
>>>>>>>>>>>>>>>>>>> because I
>>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>>> default system specs on the VR, for instance by default
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 500MHz
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> on
>>>>>>>>>>>
>>>>>>>>>>> CPU
>>>>>>>>>>>>
>>>>>>>>>>>>> and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 128MB on RAM, if you upgrade to at least 1GB on CPU and
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> 512MB of
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> RAM
>>>>>>>>>>>
>>>>>>>>>>>> your
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> VR will survive the upgrade from 4.2.1 to 4.3.1.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I am running KVM, when I upgrade from 4.2.1 to 4.3 my
>>>>>>>>>>>>>>>>>>>> VMs
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> were
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>
>>>>>>>>>>> able
>>>>>>>>>>>>
>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>> access outside world, even if I created a new router.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> wish you the best,
>>>>>>>>>>>>>>>>>>>> -motty
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Yes, I had two zones(one is basic, another is advanced
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> mode).
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> After I upgraded from 4.2.1 to 4.3, the vrouter lost.
>>>>>>>>>>>
>>>>>>>>>>>> So I rolled back to 4.2.1, the vrouter came back.
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On 29/04/14 04:54 PM, Ian Young wrote:
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> Did rolling back to 4.2 fix the problem?
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 1:22 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I met your situation before. Finally I rolled back
>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> 4.2
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 29/04/14 04:18 PM, Ian Young wrote:
>>>>>>>>>>>
>>>>>>>>>>>> I destroyed the old virtual router and was able to
>>>>>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>>
>>>>>>>>>>>> one
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> adding a new instance. However, this new router
>>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> failed
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>>>> start,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> citing the same error. After that, the expungement
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> delay
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> elapsed
>>>>>>>>>>>
>>>>>>>>>>>> and
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>> virtual router was expunged, so now I have none.
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2014 at 8:52 PM, Ian Young <
>>>>>>>>>>>>>>>>>>>>>>>> iyoung@ratespecial.com>
>>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I upgraded from 4.2.1 to 4.3.0 tonight,
>>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> instructions
>>>>>>>>>>>
>>>>>>>>>>>> here:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> http://docs.cloudstack.apache.
>>>>>>>>>>>>>>>>>>>>>>>> org/projects/cloudstack-
>>>>>>>>>>>>>>>>>>>>>>>> release-notes/en/latest/
>>>>>>>>>>>>>>>>>>>>>>>> rnotes.html#upgrade-from-4-2-
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> x-to-4-3
>>>>>>>>>>>
>>>>>>>>>>> At the last step, I tried to restart the system VMs.
>>>>>>>>>>>>
>>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> virtual
>>>>>>>>>>>
>>>>>>>>>>>> router
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> failed to start. Here is the message that was
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> displayed in
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>
>>>>>>>>>>>> web
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> UI:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable to
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>>
>>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> Unable to start VM[DomainRouter|r-4-VM] due to
>>>>>>>>>>>>>>>>>>>>>>>>> error in
>>>>>>>>>>>>>>>>>>>>>>>>> finalizeStart,
>>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>>> retrying
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> I tried running the script to restart the VMs but
>>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>>
>>>>>>>>>>>> failed
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>>> start the console proxy:
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> [root@virthost1 ~]$ cloudstack-sysvmadm -d
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> 192.168.100.6
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> -u
>>>>>>>>>>>
>>>>>>>>>>> cloud
>>>>>>>>>>>>
>>>>>>>>>>>>> -p
>>>>>>>>>>>>>>>>>>>>>>>>> -a
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 secondary storage vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting secondary storage vm(s)
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 console proxy vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>>> ERROR: Failed to start console proxy vm with id 2
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting console proxy vm(s) .
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 0 running routing vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> Is there a way to wipe the system VMs out and start
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>> over?
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>
>>>
>>
>
Re: failed to start virtual router
Posted by Ian Young <iy...@ratespecial.com>.
The address in Infrastructure > Hosts > (management server) is set to the
correct IP address, not 127.0.0.1. Why are the logs referring to 127.0.0.1?
On Wed, Apr 30, 2014 at 3:00 PM, Ian Young <iy...@ratespecial.com> wrote:
> I notice my dashboard says "Management server node 127.0.0.1 is up." It
> used to have an actual address, not localhost. Could this be causing
> problems and if so, how can I set it back?
>
>
> On Wed, Apr 30, 2014 at 12:40 PM, Ian Young <iy...@ratespecial.com>wrote:
>
>> Yes, I replaced the new files with the rpmsave ones, which allowed the
>> agent to start. However, most of the functions in the management console
>> fail.
>>
>>
>> On Wed, Apr 30, 2014 at 12:34 PM, stevenliang <st...@yesup.com>wrote:
>>
>>> Do you have the file db.properties.rpmsave on management server and
>>> agent.properties.rpmsave on agents? If so, and the date is correct, you can
>>> use it rather than db.properties and agent.properties.
>>> And then restart management and agent services.
>>>
>>>
>>> On 30/04/14 03:26 PM, Ian Young wrote:
>>>
>>>> Yes, I restored the DB from the backup. When I try to start the router
>>>> it
>>>> says:
>>>>
>>>> Resource [Host:1] is unreachable: Host 1: Unable to start instance due
>>>> to
>>>> Unable to start VM[DomainRouter|r-63-VM] due to error in finalizeStart,
>>>> not
>>>> retrying
>>>>
>>>> The management server log says:
>>>>
>>>> 2014-04-30 12:20:52,485 ERROR [cloud.async.AsyncJobManagerImpl]
>>>> (Job-Executor-7:job-520 = [ 9d7c898c-b5d0-4bd0-a711-563a91d7acc9 ])
>>>> Unexpected exception while executing
>>>> org.apache.cloudstack.api.command.admin.router.StartRouterCmd
>>>> com.cloud.exception.AgentUnavailableException: Resource [Host:1] is
>>>> unreachable: Host 1: Unable to start instance due to Unable to start
>>>> VM[DomainRouter|r-63-VM] due to error in finalizeStart, not retrying
>>>>
>>>>
>>>> On Wed, Apr 30, 2014 at 12:02 PM, stevenliang <st...@yesup.com>
>>>> wrote:
>>>>
>>>> I think you had backed up database, when you upgraded.
>>>>> When you downgraded CS, you also need to restore DB.
>>>>>
>>>>>
>>>>> On 30/04/14 02:58 PM, Ian Young wrote:
>>>>>
>>>>> I think my problem stems from a partially downloaded system VM
>>>>>> template.
>>>>>> I
>>>>>> just noticed systemvm-kvm-4.3 is stuck at 27% downloaded. It must
>>>>>> have
>>>>>> been interrupted during the upgrade to 4.3. At the moment I've rolled
>>>>>> back
>>>>>> to 4.2.1 with a somewhat usable management interface, although the
>>>>>> system
>>>>>> VMs won't start. I suspect there is something in the database that is
>>>>>> causing it to try to use the 4.3 template. How can I delete the
>>>>>> template
>>>>>> and make sure the management server is using the older one?
>>>>>>
>>>>>>
>>>>>> On Tue, Apr 29, 2014 at 8:23 PM, Ian Young <iy...@ratespecial.com>
>>>>>> wrote:
>>>>>>
>>>>>> Ok, so I've figured out a way to identify volumes in the
>>>>>> filesystem. For
>>>>>>
>>>>>>> instance, /var/storage/primary/4d324e1a-e3a6-4da8-9c4d-44ad723482ad
>>>>>>> is
>>>>>>> the
>>>>>>> root volume for an instance I want to back up. Is this in qcow2
>>>>>>> format
>>>>>>> or
>>>>>>> something else? I'm using KVM.
>>>>>>>
>>>>>>>
>>>>>>> On Tue, Apr 29, 2014 at 7:38 PM, Ian Young <iy...@ratespecial.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>> Now I can't start cloudstack-agent. The agent.log says:
>>>>>>>
>>>>>>>> Unable to start agent: Failed to get private nic name
>>>>>>>>
>>>>>>>> I know this is because the network bridge is no longer set up
>>>>>>>> correctly.
>>>>>>>> I used to have a cloud0 and a cloudbr0 interface. Now I only
>>>>>>>> have
>>>>>>>> cloudbr0. I haven't changed my network configuration. Somehow it's
>>>>>>>> been
>>>>>>>> changed by CloudStack during the upgrade/downgrade. This is getting
>>>>>>>> worse
>>>>>>>> and worse the more I try to recover my data. Is there any way to
>>>>>>>> back
>>>>>>>> up
>>>>>>>> the instances' volumes via the command line? I can't tell which is
>>>>>>>> which
>>>>>>>> because the filenames are all hashes. I really need to get these
>>>>>>>> instances
>>>>>>>> up and running--there are several months worth of work at stake
>>>>>>>> here.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Tue, Apr 29, 2014 at 6:13 PM, ma y <br...@gmail.com> wrote:
>>>>>>>>
>>>>>>>> I got the same problem, and how to downgrade CS 4.3.0 to 4.2.1
>>>>>>>> safely?
>>>>>>>>
>>>>>>>>>
>>>>>>>>> 2014-04-30 8:45 GMT+08:00 Ian Young <iy...@ratespecial.com>:
>>>>>>>>>
>>>>>>>>> Ok, my Cloudstack installation is now so broken that I think it's
>>>>>>>>> probably
>>>>>>>>>
>>>>>>>>> best to backup all my instances and templates, wipe the
>>>>>>>>>> databases, and
>>>>>>>>>> start from scratch. However, I can't take snapshots or download
>>>>>>>>>>
>>>>>>>>>> volumes
>>>>>>>>>
>>>>>>>>> anymore. What's causing these errors?
>>>>>>>>>>
>>>>>>>>>> 2014-04-29 17:40:51,264 DEBUG [o.a.c.s.m.
>>>>>>>>>> AncientDataMotionStrategy]
>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy object failed:
>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send
>>>>>>>>>>
>>>>>>>>>> command,
>>>>>>>>>
>>>>>>>>> due to Agent:1, com.cloud.exception.OperationTimedoutException:
>>>>>>>>>>
>>>>>>>>>> Commands
>>>>>>>>>
>>>>>>>>> 841744457 to Host 1 timed out after 21600
>>>>>>>>>> 2014-04-29 17:40:51,265 DEBUG [o.a.c.s.m.
>>>>>>>>>> AncientDataMotionStrategy]
>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) copy failed
>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException:
>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to send
>>>>>>>>>>
>>>>>>>>>> command,
>>>>>>>>>
>>>>>>>>> due to Agent:1, com.cloud.exception.OperationTimedoutException:
>>>>>>>>>>
>>>>>>>>>> Commands
>>>>>>>>>
>>>>>>>>> 841744457 to Host 1 timed out after 21600
>>>>>>>>>> 2014-04-29 17:40:51,269 WARN [o.a.c.s.d.
>>>>>>>>>> ObjectInDataStoreManagerImpl]
>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79 ctx-315eda05) Unsupported data
>>>>>>>>>> object
>>>>>>>>>> (VOLUME,
>>>>>>>>>> org.apache.cloudstack.storage.datastore.
>>>>>>>>>> PrimaryDataStoreImpl@7bbbd901
>>>>>>>>>> ),
>>>>>>>>>>
>>>>>>>>>> no
>>>>>>>>>
>>>>>>>>> need to delete from object in store ref table
>>>>>>>>>> 2014-04-29 17:40:51,280 ERROR [c.c.a.ApiAsyncJobDispatcher]
>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Unexpected exception while
>>>>>>>>>> executing
>>>>>>>>>> org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd
>>>>>>>>>> com.cloud.utils.exception.CloudRuntimeException: Failed to copy
>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>> volume
>>>>>>>>>
>>>>>>>>> from the source primary storage pool to secondary storage.
>>>>>>>>>> 2014-04-29 17:40:51,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
>>>>>>>>>> (Job-Executor-11:ctx-0a3ead79) Complete async job-501, jobStatus:
>>>>>>>>>>
>>>>>>>>>> FAILED,
>>>>>>>>>
>>>>>>>>> resultCode: 530, result:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> org.apache.cloudstack.api.response.ExceptionResponse/
>>>>>>>>>>
>>>>>>>>> null/{"uuidList":[],"errorcode":530,"errortext":"Failed
>>>>>>>>>
>>>>>>>>> to copy the volume from the source primary storage pool to
>>>>>>>>>> secondary
>>>>>>>>>> storage."}
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Apr 29, 2014 at 4:15 PM, Ian Young <
>>>>>>>>>> iyoung@ratespecial.com>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> I downgraded to 4.2.1 again but cloudstack-management won't start
>>>>>>>>>> because
>>>>>>>>>> the database is version 4.3. Is it safe to restore the database
>>>>>>>>>> backup I
>>>>>>>>>> made prior to this whole process? In the meantime I have destroyed
>>>>>>>>>> and
>>>>>>>>>> created system VMs, so I'm not sure it's a good idea.
>>>>>>>>>>
>>>>>>>>>>> On Apr 29, 2014 3:09 PM, "Ian Young" <iy...@ratespecial.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> @stevenliang: I take it back--you can't set the VM size when
>>>>>>>>>>> you
>>>>>>>>>>> register
>>>>>>>>>>> the template.
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Tue, Apr 29, 2014 at 3:02 PM, motty cruz <
>>>>>>>>>>>> motty.cruz@gmail.com>
>>>>>>>>>>>>
>>>>>>>>>>>> wrote:
>>>>>>>>>>> yes, you would have to shutdown the router, then click on "Change
>>>>>>>>>>>
>>>>>>>>>>>> Service
>>>>>>>>>>>>
>>>>>>>>>>> Offering"
>>>>>>>>>>>
>>>>>>>>>>>> restart the VR.
>>>>>>>>>>>>>
>>>>>>>>>>>>> To Ian,
>>>>>>>>>>>>>
>>>>>>>>>>>>> I suspect you forgot the last step: "
>>>>>>>>>>>>> cloudstack-setup-management"
>>>>>>>>>>>>>
>>>>>>>>>>>>> that would fix your issue, I think,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> I downgraded to 4.2.1 and then upgraded to 4.3. Now the
>>>>>>>>>>>>> cloudstack-management service can't start because it can't
>>>>>>>>>>>>> connect
>>>>>>>>>>>>>
>>>>>>>>>>>>> to
>>>>>>>>>>>>
>>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>> database.
>>>>>>>>>>>
>>>>>>>>>>>> 2014-04-29 14:51:36,424 ERROR [c.c.u.d.Merovingian2] (main:null)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Unable
>>>>>>>>>>>>
>>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>>>> get a new db connection
>>>>>>>>>>>>> Caused by: java.sql.SQLException: Access denied for user
>>>>>>>>>>>>> 'cloud'@
>>>>>>>>>>>>> 'localhost'
>>>>>>>>>>>>> (using password: YES)
>>>>>>>>>>>>>
>>>>>>>>>>>>> Where are the credentials stored?
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:57 PM, stevenliang <
>>>>>>>>>>>>>
>>>>>>>>>>>>> stevenliang@yesup.com>
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> oh, then change service offering for vr?
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 29/04/14 05:53 PM, motty cruz wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> for my VR, I created a new
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> "System Offering For Software Router"
>>>>>>>>>>>>>>> CPU in (MHz) 1.00GHz
>>>>>>>>>>>>>>> Memory (in MB) 1.00GB
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> this are my current offerings, I'm sure the more RAM and CPU
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> better
>>>>>>>>>>>>>>
>>>>>>>>>>>>> performance.
>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:44 PM, stevenliang <
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Thank you again, motty.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I didn't notice this earlier.
>>>>>>>>>>>>>>>> BTW, how did you make your vr had 1GB CPU and 512MB RAM?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On 29/04/14 05:33 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Stevellang,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I not sure if you saw this in the forums earlier :
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> http://mail-archives.apache.
>>>>>>>>>>>>>>>>> org/mod_mbox/cloudstack-users/
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> 201404.mbox/%
>>>>>>>>>
>>>>>>>>> 3CCALoOYy6A10bz1zOQQs1VyFb9epqLfhf7mu6hc=
>>>>>>>>>>
>>>>>>>>>>> c2rfEDRoY9mA@mail.gmail.com%3E
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I don't know if the bug was fixed yet,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I will try upgrade in the next couple of days on a testing
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> cluster,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> will
>>>>>>>>>>>
>>>>>>>>>>>> report back if the bug was fixed.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:25 PM, stevenliang <
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Thank you, motty.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I am also running kvm. Since that time I failed upgrade,
>>>>>>>>>>>>>>>>> I am
>>>>>>>>>>>>>>>>> still
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> using
>>>>>>>>>>>
>>>>>>>>>>>> 4.2.1. I'll try as your advice.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On 29/04/14 05:19 PM, motty cruz wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Stevenllang,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> I had the similar issue with VR, I notice it was
>>>>>>>>>>>>>>>>>> because I
>>>>>>>>>>>>>>>>>> leave
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> default system specs on the VR, for instance by default
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 500MHz
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> on
>>>>>>>>>>
>>>>>>>>>> CPU
>>>>>>>>>>>
>>>>>>>>>>>> and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 128MB on RAM, if you upgrade to at least 1GB on CPU and
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> 512MB of
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> RAM
>>>>>>>>>>
>>>>>>>>>>> your
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> VR will survive the upgrade from 4.2.1 to 4.3.1.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> I am running KVM, when I upgrade from 4.2.1 to 4.3 my VMs
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> were
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> not
>>>>>>>>>>
>>>>>>>>>> able
>>>>>>>>>>>
>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>> access outside world, even if I created a new router.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> wish you the best,
>>>>>>>>>>>>>>>>>>> -motty
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 2:13 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Yes, I had two zones(one is basic, another is advanced
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> mode).
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> After I upgraded from 4.2.1 to 4.3, the vrouter lost.
>>>>>>>>>>
>>>>>>>>>>> So I rolled back to 4.2.1, the vrouter came back.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On 29/04/14 04:54 PM, Ian Young wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Did rolling back to 4.2 fix the problem?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Apr 29, 2014 at 1:22 PM, stevenliang <
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> stevenliang@yesup.com
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I met your situation before. Finally I rolled back to
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> 4.2
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On 29/04/14 04:18 PM, Ian Young wrote:
>>>>>>>>>>
>>>>>>>>>>> I destroyed the old virtual router and was able to
>>>>>>>>>>>>>>>>>>>>> create
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> a new
>>>>>>>>>>
>>>>>>>>>>> one
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> by
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> adding a new instance. However, this new router
>>>>>>>>>>>>>>>>>>>>>> also
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> failed
>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>
>>>>>>>>>>>> start,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> citing the same error. After that, the expungement
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> delay
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> elapsed
>>>>>>>>>>
>>>>>>>>>>> and
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>> virtual router was expunged, so now I have none.
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> On Mon, Apr 28, 2014 at 8:52 PM, Ian Young <
>>>>>>>>>>>>>>>>>>>>>>> iyoung@ratespecial.com>
>>>>>>>>>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> I upgraded from 4.2.1 to 4.3.0 tonight,
>>>>>>>>>>>>>>>>>>>>>>> following
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>> instructions
>>>>>>>>>>
>>>>>>>>>>> here:
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>> http://docs.cloudstack.apache.
>>>>>>>>>>>>>>>>>>>>>>> org/projects/cloudstack-
>>>>>>>>>>>>>>>>>>>>>>> release-notes/en/latest/
>>>>>>>>>>>>>>>>>>>>>>> rnotes.html#upgrade-from-4-2-
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> x-to-4-3
>>>>>>>>>>
>>>>>>>>>> At the last step, I tried to restart the system VMs.
>>>>>>>>>>>
>>>>>>>>>>>> The
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> virtual
>>>>>>>>>>
>>>>>>>>>>> router
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> failed to start. Here is the message that was
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> displayed in
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>>> web
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> UI:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Resource [Host:1] is unreachable: Host 1: Unable to
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> start
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> instance
>>>>>>>>>>
>>>>>>>>>>> due
>>>>>>>>>>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> Unable to start VM[DomainRouter|r-4-VM] due to
>>>>>>>>>>>>>>>>>>>>>>>> error in
>>>>>>>>>>>>>>>>>>>>>>>> finalizeStart,
>>>>>>>>>>>>>>>>>>>>>>>> not
>>>>>>>>>>>>>>>>>>>>>>>> retrying
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> I tried running the script to restart the VMs but
>>>>>>>>>>>>>>>>>>>>>>>> this
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> time
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> it
>>>>>>>>>>
>>>>>>>>>>> failed
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> to
>>>>>>>>>>>>>>>>>>>>>>>> start the console proxy:
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> [root@virthost1 ~]$ cloudstack-sysvmadm -d
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> 192.168.100.6
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>> -u
>>>>>>>>>>
>>>>>>>>>> cloud
>>>>>>>>>>>
>>>>>>>>>>>> -p
>>>>>>>>>>>>>>>>>>>>>>>> -a
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 secondary storage vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting secondary storage vm(s)
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 1 console proxy vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>> ERROR: Failed to start console proxy vm with id 2
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Done stopping and starting console proxy vm(s) .
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Stopping and starting 0 running routing vm(s)...
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> Is there a way to wipe the system VMs out and start
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>> over?
>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>
>>>
>>
>