You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@cloudstack.apache.org by Cloud List <cl...@sg.or.id> on 2016/06/06 16:50:47 UTC

Upgrading from ACS 4.2 to 4.8 - VR upgrade issue

Hi,

I tried upgrading our CloudStack 4.2.0 on our test environment to 4.8.0,
and I am having issues on VR upgrade. I followed the upgrade instructions
at below URL:

http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upgrade-4.2.html

On the last step to restart all the system VMs and VRs, it seems that the
cloudstack-sysvmadm script managed to restart the system VMs (SSVM and
CPVM) but not the VR.

# nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a >
sysvm.log 2>&1 &

Here's the output of the sysvm.log file:

====
nohup: ignoring input
/usr/bin/cloudstack-sysvmadm: line 21: /etc/rc.d/init.d/functions: No such
file or directory

Stopping and starting 1 secondary storage vm(s)...
Done stopping and starting secondary storage vm(s)

Stopping and starting 1 console proxy vm(s)...
Done stopping and starting console proxy vm(s) .

Stopping and starting 1 running routing vm(s)...
*ERROR: Failed to restart domainRouter with id 5*

Done restarting router(s).
====

Output of the cloud.log file:

====
[2016.06.05-22.28.40] Stopping and starting 1 secondary storage vm(s)...
[2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id 3
[2016.06.05-22.28.46] INFO: Starting secondary storage vm with id 3
[2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3; jobId is
121c5d89-745a-4e8e-9f96-0b348ef42abf
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id 3
[2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm with id 3
[2016.06.05-23.29.00] Done stopping and starting secondary storage vm(s).
[2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
[2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
[2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
[2016.06.05-23.30.49] Done stopping and starting console proxy vm(s) .
[2016.06.05-23.30.49] Stopping and starting 1 running routing vm(s)...
[2016.06.05-23.30.49] INFO: Restarting router with id 5
[2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with id 5;
jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
[2016.06.06-00.31.04] Done restarting router(s).
====

Initially I noted that the VR remains in 4.2.0 version (instead of the
expected 4.8.0 version), tried to destroy the router so it'll be recreated
but it doesn't get created successfully.

Tried to run the cloudstack-sysvmadm manually with the -r option to restart
only the VR, and the problem still persists.

At the moment, the version of the VR is "UNKNOWN" and the "Require Upgrade"
state is "Yes".

Any advice how to make the VR back up and running?

Looking forward to your reply, thank you.

Cheers.

-ip-

Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue

Posted by Indra Pramana <in...@sg.or.id>.

Thanks Sanjeev,

I have re-done the upgrade again and after some other hiccups along the
way, finally managed to perform the upgrade and all the system VMs are now
updated successfully, including the VR. Not too sure what caused the VR
upgrade failed on the first upgrade attempt, though. And since I couldn't
find the solution, the only way was to revert back to version 4.2
(re-install CloudStack version 4.2, restore the DB) and restart the upgrade
process again.

Cheers.


On Wed, Jun 8, 2016 at 1:19 PM, Sanjeev Neelarapu <
sanjeev.neelarapu@accelerite.com> wrote:

> Check /etc/cloudstack-releases file inside system vms to know the system
> template version after system vm upgrades.
> Check MS log to know the reasons for VR upgrade failure.
>
> Best Regards,
> Sanjeev N
> Chief Product Engineer, Accelerite
> Off: +91 40 6722 9368 | EMail: sanjeev.neelarapu@accelerite.com
>
>
> -----Original Message-----
> From: Cloud List [mailto:cloud-list@sg.or.id]
> Sent: Wednesday, June 08, 2016 10:19 AM
> To: users@cloudstack.apache.org
> Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
>
> Hi Sam,
>
> Thanks for your reply.
>
> Connection to MS is OK, since SSVM works fine. Problem seems to be
> isolated to the VR. I am using KVM hypervisor. Testing the upgrade again
> now and will update the result.
>
> Cheers.
>
> -ip-
>
>
> On Wed, Jun 8, 2016 at 12:41 PM, Sam Ceylani <sa...@mistercertified.com>
> wrote:
>
> > So check your firewall on your management server, test that
> > connection. I believe there are some new stuff introduced that changed
> > how routers and system vms connect to MS. But you said SSVM connects
> > fine right? What Hypervisor are you using?
> > Because I believe they changed system vm connections to ssh on xen for
> > system vms etc. and there was even a port number that I know I put in
> > to my firewall which wasn’t there before...
> >
> > -----Original Message-----
> > From: Cloud List [mailto:cloud-list@sg.or.id]
> > Sent: Wednesday, June 8, 2016 12:37 AM
> > To: users@cloudstack.apache.org
> > Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
> >
> > Hi Sam,
> >
> > Thanks for your reply.
> >
> > My SSVM and CPVM seems to work fine -- although I can't verify if they
> > are really using the new template or not, but my VR doesn't work even
> > after I destroy the VR and restart the network with clean-up option.
> > So the VR is never able to be started up after the upgrade. It kept on
> > "Starting" state and agent.log file shows it keeps on trying to
> > connect to the link local IP but never able to connect.
> >
> > I tested the upgrade on a staging environment with just 1 VR, I have
> > reverted back the setup to 4.2 and now re-doing the upgrade again for
> > the second attempt, will see if it works. My production has around 10+
> > networks so I need to ensure this works before I initiate the upgrade
> > on my production server.
> >
> > Thank you.
> >
> >
> > On Wed, Jun 8, 2016 at 12:26 PM, Sam Ceylani <sa...@mistercertified.com>
> > wrote:
> >
> > > Same thing happened to me, upgrading from 4.5.2 to 4.8 and I had to
> > > restart network with clean-up option checked for each network which
> > > rebuilds the router VM. If your SSVM works And if you can test it
> > > then I don’t see any problem rebuilding network(router vm) with new
> > > template which took about 30 minutes for about
> > > 20 networks in total...may not be an option for large
> > > installations...how many routers you are running?
> > >
> > > -----Original Message-----
> > > From: Cloud List [mailto:cloud-list@sg.or.id]
> > > Sent: Wednesday, June 8, 2016 12:23 AM
> > > To: users@cloudstack.apache.org; dev@cloudstack.apache.org
> > > Subject: Re: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
> > >
> > > Dear all,
> > >
> > > Could it be the VR is not able to use the new systemVM template
> > > because we registered the template as Routing = NO? Will it resolve
> > > the problem if during upgrade, we register the template as Routing =
> YES?
> > >
> > > Based on the upgrade instruction at
> > >
> > > http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upg
> > > ra de-4.2.html , it says that routing option should be "no", is this
> > > correct? Since the systemVM template will also be used by VR?
> > >
> > > Any advice is appreciated.
> > >
> > > Looking forward to your reply, thank you.
> > >
> > > Cheers.
> > >
> > > -ip-
> > >
> > >
> > > On Tue, Jun 7, 2016 at 10:22 AM, Cloud List <cl...@sg.or.id>
> wrote:
> > >
> > > > Hi Kirk,
> > > >
> > > > Thanks for your reply.
> > > >
> > > > Here's the status of the CPVM and SSVM in the UI:
> > > >
> > > > Name    v-2-VM
> > > > ID    fc6d23bf-58ac-4641-af9b-506d1dc46df3
> > > > State    Running
> > > > Type    Console Proxy VM
> > > > Zone    sc-test-cs42-zone
> > > > Public IP Address    192.168.0.162
> > > > Private IP Address    192.168.0.160
> > > > Link Local IP Address    169.254.3.147
> > > > Host    sc-test-kvm-03
> > > > Gateway    192.168.0.1
> > > > Created    31 May 2016 15:24:20
> > > > Active Sessions    0
> > > >
> > > > Name    s-29-VM
> > > > ID    d26f54f8-1c1f-4f3d-9862-fc0405159620
> > > > State    Running
> > > > Type    Secondary Storage VM
> > > > Zone    sc-test-cs42-zone
> > > > Public IP Address    192.168.0.161
> > > > Private IP Address    192.168.0.151
> > > > Link Local IP Address    169.254.1.207
> > > > Host    sc-test-kvm-03
> > > > Gateway    192.168.0.1
> > > > Created    06 Jun 2016 00:31:04
> > > > Active Sessions
> > > >
> > > > Both system VMs seem to be on "Running" state but I can't verify
> > > > what template version they are using, whether they are already on
> > > > 4.6/4.8 template or still using 4.2 template, are you able to
> > > > advise how can I tell? The SSVM creation date seem to be after we
> > > > performed the upgrade but the CPVM created date is before the
> upgrade.
> > > >
> > > > How to ensure that all systemVMs using the new version template?
> > > > Shall I destroy the system VMs so new ones will be created, or
> > > > shall I re-run the /usr/bin/cloudstack-sysvmadm again in full?
> > > >
> > > > Looking forward to your reply, thank you.
> > > >
> > > > Cheers.
> > > >
> > > > -ip-
> > > >
> > > >
> > > >
> > > > On Tue, Jun 7, 2016 at 7:22 AM, Kirk Kosinski
> > > > <kirk.kosinski@shapeblue.com
> > > > > wrote:
> > > >
> > > >> Hi, what is the status of the CPVM and SSVM in the UI?  Were they
> > > >> really upgraded successfully?
> > > >>
> > > >> Also, are there any errors in the management-server.log?
> > > >>
> > > >> kirk.kosinski@shapeblue.com
> > > >> www.shapeblue.com
> > > >> 53 Chandos Place, Covent Garden, London  WC2N 4HS @shapeblue
> > > >>
> > > >>
> > > >>
> > > >> -----Original Message-----
> > > >> From: Cloud List [mailto:cloud-list@sg.or.id]
> > > >> Sent: Monday, June 06, 2016 9:51 AM
> > > >> To: users@cloudstack.apache.org; dev@cloudstack.apache.org
> > > >> Subject: Upgrading from ACS 4.2 to 4.8 - VR upgrade issue
> > > >>
> > > >> Hi,
> > > >>
> > > >> I tried upgrading our CloudStack 4.2.0 on our test environment to
> > > >> 4.8.0, and I am having issues on VR upgrade. I followed the
> > > >> upgrade
> > > instructions
> > > >> at below URL:
> > > >>
> > > >>
> > > >>
> > > http://cloudstack-release-notes.readthedocs.io/ja/stable/upgrade/upg
> > > ra
> > > de-4.2.html
> > > >>
> > > >> On the last step to restart all the system VMs and VRs, it seems
> > > >> that
> > > the
> > > >> cloudstack-sysvmadm script managed to restart the system VMs
> > > >> (SSVM and
> > > >> CPVM) but not the VR.
> > > >>
> > > >> # nohup cloudstack-sysvmadm -d IPaddress -u cloud -p password -a
> > > >> > sysvm.log 2>&1 &
> > > >>
> > > >> Here's the output of the sysvm.log file:
> > > >>
> > > >> ====
> > > >> nohup: ignoring input
> > > >> /usr/bin/cloudstack-sysvmadm: line 21: /etc/rc.d/init.d/functions:
> > > >> No such file or directory
> > > >>
> > > >> Stopping and starting 1 secondary storage vm(s)...
> > > >> Done stopping and starting secondary storage vm(s)
> > > >>
> > > >> Stopping and starting 1 console proxy vm(s)...
> > > >> Done stopping and starting console proxy vm(s) .
> > > >>
> > > >> Stopping and starting 1 running routing vm(s)...
> > > >> *ERROR: Failed to restart domainRouter with id 5*
> > > >>
> > > >> Done restarting router(s).
> > > >> ====
> > > >>
> > > >> Output of the cloud.log file:
> > > >>
> > > >> ====
> > > >> [2016.06.05-22.28.40] Stopping and starting 1 secondary storage
> > vm(s)...
> > > >> [2016.06.05-22.28.40] INFO: Stopping secondary storage vm with id
> > > >> 3 [2016.06.05-22.28.46] INFO: Starting secondary storage vm with
> > > >> id 3 [2016.06.05-23.29.00] ERROR: Failed to startSystemVm id=3;
> > > >> jobId is 121c5d89-745a-4e8e-9f96-0b348ef42abf
> > > >> [2016.06.05-23.29.00] ERROR: Failed to start secondary storage vm
> > > >> with
> > > id
> > > >> 3 [2016.06.05-23.29.00] ERROR: Failed to start secondary storage
> > > >> vm
> > > with id
> > > >> 3 [2016.06.05-23.29.00] Done stopping and starting secondary
> > > >> storage
> > > vm(s).
> > > >> [2016.06.05-23.29.00] Stopping and starting 1 console proxy vm(s)...
> > > >> [2016.06.05-23.29.00] INFO: Stopping console proxy with id 2
> > > >> [2016.06.05-23.29.06] INFO: Starting console proxy vm with id 2
> > > >> [2016.06.05-23.30.49] Done stopping and starting console proxy
> vm(s) .
> > > >> [2016.06.05-23.30.49] Stopping and starting 1 running routing
> vm(s)...
> > > >> [2016.06.05-23.30.49] INFO: Restarting router with id 5
> > > >> [2016.06.06-00.30.59] ERROR: Failed to restart domainRouter with
> > > >> id 5; jobId 4a3f681e-c4a6-4856-9d42-0d3b79bffdae
> > > >> [2016.06.06-00.31.04] Done restarting router(s).
> > > >> ====
> > > >>
> > > >> Initially I noted that the VR remains in 4.2.0 version (instead
> > > >> of the expected 4.8.0 version), tried to destroy the router so
> > > >> it'll be
> > > recreated
> > > >> but it doesn't get created successfully.
> > > >>
> > > >> Tried to run the cloudstack-sysvmadm manually with the -r option
> > > >> to restart only the VR, and the problem still persists.
> > > >>
> > > >> At the moment, the version of the VR is "UNKNOWN" and the
> > > >> "Require Upgrade"
> > > >> state is "Yes".
> > > >>
> > > >> Any advice how to make the VR back up and running?
> > > >>
> > > >> Looking forward to your reply, thank you.
> > > >>
> > > >> Cheers.
> > > >>
> > > >> -ip-
> > > >>
> > > >
> > > >
> > >
> >
>
>
>
> DISCLAIMER
> ==========
> This e-mail may contain privileged and confidential information which is
> the property of Accelerite, a Persistent Systems business. It is intended
> only for the use of the individual or entity to which it is addressed. If
> you are not the intended recipient, you are not authorized to read, retain,
> copy, print, distribute or use this message. If you have received this
> communication in error, please notify the sender and delete all copies of
> this message. Accelerite, a Persistent Systems business does not accept any
> liability for virus infected mails.
>