You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Eric Lee Green <er...@gmail.com> on 2019/05/22 00:51:16 UTC
Upgrade to Cloudstack 4.11.2 fails *AGAIN*
You may remember me as the person who had to roll back to Cloudstack
4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once
I upgraded to it, claiming that there were inadequate resources even
though I had over 150 gigabytes of memory free in my cluster and oodles
of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb
router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
*again* it's misbehaving.
The symptom is that my virtual routers when I log into their console
show 4.11.2 but when I look at them in the console they say 'Version:
UNKNOWN'. Also when I try to ssh into their guest IP address or link
local IP address it fails. And when I try to start up a virtual machine
that uses that virtual network, it says "Network unavailable', even
though the router for that network is showing up and running.
Clearly something's broken in the virtual routers but I don't know what
because I can't get into the router virtual machines. How do I get the
console password to get into the router virtual machines? It's encrypted
in the database (duh), how do I decrypt it?
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Nux! <nu...@li.nux.ro>.
While the VR does not use cloud-init (afaik!), it's worth mentioning that in recent tests with CentOS 7 VMs and cloud-init there were problems when using too low RAM offerings.
In my particular case 512MB turned out to not be enough for cloud-init to execute properly, upgrading the offering "fixed" the issue.
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Eric Lee Green" <er...@gmail.com>
> To: "users" <us...@cloudstack.apache.org>
> Sent: Wednesday, 22 May, 2019 07:48:05
> Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>
> That is not 512 MB of RAM, that is 400MB of ram, but that is still
> enough to fire up the VM and the VM did in fact start up. It just didn't
> do anything once it started up, it was as if cloud-init was not getting
> any input telling it what to do (that big JSON blob that I see flowing
> from the management server to the agent to feed to the instance).
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Eric Lee Green <er...@gmail.com>.
On 5/21/2019 11:14 PM, Rene Moser wrote:
> Just a suspicion: have you checked, the system VMs actually have
> allocated 512 MB RAM? I remember the systemvm templates had the setting
> to only use 256 MB RAM in previous versions which is too low. IMHO this
> setting must be adjusted manually in the database before the upgrade
> (you can do it while running cloudstack 4.9)
Hmm... during the process of when it was trying to start up the virtual
router, I see this:
> Trying to allocate a host and storage pools from dc:1,
pod:null,cluster:null, requested cpu: 4000, requested ram: 4294967296
That is not 512 MB of RAM, that is 400MB of ram, but that is still
enough to fire up the VM and the VM did in fact start up. It just didn't
do anything once it started up, it was as if cloud-init was not getting
any input telling it what to do (that big JSON blob that I see flowing
from the management server to the agent to feed to the instance).
>
> I would appreciate if one of the list could assist to check and change
> if necessary.
>
> Regards
> René
>
>
>
> On 5/22/19 2:51 AM, Eric Lee Green wrote:
>> You may remember me as the person who had to roll back to Cloudstack
>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once
>> I upgraded to it, claiming that there were inadequate resources even
>> though I had over 150 gigabytes of memory free in my cluster and oodles
>> of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb
>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
>> *again* it's misbehaving.
>>
>> The symptom is that my virtual routers when I log into their console
>> show 4.11.2 but when I look at them in the console they say 'Version:
>> UNKNOWN'. Also when I try to ssh into their guest IP address or link
>> local IP address it fails. And when I try to start up a virtual machine
>> that uses that virtual network, it says "Network unavailable', even
>> though the router for that network is showing up and running.
>>
>> Clearly something's broken in the virtual routers but I don't know what
>> because I can't get into the router virtual machines. How do I get the
>> console password to get into the router virtual machines? It's encrypted
>> in the database (duh), how do I decrypt it?
>>
>>
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Rene Moser <ma...@renemoser.net>.
Just a suspicion: have you checked, the system VMs actually have
allocated 512 MB RAM? I remember the systemvm templates had the setting
to only use 256 MB RAM in previous versions which is too low. IMHO this
setting must be adjusted manually in the database before the upgrade
(you can do it while running cloudstack 4.9)
I would appreciate if one of the list could assist to check and change
if necessary.
Regards
René
On 5/22/19 2:51 AM, Eric Lee Green wrote:
> You may remember me as the person who had to roll back to Cloudstack
> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once
> I upgraded to it, claiming that there were inadequate resources even
> though I had over 150 gigabytes of memory free in my cluster and oodles
> of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb
> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
> *again* it's misbehaving.
>
> The symptom is that my virtual routers when I log into their console
> show 4.11.2 but when I look at them in the console they say 'Version:
> UNKNOWN'. Also when I try to ssh into their guest IP address or link
> local IP address it fails. And when I try to start up a virtual machine
> that uses that virtual network, it says "Network unavailable', even
> though the router for that network is showing up and running.
>
> Clearly something's broken in the virtual routers but I don't know what
> because I can't get into the router virtual machines. How do I get the
> console password to get into the router virtual machines? It's encrypted
> in the database (duh), how do I decrypt it?
>
>
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Eric Lee Green <er...@gmail.com>.
I don't have spare hardware. I work for a small business, i.e., poor,
which is why the hardware is the antiques I listed, it's all basically
other company's garbage sourced from eBay. I tested it on a small
cluster used by QA to test software deployments on various versions of
Windows and Ubuntu Linux, our infrastructure is Centos for historical
reasons, not because there's any real reason to run Centos other than
standardization. We don't release our software until we make sure it
doesn't break anything, that's why we have a QA department (which is
peeved at me now but what else is new). Apparently Red Hat Software
thinks one of them (a QA department) is something that doesn't matter.
(Sorry if I sound annoyed, other people breaking my stuff annoys me). I
can roll things back to an earlier Centos, but it's going to be a major
effort, since I have to reinstall the OS on each of the servers one by
one. (Data lives elsewhere and configuration info is backed up there
nightly, that's not a problem, but we're still talking hours
reinstalling then restoring things from backups where I'd prefer to be
doing something that makes money for the company). Frankly, I'd prefer
to not do that, but if it's the only way to make things work,
downgrading to a insecure version of the OS with multiple known security
vulnerabilities that don't pass our corporate security policy, that's
what I'll do (I wrote the bloody policy, I'll just shut up the security
scanner when it yells and screams at me) . Or just switch to a real
Linux that doesn't break things when security fixes are released, if
such a thing exists, but that's a huge effort too, especially since my
configuration backups wouldn't just slide right in.
SIGH.
Ah well, off to work.
On 5/22/2019 7:56 AM, Andrija Panic wrote:
> Eric,
>
> did you actually test this in production?
>
> Andrija
>
> On Wed, 22 May 2019 at 16:33, Eric Lee Green <er...@gmail.com>
> wrote:
>
>> Okay. This makes sense.
>>
>> And people wonder why Amazon decided to make their own Linux rather than
>> use Centos and why Ubuntu has seized huge market share from Red Hat in
>> the past few years. SIGH.
>>
>> Downgrading my CentOS is not going to be easy. There were security
>> patches in the latest CentOS that I am required to have. It would
>> actually be easier to just switch to Ubuntu at this point since we're
>> talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software.
>> I hope IBM fixes them, but I have my doubts.
>>
>> On 5/22/2019 1:05 AM, Andrija Panic wrote:
>>> Hi Eric, all,
>>>
>>> I believe you might be hitting this one - issues on latest CentOS 7.6:
>>> https://github.com/apache/cloudstack/pull/3333 due to changes in the OS
>>> itself...
>>>
>>> If you believe that is the case, please try with CentOS 7.4 (can confirm
>>> works fine) and/or CentOS 7.5.
>>>
>>> Best,
>>> Andrija
>>>
>>>
>>>
>>> On Wed, 22 May 2019 at 09:04, Eric Lee Green <er...@gmail.com>
>>> wrote:
>>>
>>>> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
>>>>> Hello Eric,
>>>>>
>>>>> If the router version is displayed as UNKNOWN in the portal, it
>>>> would be
>>>>> a connectivity issue. Check your connections related to firewall rules
>>>>> between the ACP management hosts, hypervisor and VR. Is your VR
>>>> management
>>>>> network setup correctly?
>>>> Communications between the hosts is working fine and I have not changed
>>>> the VR management network between the running 4.9 and the not-running
>>>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
>>>> should properly forward VR traffic.
>>>>
>>>> More tomorrow when I have some sleep since it is now midnight here in
>>>> the SF Bay area.
>>>>
>>>>
>>>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <eric.lee.green@gmail.com
>>>>> wrote:
>>>>>
>>>>>> Thanks for the response, sorry if I sound frustrated, but this is
>>>>>> supposed to be a simple easy process and it's been horrible all the
>> way
>>>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now
>> 4.11.2
>>>>>> has failed to upgrade too. I've thus far spent around 16 hours of my
>>>>>> time on what should have taken an hour at most. I'm frustrated and
>>>> bummed.
>>>>>> [root@cloud2 ~]# rpm -q centos-release
>>>>>> centos-release-7-6.1810.2.el7.centos.x86_64
>>>>>> [root@cloud2 ~]# rpm -q libvirt
>>>>>> libvirt-4.5.0-10.el7_6.9.x86_64
>>>>>> [root@cloud1 ~]# rpm -qa | grep kvm
>>>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
>>>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
>>>>>> [root@cloud2 ~]# cat /proc/version
>>>>>> Linux version 3.10.0-862.6.3.el7.x86_64
>>>>>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red
>> Hat
>>>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
>>>>>> [root@cloud2 ~]# cat /proc/cpuinfo
>>>>>> ...
>>>>>> processor : 23
>>>>>> vendor_id : GenuineIntel
>>>>>> cpu family : 6
>>>>>> model : 44
>>>>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
>>>>>> stepping : 2
>>>>>> microcode : 0x1f
>>>>>> cpu MHz : 3068.000
>>>>>> cache size : 12288 KB
>>>>>> physical id : 1
>>>>>> siblings : 12
>>>>>> core id : 10
>>>>>> cpu cores : 6
>>>>>> apicid : 53
>>>>>> initial apicid : 53
>>>>>> fpu : yes
>>>>>> fpu_exception : yes
>>>>>> cpuid level : 11
>>>>>> wp : yes
>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
>> rep_good
>>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor
>> ds_cpl
>>>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
>>>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid
>> dtherm
>>>>>> ida arat
>>>>>> bogomips : 6133.21
>>>>>> clflush size : 64
>>>>>> cache_alignment : 64
>>>>>> address sizes : 40 bits physical, 48 bits virtual
>>>>>> [root@cloud1 ~]# free
>>>>>> total used free shared buff/cache
>>>>>> available
>>>>>> Mem: 123596388 79795792 34047696 632116 9752900
>>>> 39999332
>>>>>> Swap: 4194300 1956824 2237476
>>>>>> [root@cloud1 ~]# yum update
>>>>>>
>>>>>>
>> =========================================================================================================================================================================
>>>>>> Package Arch Version Repository Size
>>>>>>
>>>>>>
>> =========================================================================================================================================================================
>>>>>> Updating:
>>>>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
>>>>>> cloudstack411 47 M
>>>>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos
>>>>>> cloudstack411 58 M
>>>>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos
>>>>>> cloudstack411 82 M
>>>>>>
>>>>>> Three compute servers running the above processor averaging 128gb of
>>>>>> memory apiece, two data servers running NFS for primary and secondary
>>>>>> storage. Running the 4.11.2 community RPM's above.
>>>>>>
>>>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating. The
>>>>>> routers in fact started with the template, it said 4.11.2 when I
>> opened
>>>>>> up the console window and looked at the login prompt. But they never
>> got
>>>>>> configured, when I logged in with root/password at the console prompt
>>>>>> they had no networking set up and no configuration provided, the
>>>>>> cloud-init output in /var/log was essentially blank.
>>>>>>
>>>>>> Do I recall correctly that there is an issue with a particular version
>>>>>> of the hypervisor on the latest Centos 7? Any other information that
>> you
>>>>>> need? I think I provided the complete log file for the cloud-init of
>> the
>>>>>> router in another email...
>>>>>>
>>>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
>>>>>>> Hi Eric,
>>>>>>>
>>>>>>> Can you describe your environment in detail such as management server
>>>>>> host distro, hypervisor version, current CloudStack version, did you
>>>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
>>>>>>> Regards.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Rohit Yadav
>>>>>>>
>>>>>>> ________________________________
>>>>>>> From: Eric Lee Green <er...@gmail.com>
>>>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
>>>>>>> To: users@cloudstack.apache.org
>>>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>>>>>>>
>>>>>>> You may remember me as the person who had to roll back to Cloudstack
>>>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines
>>>> once
>>>>>>> I upgraded to it, claiming that there were inadequate resources even
>>>>>>> though I had over 150 gigabytes of memory free in my cluster and
>> oodles
>>>>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a
>>>> 512mb
>>>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
>>>>>>> *again* it's misbehaving.
>>>>>>>
>>>>>>> The symptom is that my virtual routers when I log into their console
>>>>>>> show 4.11.2 but when I look at them in the console they say 'Version:
>>>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or link
>>>>>>> local IP address it fails. And when I try to start up a virtual
>> machine
>>>>>>> that uses that virtual network, it says "Network unavailable', even
>>>>>>> though the router for that network is showing up and running.
>>>>>>>
>>>>>>> Clearly something's broken in the virtual routers but I don't know
>> what
>>>>>>> because I can't get into the router virtual machines. How do I get
>> the
>>>>>>> console password to get into the router virtual machines? It's
>>>> encrypted
>>>>>>> in the database (duh), how do I decrypt it?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> rohit.yadav@shapeblue.com
>>>>>>> www.shapeblue.com
>>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK
>>>>>>> @shapeblue
>>>>>>>
>>>>>>>
>>>>>>>
>
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Andrija Panic <an...@gmail.com>.
Eric,
did you actually test this in production?
Andrija
On Wed, 22 May 2019 at 16:33, Eric Lee Green <er...@gmail.com>
wrote:
> Okay. This makes sense.
>
> And people wonder why Amazon decided to make their own Linux rather than
> use Centos and why Ubuntu has seized huge market share from Red Hat in
> the past few years. SIGH.
>
> Downgrading my CentOS is not going to be easy. There were security
> patches in the latest CentOS that I am required to have. It would
> actually be easier to just switch to Ubuntu at this point since we're
> talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software.
> I hope IBM fixes them, but I have my doubts.
>
> On 5/22/2019 1:05 AM, Andrija Panic wrote:
> > Hi Eric, all,
> >
> > I believe you might be hitting this one - issues on latest CentOS 7.6:
> > https://github.com/apache/cloudstack/pull/3333 due to changes in the OS
> > itself...
> >
> > If you believe that is the case, please try with CentOS 7.4 (can confirm
> > works fine) and/or CentOS 7.5.
> >
> > Best,
> > Andrija
> >
> >
> >
> > On Wed, 22 May 2019 at 09:04, Eric Lee Green <er...@gmail.com>
> > wrote:
> >
> >> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
> >>> Hello Eric,
> >>>
> >>> If the router version is displayed as UNKNOWN in the portal, it
> >> would be
> >>> a connectivity issue. Check your connections related to firewall rules
> >>> between the ACP management hosts, hypervisor and VR. Is your VR
> >> management
> >>> network setup correctly?
> >> Communications between the hosts is working fine and I have not changed
> >> the VR management network between the running 4.9 and the not-running
> >> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
> >> should properly forward VR traffic.
> >>
> >> More tomorrow when I have some sleep since it is now midnight here in
> >> the SF Bay area.
> >>
> >>
> >>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <eric.lee.green@gmail.com
> >
> >>> wrote:
> >>>
> >>>> Thanks for the response, sorry if I sound frustrated, but this is
> >>>> supposed to be a simple easy process and it's been horrible all the
> way
> >>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now
> 4.11.2
> >>>> has failed to upgrade too. I've thus far spent around 16 hours of my
> >>>> time on what should have taken an hour at most. I'm frustrated and
> >> bummed.
> >>>> [root@cloud2 ~]# rpm -q centos-release
> >>>> centos-release-7-6.1810.2.el7.centos.x86_64
> >>>> [root@cloud2 ~]# rpm -q libvirt
> >>>> libvirt-4.5.0-10.el7_6.9.x86_64
> >>>> [root@cloud1 ~]# rpm -qa | grep kvm
> >>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
> >>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
> >>>> [root@cloud2 ~]# cat /proc/version
> >>>> Linux version 3.10.0-862.6.3.el7.x86_64
> >>>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red
> Hat
> >>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
> >>>> [root@cloud2 ~]# cat /proc/cpuinfo
> >>>> ...
> >>>> processor : 23
> >>>> vendor_id : GenuineIntel
> >>>> cpu family : 6
> >>>> model : 44
> >>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
> >>>> stepping : 2
> >>>> microcode : 0x1f
> >>>> cpu MHz : 3068.000
> >>>> cache size : 12288 KB
> >>>> physical id : 1
> >>>> siblings : 12
> >>>> core id : 10
> >>>> cpu cores : 6
> >>>> apicid : 53
> >>>> initial apicid : 53
> >>>> fpu : yes
> >>>> fpu_exception : yes
> >>>> cpuid level : 11
> >>>> wp : yes
> >>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> >>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> >>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
> rep_good
> >>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor
> ds_cpl
> >>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
> >>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid
> dtherm
> >>>> ida arat
> >>>> bogomips : 6133.21
> >>>> clflush size : 64
> >>>> cache_alignment : 64
> >>>> address sizes : 40 bits physical, 48 bits virtual
> >>>> [root@cloud1 ~]# free
> >>>> total used free shared buff/cache
> >>>> available
> >>>> Mem: 123596388 79795792 34047696 632116 9752900
> >> 39999332
> >>>> Swap: 4194300 1956824 2237476
> >>>> [root@cloud1 ~]# yum update
> >>>>
> >>>>
> >>
> =========================================================================================================================================================================
> >>>> Package Arch Version Repository Size
> >>>>
> >>>>
> >>
> =========================================================================================================================================================================
> >>>> Updating:
> >>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
> >>>> cloudstack411 47 M
> >>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos
> >>>> cloudstack411 58 M
> >>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos
> >>>> cloudstack411 82 M
> >>>>
> >>>> Three compute servers running the above processor averaging 128gb of
> >>>> memory apiece, two data servers running NFS for primary and secondary
> >>>> storage. Running the 4.11.2 community RPM's above.
> >>>>
> >>>> Yes, I did register the 4.11.2 systemvmtemplate before updating. The
> >>>> routers in fact started with the template, it said 4.11.2 when I
> opened
> >>>> up the console window and looked at the login prompt. But they never
> got
> >>>> configured, when I logged in with root/password at the console prompt
> >>>> they had no networking set up and no configuration provided, the
> >>>> cloud-init output in /var/log was essentially blank.
> >>>>
> >>>> Do I recall correctly that there is an issue with a particular version
> >>>> of the hypervisor on the latest Centos 7? Any other information that
> you
> >>>> need? I think I provided the complete log file for the cloud-init of
> the
> >>>> router in another email...
> >>>>
> >>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
> >>>>> Hi Eric,
> >>>>>
> >>>>> Can you describe your environment in detail such as management server
> >>>> host distro, hypervisor version, current CloudStack version, did you
> >>>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
> >>>>> Regards.
> >>>>>
> >>>>> Regards,
> >>>>> Rohit Yadav
> >>>>>
> >>>>> ________________________________
> >>>>> From: Eric Lee Green <er...@gmail.com>
> >>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
> >>>>> To: users@cloudstack.apache.org
> >>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> >>>>>
> >>>>> You may remember me as the person who had to roll back to Cloudstack
> >>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines
> >> once
> >>>>> I upgraded to it, claiming that there were inadequate resources even
> >>>>> though I had over 150 gigabytes of memory free in my cluster and
> oodles
> >>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a
> >> 512mb
> >>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
> >>>>> *again* it's misbehaving.
> >>>>>
> >>>>> The symptom is that my virtual routers when I log into their console
> >>>>> show 4.11.2 but when I look at them in the console they say 'Version:
> >>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or link
> >>>>> local IP address it fails. And when I try to start up a virtual
> machine
> >>>>> that uses that virtual network, it says "Network unavailable', even
> >>>>> though the router for that network is showing up and running.
> >>>>>
> >>>>> Clearly something's broken in the virtual routers but I don't know
> what
> >>>>> because I can't get into the router virtual machines. How do I get
> the
> >>>>> console password to get into the router virtual machines? It's
> >> encrypted
> >>>>> in the database (duh), how do I decrypt it?
> >>>>>
> >>>>>
> >>>>>
> >>>>> rohit.yadav@shapeblue.com
> >>>>> www.shapeblue.com
> >>>>> Amadeus House, Floral Street, London WC2E 9DPUK
> >>>>> @shapeblue
> >>>>>
> >>>>>
> >>>>>
> >
>
--
Andrija Panić
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Darius Kasparavičius <da...@gmail.com>.
Hi,
There is no need to get rid of qemu-ev packages. Just downgrade them
to 2.10 and it works fine.
On Wed, May 22, 2019 at 10:52 PM Andrija Panic <an...@gmail.com> wrote:
>
> Important to see you happy with 4.11.2 - the rest (HW) will follow ;)
>
> On Wed, 22 May 2019 at 21:49, Eric Lee Green <er...@gmail.com>
> wrote:
>
> > EXCELLENT!
> >
> > That did it. I downgraded to the libvirt and openssh rpms from Centos
> > 7.5, which was the latest DVD I had hanging around (likely the one I
> > used to install the systems with in the first place). Turns out it was
> > the Red Hat updates that came in when I did 'rpm update' that broke
> > things, not the Cloudstack upgrade. I am now happily running on
> > Cloudstack 4.11.2. Got rid of the qemu-kvm-ev stuff, btw, I had
> > installed that at one point trying to resolve the problems, but have now
> > downgraded it back to the release stuff.
> >
> > In case it is not obvious, I am the person who is serious about security
> > policies. The company itself doesn't care as long as we have one to show
> > to investors and customers.
> >
> > Regarding hardware, security policies don't cost a lot of money given
> > all the open source tools we have available now. New hardware does cost
> > money. Old commercial grade hardware is ridiculously cheap from Fleabay
> > right now as big corporations dump "obsolete" gear. I bought three used
> > commercial-grade 24-port 10 gigabit switches for the price of a single
> > new 1 gigabit switch, for example. I won't have cheap white box junk in
> > the back room, it's three racks of commercial-quality gear with dual
> > processors, ECC, 10gbit Ethernet, IPMI for remote management, etc.,
> > reliable as a brick -- just old and a bit power hungry. Power is the
> > biggest issue that will lead to an upgrade, not reliability or
> > performance -- I only have power budget for three racks, and new
> > hardware is much more dense for a specific power consumption (modern 16
> > core processors use about the same power as the 6 core processors I have
> > in my nodes). When I have the budget to upgrade the hardware, I will,
> > but what's there works, is reliable as a brick and not an issue, unlike
> > Red Hat breaking my infrastructure with incompatible updates GRR!
> >
> > On 5/22/19 9:20 AM, Andrija Panic wrote:
> > > +1 on what Nux said - simply downgrade the packages and check if that
> > works
> > > - no need to reinstall everything !
> > > As for the $3333 from github (CentOS 7.6 issue) - simple new line at the
> > > end of id_rsa file and/or downgrading the ssh-keygen will work.
> > >
> > > Don't want to annoy you, but if the company is so serious about security
> > > policies ask them to be serious about HW as well (regular white trash PC
> > > will do...so you can test software itself) - otherwise, you personally
> > > suffer the consequences and look bad (had the same pain many years
> > ago...).
> > >
> > > On Wed, 22 May 2019 at 17:44, Nux! <nu...@li.nux.ro> wrote:
> > >
> > >> Eric,
> > >>
> > >> I think you can get away with just downgrading to stock qemu-kvm
> > packages
> > >> (or patch your cloudstack installation).
> > >> BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages
> > >> don't get the same level of testing and QA.
> > >>
> > >> --
> > >> Sent from the Delta quadrant using Borg technology!
> > >>
> > >> Nux!
> > >> www.nux.ro
> > >>
> > >> ----- Original Message -----
> > >>> From: "Eric Lee Green" <er...@gmail.com>
> > >>> To: "users" <us...@cloudstack.apache.org>
> > >>> Sent: Wednesday, 22 May, 2019 15:33:15
> > >>> Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> > >>> Okay. This makes sense.
> > >>>
> > >>> And people wonder why Amazon decided to make their own Linux rather
> > than
> > >>> use Centos and why Ubuntu has seized huge market share from Red Hat in
> > >>> the past few years. SIGH.
> > >>>
> > >>> Downgrading my CentOS is not going to be easy. There were security
> > >>> patches in the latest CentOS that I am required to have. It would
> > >>> actually be easier to just switch to Ubuntu at this point since we're
> > >>> talking a complete re-install anyhow. SIGH. Thank you, Red Hat
> > Software.
> > >>> I hope IBM fixes them, but I have my doubts.
> > >>>
> > >>> On 5/22/2019 1:05 AM, Andrija Panic wrote:
> > >>>> Hi Eric, all,
> > >>>>
> > >>>> I believe you might be hitting this one - issues on latest CentOS 7.6:
> > >>>> https://github.com/apache/cloudstack/pull/3333 due to changes in the
> > OS
> > >>>> itself...
> > >>>>
> > >>>> If you believe that is the case, please try with CentOS 7.4 (can
> > confirm
> > >>>> works fine) and/or CentOS 7.5.
> > >>>>
> > >>>> Best,
> > >>>> Andrija
> > >>>>
> > >>>>
> > >>>>
> > >>>> On Wed, 22 May 2019 at 09:04, Eric Lee Green <
> > eric.lee.green@gmail.com>
> > >>>> wrote:
> > >>>>
> > >>>>> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
> > >>>>>> Hello Eric,
> > >>>>>>
> > >>>>>> If the router version is displayed as UNKNOWN in the portal,
> > it
> > >>>>> would be
> > >>>>>> a connectivity issue. Check your connections related to firewall
> > rules
> > >>>>>> between the ACP management hosts, hypervisor and VR. Is your VR
> > >>>>> management
> > >>>>>> network setup correctly?
> > >>>>> Communications between the hosts is working fine and I have not
> > changed
> > >>>>> the VR management network between the running 4.9 and the not-running
> > >>>>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack
> > and
> > >>>>> should properly forward VR traffic.
> > >>>>>
> > >>>>> More tomorrow when I have some sleep since it is now midnight here in
> > >>>>> the SF Bay area.
> > >>>>>
> > >>>>>
> > >>>>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <
> > >> eric.lee.green@gmail.com>
> > >>>>>> wrote:
> > >>>>>>
> > >>>>>>> Thanks for the response, sorry if I sound frustrated, but this is
> > >>>>>>> supposed to be a simple easy process and it's been horrible all the
> > >> way
> > >>>>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now
> > >> 4.11.2
> > >>>>>>> has failed to upgrade too. I've thus far spent around 16 hours of
> > my
> > >>>>>>> time on what should have taken an hour at most. I'm frustrated and
> > >>>>> bummed.
> > >>>>>>> [root@cloud2 ~]# rpm -q centos-release
> > >>>>>>> centos-release-7-6.1810.2.el7.centos.x86_64
> > >>>>>>> [root@cloud2 ~]# rpm -q libvirt
> > >>>>>>> libvirt-4.5.0-10.el7_6.9.x86_64
> > >>>>>>> [root@cloud1 ~]# rpm -qa | grep kvm
> > >>>>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
> > >>>>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
> > >>>>>>> [root@cloud2 ~]# cat /proc/version
> > >>>>>>> Linux version 3.10.0-862.6.3.el7.x86_64
> > >>>>>>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red
> > >> Hat
> > >>>>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
> > >>>>>>> [root@cloud2 ~]# cat /proc/cpuinfo
> > >>>>>>> ...
> > >>>>>>> processor : 23
> > >>>>>>> vendor_id : GenuineIntel
> > >>>>>>> cpu family : 6
> > >>>>>>> model : 44
> > >>>>>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
> > >>>>>>> stepping : 2
> > >>>>>>> microcode : 0x1f
> > >>>>>>> cpu MHz : 3068.000
> > >>>>>>> cache size : 12288 KB
> > >>>>>>> physical id : 1
> > >>>>>>> siblings : 12
> > >>>>>>> core id : 10
> > >>>>>>> cpu cores : 6
> > >>>>>>> apicid : 53
> > >>>>>>> initial apicid : 53
> > >>>>>>> fpu : yes
> > >>>>>>> fpu_exception : yes
> > >>>>>>> cpuid level : 11
> > >>>>>>> wp : yes
> > >>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> > >> pge
> > >>>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> > >>>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
> > >> rep_good
> > >>>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor
> > >> ds_cpl
> > >>>>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
> > >>>>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid
> > >> dtherm
> > >>>>>>> ida arat
> > >>>>>>> bogomips : 6133.21
> > >>>>>>> clflush size : 64
> > >>>>>>> cache_alignment : 64
> > >>>>>>> address sizes : 40 bits physical, 48 bits virtual
> > >>>>>>> [root@cloud1 ~]# free
> > >>>>>>> total used free shared
> > buff/cache
> > >>>>>>> available
> > >>>>>>> Mem: 123596388 79795792 34047696 632116 9752900
> > >>>>> 39999332
> > >>>>>>> Swap: 4194300 1956824 2237476
> > >>>>>>> [root@cloud1 ~]# yum update
> > >>>>>>>
> > >>>>>>>
> > >>
> > =========================================================================================================================================================================
> > >>>>>>> Package Arch Version Repository
> > Size
> > >>>>>>>
> > >>>>>>>
> > >>
> > =========================================================================================================================================================================
> > >>>>>>> Updating:
> > >>>>>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
> > >>>>>>> cloudstack411 47 M
> > >>>>>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos
> > >>>>>>> cloudstack411 58 M
> > >>>>>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos
> > >>>>>>> cloudstack411 82 M
> > >>>>>>>
> > >>>>>>> Three compute servers running the above processor averaging 128gb
> > of
> > >>>>>>> memory apiece, two data servers running NFS for primary and
> > secondary
> > >>>>>>> storage. Running the 4.11.2 community RPM's above.
> > >>>>>>>
> > >>>>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating.
> > The
> > >>>>>>> routers in fact started with the template, it said 4.11.2 when I
> > >> opened
> > >>>>>>> up the console window and looked at the login prompt. But they
> > never
> > >> got
> > >>>>>>> configured, when I logged in with root/password at the console
> > prompt
> > >>>>>>> they had no networking set up and no configuration provided, the
> > >>>>>>> cloud-init output in /var/log was essentially blank.
> > >>>>>>>
> > >>>>>>> Do I recall correctly that there is an issue with a particular
> > >> version
> > >>>>>>> of the hypervisor on the latest Centos 7? Any other information
> > that
> > >> you
> > >>>>>>> need? I think I provided the complete log file for the cloud-init
> > of
> > >> the
> > >>>>>>> router in another email...
> > >>>>>>>
> > >>>>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
> > >>>>>>>> Hi Eric,
> > >>>>>>>>
> > >>>>>>>> Can you describe your environment in detail such as management
> > >> server
> > >>>>>>> host distro, hypervisor version, current CloudStack version, did
> > you
> > >>>>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
> > >>>>>>>> Regards.
> > >>>>>>>>
> > >>>>>>>> Regards,
> > >>>>>>>> Rohit Yadav
> > >>>>>>>>
> > >>>>>>>> ________________________________
> > >>>>>>>> From: Eric Lee Green <er...@gmail.com>
> > >>>>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
> > >>>>>>>> To: users@cloudstack.apache.org
> > >>>>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> > >>>>>>>>
> > >>>>>>>> You may remember me as the person who had to roll back to
> > Cloudstack
> > >>>>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual
> > machines
> > >>>>> once
> > >>>>>>>> I upgraded to it, claiming that there were inadequate resources
> > even
> > >>>>>>>> though I had over 150 gigabytes of memory free in my cluster and
> > >> oodles
> > >>>>>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a
> > >>>>> 512mb
> > >>>>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
> > >>>>>>>> *again* it's misbehaving.
> > >>>>>>>>
> > >>>>>>>> The symptom is that my virtual routers when I log into their
> > console
> > >>>>>>>> show 4.11.2 but when I look at them in the console they say
> > >> 'Version:
> > >>>>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or
> > link
> > >>>>>>>> local IP address it fails. And when I try to start up a virtual
> > >> machine
> > >>>>>>>> that uses that virtual network, it says "Network unavailable',
> > even
> > >>>>>>>> though the router for that network is showing up and running.
> > >>>>>>>>
> > >>>>>>>> Clearly something's broken in the virtual routers but I don't know
> > >> what
> > >>>>>>>> because I can't get into the router virtual machines. How do I get
> > >> the
> > >>>>>>>> console password to get into the router virtual machines? It's
> > >>>>> encrypted
> > >>>>>>>> in the database (duh), how do I decrypt it?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> rohit.yadav@shapeblue.com
> > >>>>>>>> www.shapeblue.com
> > >>>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK
> > >>>>>>>> @shapeblue
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >
> >
> >
>
> --
>
> Andrija Panić
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Andrija Panic <an...@gmail.com>.
Important to see you happy with 4.11.2 - the rest (HW) will follow ;)
On Wed, 22 May 2019 at 21:49, Eric Lee Green <er...@gmail.com>
wrote:
> EXCELLENT!
>
> That did it. I downgraded to the libvirt and openssh rpms from Centos
> 7.5, which was the latest DVD I had hanging around (likely the one I
> used to install the systems with in the first place). Turns out it was
> the Red Hat updates that came in when I did 'rpm update' that broke
> things, not the Cloudstack upgrade. I am now happily running on
> Cloudstack 4.11.2. Got rid of the qemu-kvm-ev stuff, btw, I had
> installed that at one point trying to resolve the problems, but have now
> downgraded it back to the release stuff.
>
> In case it is not obvious, I am the person who is serious about security
> policies. The company itself doesn't care as long as we have one to show
> to investors and customers.
>
> Regarding hardware, security policies don't cost a lot of money given
> all the open source tools we have available now. New hardware does cost
> money. Old commercial grade hardware is ridiculously cheap from Fleabay
> right now as big corporations dump "obsolete" gear. I bought three used
> commercial-grade 24-port 10 gigabit switches for the price of a single
> new 1 gigabit switch, for example. I won't have cheap white box junk in
> the back room, it's three racks of commercial-quality gear with dual
> processors, ECC, 10gbit Ethernet, IPMI for remote management, etc.,
> reliable as a brick -- just old and a bit power hungry. Power is the
> biggest issue that will lead to an upgrade, not reliability or
> performance -- I only have power budget for three racks, and new
> hardware is much more dense for a specific power consumption (modern 16
> core processors use about the same power as the 6 core processors I have
> in my nodes). When I have the budget to upgrade the hardware, I will,
> but what's there works, is reliable as a brick and not an issue, unlike
> Red Hat breaking my infrastructure with incompatible updates GRR!
>
> On 5/22/19 9:20 AM, Andrija Panic wrote:
> > +1 on what Nux said - simply downgrade the packages and check if that
> works
> > - no need to reinstall everything !
> > As for the $3333 from github (CentOS 7.6 issue) - simple new line at the
> > end of id_rsa file and/or downgrading the ssh-keygen will work.
> >
> > Don't want to annoy you, but if the company is so serious about security
> > policies ask them to be serious about HW as well (regular white trash PC
> > will do...so you can test software itself) - otherwise, you personally
> > suffer the consequences and look bad (had the same pain many years
> ago...).
> >
> > On Wed, 22 May 2019 at 17:44, Nux! <nu...@li.nux.ro> wrote:
> >
> >> Eric,
> >>
> >> I think you can get away with just downgrading to stock qemu-kvm
> packages
> >> (or patch your cloudstack installation).
> >> BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages
> >> don't get the same level of testing and QA.
> >>
> >> --
> >> Sent from the Delta quadrant using Borg technology!
> >>
> >> Nux!
> >> www.nux.ro
> >>
> >> ----- Original Message -----
> >>> From: "Eric Lee Green" <er...@gmail.com>
> >>> To: "users" <us...@cloudstack.apache.org>
> >>> Sent: Wednesday, 22 May, 2019 15:33:15
> >>> Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> >>> Okay. This makes sense.
> >>>
> >>> And people wonder why Amazon decided to make their own Linux rather
> than
> >>> use Centos and why Ubuntu has seized huge market share from Red Hat in
> >>> the past few years. SIGH.
> >>>
> >>> Downgrading my CentOS is not going to be easy. There were security
> >>> patches in the latest CentOS that I am required to have. It would
> >>> actually be easier to just switch to Ubuntu at this point since we're
> >>> talking a complete re-install anyhow. SIGH. Thank you, Red Hat
> Software.
> >>> I hope IBM fixes them, but I have my doubts.
> >>>
> >>> On 5/22/2019 1:05 AM, Andrija Panic wrote:
> >>>> Hi Eric, all,
> >>>>
> >>>> I believe you might be hitting this one - issues on latest CentOS 7.6:
> >>>> https://github.com/apache/cloudstack/pull/3333 due to changes in the
> OS
> >>>> itself...
> >>>>
> >>>> If you believe that is the case, please try with CentOS 7.4 (can
> confirm
> >>>> works fine) and/or CentOS 7.5.
> >>>>
> >>>> Best,
> >>>> Andrija
> >>>>
> >>>>
> >>>>
> >>>> On Wed, 22 May 2019 at 09:04, Eric Lee Green <
> eric.lee.green@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
> >>>>>> Hello Eric,
> >>>>>>
> >>>>>> If the router version is displayed as UNKNOWN in the portal,
> it
> >>>>> would be
> >>>>>> a connectivity issue. Check your connections related to firewall
> rules
> >>>>>> between the ACP management hosts, hypervisor and VR. Is your VR
> >>>>> management
> >>>>>> network setup correctly?
> >>>>> Communications between the hosts is working fine and I have not
> changed
> >>>>> the VR management network between the running 4.9 and the not-running
> >>>>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack
> and
> >>>>> should properly forward VR traffic.
> >>>>>
> >>>>> More tomorrow when I have some sleep since it is now midnight here in
> >>>>> the SF Bay area.
> >>>>>
> >>>>>
> >>>>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <
> >> eric.lee.green@gmail.com>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Thanks for the response, sorry if I sound frustrated, but this is
> >>>>>>> supposed to be a simple easy process and it's been horrible all the
> >> way
> >>>>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now
> >> 4.11.2
> >>>>>>> has failed to upgrade too. I've thus far spent around 16 hours of
> my
> >>>>>>> time on what should have taken an hour at most. I'm frustrated and
> >>>>> bummed.
> >>>>>>> [root@cloud2 ~]# rpm -q centos-release
> >>>>>>> centos-release-7-6.1810.2.el7.centos.x86_64
> >>>>>>> [root@cloud2 ~]# rpm -q libvirt
> >>>>>>> libvirt-4.5.0-10.el7_6.9.x86_64
> >>>>>>> [root@cloud1 ~]# rpm -qa | grep kvm
> >>>>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
> >>>>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
> >>>>>>> [root@cloud2 ~]# cat /proc/version
> >>>>>>> Linux version 3.10.0-862.6.3.el7.x86_64
> >>>>>>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red
> >> Hat
> >>>>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
> >>>>>>> [root@cloud2 ~]# cat /proc/cpuinfo
> >>>>>>> ...
> >>>>>>> processor : 23
> >>>>>>> vendor_id : GenuineIntel
> >>>>>>> cpu family : 6
> >>>>>>> model : 44
> >>>>>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
> >>>>>>> stepping : 2
> >>>>>>> microcode : 0x1f
> >>>>>>> cpu MHz : 3068.000
> >>>>>>> cache size : 12288 KB
> >>>>>>> physical id : 1
> >>>>>>> siblings : 12
> >>>>>>> core id : 10
> >>>>>>> cpu cores : 6
> >>>>>>> apicid : 53
> >>>>>>> initial apicid : 53
> >>>>>>> fpu : yes
> >>>>>>> fpu_exception : yes
> >>>>>>> cpuid level : 11
> >>>>>>> wp : yes
> >>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> >> pge
> >>>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> >>>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
> >> rep_good
> >>>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor
> >> ds_cpl
> >>>>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
> >>>>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid
> >> dtherm
> >>>>>>> ida arat
> >>>>>>> bogomips : 6133.21
> >>>>>>> clflush size : 64
> >>>>>>> cache_alignment : 64
> >>>>>>> address sizes : 40 bits physical, 48 bits virtual
> >>>>>>> [root@cloud1 ~]# free
> >>>>>>> total used free shared
> buff/cache
> >>>>>>> available
> >>>>>>> Mem: 123596388 79795792 34047696 632116 9752900
> >>>>> 39999332
> >>>>>>> Swap: 4194300 1956824 2237476
> >>>>>>> [root@cloud1 ~]# yum update
> >>>>>>>
> >>>>>>>
> >>
> =========================================================================================================================================================================
> >>>>>>> Package Arch Version Repository
> Size
> >>>>>>>
> >>>>>>>
> >>
> =========================================================================================================================================================================
> >>>>>>> Updating:
> >>>>>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
> >>>>>>> cloudstack411 47 M
> >>>>>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos
> >>>>>>> cloudstack411 58 M
> >>>>>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos
> >>>>>>> cloudstack411 82 M
> >>>>>>>
> >>>>>>> Three compute servers running the above processor averaging 128gb
> of
> >>>>>>> memory apiece, two data servers running NFS for primary and
> secondary
> >>>>>>> storage. Running the 4.11.2 community RPM's above.
> >>>>>>>
> >>>>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating.
> The
> >>>>>>> routers in fact started with the template, it said 4.11.2 when I
> >> opened
> >>>>>>> up the console window and looked at the login prompt. But they
> never
> >> got
> >>>>>>> configured, when I logged in with root/password at the console
> prompt
> >>>>>>> they had no networking set up and no configuration provided, the
> >>>>>>> cloud-init output in /var/log was essentially blank.
> >>>>>>>
> >>>>>>> Do I recall correctly that there is an issue with a particular
> >> version
> >>>>>>> of the hypervisor on the latest Centos 7? Any other information
> that
> >> you
> >>>>>>> need? I think I provided the complete log file for the cloud-init
> of
> >> the
> >>>>>>> router in another email...
> >>>>>>>
> >>>>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
> >>>>>>>> Hi Eric,
> >>>>>>>>
> >>>>>>>> Can you describe your environment in detail such as management
> >> server
> >>>>>>> host distro, hypervisor version, current CloudStack version, did
> you
> >>>>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
> >>>>>>>> Regards.
> >>>>>>>>
> >>>>>>>> Regards,
> >>>>>>>> Rohit Yadav
> >>>>>>>>
> >>>>>>>> ________________________________
> >>>>>>>> From: Eric Lee Green <er...@gmail.com>
> >>>>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
> >>>>>>>> To: users@cloudstack.apache.org
> >>>>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> >>>>>>>>
> >>>>>>>> You may remember me as the person who had to roll back to
> Cloudstack
> >>>>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual
> machines
> >>>>> once
> >>>>>>>> I upgraded to it, claiming that there were inadequate resources
> even
> >>>>>>>> though I had over 150 gigabytes of memory free in my cluster and
> >> oodles
> >>>>>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a
> >>>>> 512mb
> >>>>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
> >>>>>>>> *again* it's misbehaving.
> >>>>>>>>
> >>>>>>>> The symptom is that my virtual routers when I log into their
> console
> >>>>>>>> show 4.11.2 but when I look at them in the console they say
> >> 'Version:
> >>>>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or
> link
> >>>>>>>> local IP address it fails. And when I try to start up a virtual
> >> machine
> >>>>>>>> that uses that virtual network, it says "Network unavailable',
> even
> >>>>>>>> though the router for that network is showing up and running.
> >>>>>>>>
> >>>>>>>> Clearly something's broken in the virtual routers but I don't know
> >> what
> >>>>>>>> because I can't get into the router virtual machines. How do I get
> >> the
> >>>>>>>> console password to get into the router virtual machines? It's
> >>>>> encrypted
> >>>>>>>> in the database (duh), how do I decrypt it?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> rohit.yadav@shapeblue.com
> >>>>>>>> www.shapeblue.com
> >>>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK
> >>>>>>>> @shapeblue
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >
>
>
--
Andrija Panić
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Eric Lee Green <er...@gmail.com>.
EXCELLENT!
That did it. I downgraded to the libvirt and openssh rpms from Centos
7.5, which was the latest DVD I had hanging around (likely the one I
used to install the systems with in the first place). Turns out it was
the Red Hat updates that came in when I did 'rpm update' that broke
things, not the Cloudstack upgrade. I am now happily running on
Cloudstack 4.11.2. Got rid of the qemu-kvm-ev stuff, btw, I had
installed that at one point trying to resolve the problems, but have now
downgraded it back to the release stuff.
In case it is not obvious, I am the person who is serious about security
policies. The company itself doesn't care as long as we have one to show
to investors and customers.
Regarding hardware, security policies don't cost a lot of money given
all the open source tools we have available now. New hardware does cost
money. Old commercial grade hardware is ridiculously cheap from Fleabay
right now as big corporations dump "obsolete" gear. I bought three used
commercial-grade 24-port 10 gigabit switches for the price of a single
new 1 gigabit switch, for example. I won't have cheap white box junk in
the back room, it's three racks of commercial-quality gear with dual
processors, ECC, 10gbit Ethernet, IPMI for remote management, etc.,
reliable as a brick -- just old and a bit power hungry. Power is the
biggest issue that will lead to an upgrade, not reliability or
performance -- I only have power budget for three racks, and new
hardware is much more dense for a specific power consumption (modern 16
core processors use about the same power as the 6 core processors I have
in my nodes). When I have the budget to upgrade the hardware, I will,
but what's there works, is reliable as a brick and not an issue, unlike
Red Hat breaking my infrastructure with incompatible updates GRR!
On 5/22/19 9:20 AM, Andrija Panic wrote:
> +1 on what Nux said - simply downgrade the packages and check if that works
> - no need to reinstall everything !
> As for the $3333 from github (CentOS 7.6 issue) - simple new line at the
> end of id_rsa file and/or downgrading the ssh-keygen will work.
>
> Don't want to annoy you, but if the company is so serious about security
> policies ask them to be serious about HW as well (regular white trash PC
> will do...so you can test software itself) - otherwise, you personally
> suffer the consequences and look bad (had the same pain many years ago...).
>
> On Wed, 22 May 2019 at 17:44, Nux! <nu...@li.nux.ro> wrote:
>
>> Eric,
>>
>> I think you can get away with just downgrading to stock qemu-kvm packages
>> (or patch your cloudstack installation).
>> BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages
>> don't get the same level of testing and QA.
>>
>> --
>> Sent from the Delta quadrant using Borg technology!
>>
>> Nux!
>> www.nux.ro
>>
>> ----- Original Message -----
>>> From: "Eric Lee Green" <er...@gmail.com>
>>> To: "users" <us...@cloudstack.apache.org>
>>> Sent: Wednesday, 22 May, 2019 15:33:15
>>> Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>>> Okay. This makes sense.
>>>
>>> And people wonder why Amazon decided to make their own Linux rather than
>>> use Centos and why Ubuntu has seized huge market share from Red Hat in
>>> the past few years. SIGH.
>>>
>>> Downgrading my CentOS is not going to be easy. There were security
>>> patches in the latest CentOS that I am required to have. It would
>>> actually be easier to just switch to Ubuntu at this point since we're
>>> talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software.
>>> I hope IBM fixes them, but I have my doubts.
>>>
>>> On 5/22/2019 1:05 AM, Andrija Panic wrote:
>>>> Hi Eric, all,
>>>>
>>>> I believe you might be hitting this one - issues on latest CentOS 7.6:
>>>> https://github.com/apache/cloudstack/pull/3333 due to changes in the OS
>>>> itself...
>>>>
>>>> If you believe that is the case, please try with CentOS 7.4 (can confirm
>>>> works fine) and/or CentOS 7.5.
>>>>
>>>> Best,
>>>> Andrija
>>>>
>>>>
>>>>
>>>> On Wed, 22 May 2019 at 09:04, Eric Lee Green <er...@gmail.com>
>>>> wrote:
>>>>
>>>>> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
>>>>>> Hello Eric,
>>>>>>
>>>>>> If the router version is displayed as UNKNOWN in the portal, it
>>>>> would be
>>>>>> a connectivity issue. Check your connections related to firewall rules
>>>>>> between the ACP management hosts, hypervisor and VR. Is your VR
>>>>> management
>>>>>> network setup correctly?
>>>>> Communications between the hosts is working fine and I have not changed
>>>>> the VR management network between the running 4.9 and the not-running
>>>>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
>>>>> should properly forward VR traffic.
>>>>>
>>>>> More tomorrow when I have some sleep since it is now midnight here in
>>>>> the SF Bay area.
>>>>>
>>>>>
>>>>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <
>> eric.lee.green@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thanks for the response, sorry if I sound frustrated, but this is
>>>>>>> supposed to be a simple easy process and it's been horrible all the
>> way
>>>>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now
>> 4.11.2
>>>>>>> has failed to upgrade too. I've thus far spent around 16 hours of my
>>>>>>> time on what should have taken an hour at most. I'm frustrated and
>>>>> bummed.
>>>>>>> [root@cloud2 ~]# rpm -q centos-release
>>>>>>> centos-release-7-6.1810.2.el7.centos.x86_64
>>>>>>> [root@cloud2 ~]# rpm -q libvirt
>>>>>>> libvirt-4.5.0-10.el7_6.9.x86_64
>>>>>>> [root@cloud1 ~]# rpm -qa | grep kvm
>>>>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
>>>>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
>>>>>>> [root@cloud2 ~]# cat /proc/version
>>>>>>> Linux version 3.10.0-862.6.3.el7.x86_64
>>>>>>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red
>> Hat
>>>>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
>>>>>>> [root@cloud2 ~]# cat /proc/cpuinfo
>>>>>>> ...
>>>>>>> processor : 23
>>>>>>> vendor_id : GenuineIntel
>>>>>>> cpu family : 6
>>>>>>> model : 44
>>>>>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
>>>>>>> stepping : 2
>>>>>>> microcode : 0x1f
>>>>>>> cpu MHz : 3068.000
>>>>>>> cache size : 12288 KB
>>>>>>> physical id : 1
>>>>>>> siblings : 12
>>>>>>> core id : 10
>>>>>>> cpu cores : 6
>>>>>>> apicid : 53
>>>>>>> initial apicid : 53
>>>>>>> fpu : yes
>>>>>>> fpu_exception : yes
>>>>>>> cpuid level : 11
>>>>>>> wp : yes
>>>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
>> pge
>>>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>>>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
>> rep_good
>>>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor
>> ds_cpl
>>>>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
>>>>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid
>> dtherm
>>>>>>> ida arat
>>>>>>> bogomips : 6133.21
>>>>>>> clflush size : 64
>>>>>>> cache_alignment : 64
>>>>>>> address sizes : 40 bits physical, 48 bits virtual
>>>>>>> [root@cloud1 ~]# free
>>>>>>> total used free shared buff/cache
>>>>>>> available
>>>>>>> Mem: 123596388 79795792 34047696 632116 9752900
>>>>> 39999332
>>>>>>> Swap: 4194300 1956824 2237476
>>>>>>> [root@cloud1 ~]# yum update
>>>>>>>
>>>>>>>
>> =========================================================================================================================================================================
>>>>>>> Package Arch Version Repository Size
>>>>>>>
>>>>>>>
>> =========================================================================================================================================================================
>>>>>>> Updating:
>>>>>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
>>>>>>> cloudstack411 47 M
>>>>>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos
>>>>>>> cloudstack411 58 M
>>>>>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos
>>>>>>> cloudstack411 82 M
>>>>>>>
>>>>>>> Three compute servers running the above processor averaging 128gb of
>>>>>>> memory apiece, two data servers running NFS for primary and secondary
>>>>>>> storage. Running the 4.11.2 community RPM's above.
>>>>>>>
>>>>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating. The
>>>>>>> routers in fact started with the template, it said 4.11.2 when I
>> opened
>>>>>>> up the console window and looked at the login prompt. But they never
>> got
>>>>>>> configured, when I logged in with root/password at the console prompt
>>>>>>> they had no networking set up and no configuration provided, the
>>>>>>> cloud-init output in /var/log was essentially blank.
>>>>>>>
>>>>>>> Do I recall correctly that there is an issue with a particular
>> version
>>>>>>> of the hypervisor on the latest Centos 7? Any other information that
>> you
>>>>>>> need? I think I provided the complete log file for the cloud-init of
>> the
>>>>>>> router in another email...
>>>>>>>
>>>>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
>>>>>>>> Hi Eric,
>>>>>>>>
>>>>>>>> Can you describe your environment in detail such as management
>> server
>>>>>>> host distro, hypervisor version, current CloudStack version, did you
>>>>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
>>>>>>>> Regards.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Rohit Yadav
>>>>>>>>
>>>>>>>> ________________________________
>>>>>>>> From: Eric Lee Green <er...@gmail.com>
>>>>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
>>>>>>>> To: users@cloudstack.apache.org
>>>>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>>>>>>>>
>>>>>>>> You may remember me as the person who had to roll back to Cloudstack
>>>>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines
>>>>> once
>>>>>>>> I upgraded to it, claiming that there were inadequate resources even
>>>>>>>> though I had over 150 gigabytes of memory free in my cluster and
>> oodles
>>>>>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a
>>>>> 512mb
>>>>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
>>>>>>>> *again* it's misbehaving.
>>>>>>>>
>>>>>>>> The symptom is that my virtual routers when I log into their console
>>>>>>>> show 4.11.2 but when I look at them in the console they say
>> 'Version:
>>>>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or link
>>>>>>>> local IP address it fails. And when I try to start up a virtual
>> machine
>>>>>>>> that uses that virtual network, it says "Network unavailable', even
>>>>>>>> though the router for that network is showing up and running.
>>>>>>>>
>>>>>>>> Clearly something's broken in the virtual routers but I don't know
>> what
>>>>>>>> because I can't get into the router virtual machines. How do I get
>> the
>>>>>>>> console password to get into the router virtual machines? It's
>>>>> encrypted
>>>>>>>> in the database (duh), how do I decrypt it?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> rohit.yadav@shapeblue.com
>>>>>>>> www.shapeblue.com
>>>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK
>>>>>>>> @shapeblue
>>>>>>>>
>>>>>>>>
>>>>>>>>
>
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Andrija Panic <an...@gmail.com>.
+1 on what Nux said - simply downgrade the packages and check if that works
- no need to reinstall everything !
As for the $3333 from github (CentOS 7.6 issue) - simple new line at the
end of id_rsa file and/or downgrading the ssh-keygen will work.
Don't want to annoy you, but if the company is so serious about security
policies ask them to be serious about HW as well (regular white trash PC
will do...so you can test software itself) - otherwise, you personally
suffer the consequences and look bad (had the same pain many years ago...).
On Wed, 22 May 2019 at 17:44, Nux! <nu...@li.nux.ro> wrote:
> Eric,
>
> I think you can get away with just downgrading to stock qemu-kvm packages
> (or patch your cloudstack installation).
> BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages
> don't get the same level of testing and QA.
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Eric Lee Green" <er...@gmail.com>
> > To: "users" <us...@cloudstack.apache.org>
> > Sent: Wednesday, 22 May, 2019 15:33:15
> > Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>
> > Okay. This makes sense.
> >
> > And people wonder why Amazon decided to make their own Linux rather than
> > use Centos and why Ubuntu has seized huge market share from Red Hat in
> > the past few years. SIGH.
> >
> > Downgrading my CentOS is not going to be easy. There were security
> > patches in the latest CentOS that I am required to have. It would
> > actually be easier to just switch to Ubuntu at this point since we're
> > talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software.
> > I hope IBM fixes them, but I have my doubts.
> >
> > On 5/22/2019 1:05 AM, Andrija Panic wrote:
> >> Hi Eric, all,
> >>
> >> I believe you might be hitting this one - issues on latest CentOS 7.6:
> >> https://github.com/apache/cloudstack/pull/3333 due to changes in the OS
> >> itself...
> >>
> >> If you believe that is the case, please try with CentOS 7.4 (can confirm
> >> works fine) and/or CentOS 7.5.
> >>
> >> Best,
> >> Andrija
> >>
> >>
> >>
> >> On Wed, 22 May 2019 at 09:04, Eric Lee Green <er...@gmail.com>
> >> wrote:
> >>
> >>> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
> >>>> Hello Eric,
> >>>>
> >>>> If the router version is displayed as UNKNOWN in the portal, it
> >>> would be
> >>>> a connectivity issue. Check your connections related to firewall rules
> >>>> between the ACP management hosts, hypervisor and VR. Is your VR
> >>> management
> >>>> network setup correctly?
> >>> Communications between the hosts is working fine and I have not changed
> >>> the VR management network between the running 4.9 and the not-running
> >>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
> >>> should properly forward VR traffic.
> >>>
> >>> More tomorrow when I have some sleep since it is now midnight here in
> >>> the SF Bay area.
> >>>
> >>>
> >>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <
> eric.lee.green@gmail.com>
> >>>> wrote:
> >>>>
> >>>>> Thanks for the response, sorry if I sound frustrated, but this is
> >>>>> supposed to be a simple easy process and it's been horrible all the
> way
> >>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now
> 4.11.2
> >>>>> has failed to upgrade too. I've thus far spent around 16 hours of my
> >>>>> time on what should have taken an hour at most. I'm frustrated and
> >>> bummed.
> >>>>> [root@cloud2 ~]# rpm -q centos-release
> >>>>> centos-release-7-6.1810.2.el7.centos.x86_64
> >>>>> [root@cloud2 ~]# rpm -q libvirt
> >>>>> libvirt-4.5.0-10.el7_6.9.x86_64
> >>>>> [root@cloud1 ~]# rpm -qa | grep kvm
> >>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
> >>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
> >>>>> [root@cloud2 ~]# cat /proc/version
> >>>>> Linux version 3.10.0-862.6.3.el7.x86_64
> >>>>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red
> Hat
> >>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
> >>>>> [root@cloud2 ~]# cat /proc/cpuinfo
> >>>>> ...
> >>>>> processor : 23
> >>>>> vendor_id : GenuineIntel
> >>>>> cpu family : 6
> >>>>> model : 44
> >>>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
> >>>>> stepping : 2
> >>>>> microcode : 0x1f
> >>>>> cpu MHz : 3068.000
> >>>>> cache size : 12288 KB
> >>>>> physical id : 1
> >>>>> siblings : 12
> >>>>> core id : 10
> >>>>> cpu cores : 6
> >>>>> apicid : 53
> >>>>> initial apicid : 53
> >>>>> fpu : yes
> >>>>> fpu_exception : yes
> >>>>> cpuid level : 11
> >>>>> wp : yes
> >>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr
> pge
> >>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> >>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts
> rep_good
> >>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor
> ds_cpl
> >>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
> >>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid
> dtherm
> >>>>> ida arat
> >>>>> bogomips : 6133.21
> >>>>> clflush size : 64
> >>>>> cache_alignment : 64
> >>>>> address sizes : 40 bits physical, 48 bits virtual
> >>>>> [root@cloud1 ~]# free
> >>>>> total used free shared buff/cache
> >>>>> available
> >>>>> Mem: 123596388 79795792 34047696 632116 9752900
> >>> 39999332
> >>>>> Swap: 4194300 1956824 2237476
> >>>>> [root@cloud1 ~]# yum update
> >>>>>
> >>>>>
> >>>
> =========================================================================================================================================================================
> >>>>> Package Arch Version Repository Size
> >>>>>
> >>>>>
> >>>
> =========================================================================================================================================================================
> >>>>> Updating:
> >>>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
> >>>>> cloudstack411 47 M
> >>>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos
> >>>>> cloudstack411 58 M
> >>>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos
> >>>>> cloudstack411 82 M
> >>>>>
> >>>>> Three compute servers running the above processor averaging 128gb of
> >>>>> memory apiece, two data servers running NFS for primary and secondary
> >>>>> storage. Running the 4.11.2 community RPM's above.
> >>>>>
> >>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating. The
> >>>>> routers in fact started with the template, it said 4.11.2 when I
> opened
> >>>>> up the console window and looked at the login prompt. But they never
> got
> >>>>> configured, when I logged in with root/password at the console prompt
> >>>>> they had no networking set up and no configuration provided, the
> >>>>> cloud-init output in /var/log was essentially blank.
> >>>>>
> >>>>> Do I recall correctly that there is an issue with a particular
> version
> >>>>> of the hypervisor on the latest Centos 7? Any other information that
> you
> >>>>> need? I think I provided the complete log file for the cloud-init of
> the
> >>>>> router in another email...
> >>>>>
> >>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
> >>>>>> Hi Eric,
> >>>>>>
> >>>>>> Can you describe your environment in detail such as management
> server
> >>>>> host distro, hypervisor version, current CloudStack version, did you
> >>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
> >>>>>> Regards.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Rohit Yadav
> >>>>>>
> >>>>>> ________________________________
> >>>>>> From: Eric Lee Green <er...@gmail.com>
> >>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
> >>>>>> To: users@cloudstack.apache.org
> >>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> >>>>>>
> >>>>>> You may remember me as the person who had to roll back to Cloudstack
> >>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines
> >>> once
> >>>>>> I upgraded to it, claiming that there were inadequate resources even
> >>>>>> though I had over 150 gigabytes of memory free in my cluster and
> oodles
> >>>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a
> >>> 512mb
> >>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
> >>>>>> *again* it's misbehaving.
> >>>>>>
> >>>>>> The symptom is that my virtual routers when I log into their console
> >>>>>> show 4.11.2 but when I look at them in the console they say
> 'Version:
> >>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or link
> >>>>>> local IP address it fails. And when I try to start up a virtual
> machine
> >>>>>> that uses that virtual network, it says "Network unavailable', even
> >>>>>> though the router for that network is showing up and running.
> >>>>>>
> >>>>>> Clearly something's broken in the virtual routers but I don't know
> what
> >>>>>> because I can't get into the router virtual machines. How do I get
> the
> >>>>>> console password to get into the router virtual machines? It's
> >>> encrypted
> >>>>>> in the database (duh), how do I decrypt it?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> rohit.yadav@shapeblue.com
> >>>>>> www.shapeblue.com
> >>>>>> Amadeus House, Floral Street, London WC2E 9DPUK
> >>>>>> @shapeblue
> >>>>>>
> >>>>>>
> >>>>>>
>
--
Andrija Panić
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Nux! <nu...@li.nux.ro>.
Eric,
I think you can get away with just downgrading to stock qemu-kvm packages (or patch your cloudstack installation).
BTW qemu-kvm-ev is not part of stock CentOS for a reason, SIG packages don't get the same level of testing and QA.
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Eric Lee Green" <er...@gmail.com>
> To: "users" <us...@cloudstack.apache.org>
> Sent: Wednesday, 22 May, 2019 15:33:15
> Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> Okay. This makes sense.
>
> And people wonder why Amazon decided to make their own Linux rather than
> use Centos and why Ubuntu has seized huge market share from Red Hat in
> the past few years. SIGH.
>
> Downgrading my CentOS is not going to be easy. There were security
> patches in the latest CentOS that I am required to have. It would
> actually be easier to just switch to Ubuntu at this point since we're
> talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software.
> I hope IBM fixes them, but I have my doubts.
>
> On 5/22/2019 1:05 AM, Andrija Panic wrote:
>> Hi Eric, all,
>>
>> I believe you might be hitting this one - issues on latest CentOS 7.6:
>> https://github.com/apache/cloudstack/pull/3333 due to changes in the OS
>> itself...
>>
>> If you believe that is the case, please try with CentOS 7.4 (can confirm
>> works fine) and/or CentOS 7.5.
>>
>> Best,
>> Andrija
>>
>>
>>
>> On Wed, 22 May 2019 at 09:04, Eric Lee Green <er...@gmail.com>
>> wrote:
>>
>>> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
>>>> Hello Eric,
>>>>
>>>> If the router version is displayed as UNKNOWN in the portal, it
>>> would be
>>>> a connectivity issue. Check your connections related to firewall rules
>>>> between the ACP management hosts, hypervisor and VR. Is your VR
>>> management
>>>> network setup correctly?
>>> Communications between the hosts is working fine and I have not changed
>>> the VR management network between the running 4.9 and the not-running
>>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
>>> should properly forward VR traffic.
>>>
>>> More tomorrow when I have some sleep since it is now midnight here in
>>> the SF Bay area.
>>>
>>>
>>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <er...@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks for the response, sorry if I sound frustrated, but this is
>>>>> supposed to be a simple easy process and it's been horrible all the way
>>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2
>>>>> has failed to upgrade too. I've thus far spent around 16 hours of my
>>>>> time on what should have taken an hour at most. I'm frustrated and
>>> bummed.
>>>>> [root@cloud2 ~]# rpm -q centos-release
>>>>> centos-release-7-6.1810.2.el7.centos.x86_64
>>>>> [root@cloud2 ~]# rpm -q libvirt
>>>>> libvirt-4.5.0-10.el7_6.9.x86_64
>>>>> [root@cloud1 ~]# rpm -qa | grep kvm
>>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
>>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
>>>>> [root@cloud2 ~]# cat /proc/version
>>>>> Linux version 3.10.0-862.6.3.el7.x86_64
>>>>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
>>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
>>>>> [root@cloud2 ~]# cat /proc/cpuinfo
>>>>> ...
>>>>> processor : 23
>>>>> vendor_id : GenuineIntel
>>>>> cpu family : 6
>>>>> model : 44
>>>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
>>>>> stepping : 2
>>>>> microcode : 0x1f
>>>>> cpu MHz : 3068.000
>>>>> cache size : 12288 KB
>>>>> physical id : 1
>>>>> siblings : 12
>>>>> core id : 10
>>>>> cpu cores : 6
>>>>> apicid : 53
>>>>> initial apicid : 53
>>>>> fpu : yes
>>>>> fpu_exception : yes
>>>>> cpuid level : 11
>>>>> wp : yes
>>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
>>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl
>>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
>>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm
>>>>> ida arat
>>>>> bogomips : 6133.21
>>>>> clflush size : 64
>>>>> cache_alignment : 64
>>>>> address sizes : 40 bits physical, 48 bits virtual
>>>>> [root@cloud1 ~]# free
>>>>> total used free shared buff/cache
>>>>> available
>>>>> Mem: 123596388 79795792 34047696 632116 9752900
>>> 39999332
>>>>> Swap: 4194300 1956824 2237476
>>>>> [root@cloud1 ~]# yum update
>>>>>
>>>>>
>>> =========================================================================================================================================================================
>>>>> Package Arch Version Repository Size
>>>>>
>>>>>
>>> =========================================================================================================================================================================
>>>>> Updating:
>>>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
>>>>> cloudstack411 47 M
>>>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos
>>>>> cloudstack411 58 M
>>>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos
>>>>> cloudstack411 82 M
>>>>>
>>>>> Three compute servers running the above processor averaging 128gb of
>>>>> memory apiece, two data servers running NFS for primary and secondary
>>>>> storage. Running the 4.11.2 community RPM's above.
>>>>>
>>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating. The
>>>>> routers in fact started with the template, it said 4.11.2 when I opened
>>>>> up the console window and looked at the login prompt. But they never got
>>>>> configured, when I logged in with root/password at the console prompt
>>>>> they had no networking set up and no configuration provided, the
>>>>> cloud-init output in /var/log was essentially blank.
>>>>>
>>>>> Do I recall correctly that there is an issue with a particular version
>>>>> of the hypervisor on the latest Centos 7? Any other information that you
>>>>> need? I think I provided the complete log file for the cloud-init of the
>>>>> router in another email...
>>>>>
>>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
>>>>>> Hi Eric,
>>>>>>
>>>>>> Can you describe your environment in detail such as management server
>>>>> host distro, hypervisor version, current CloudStack version, did you
>>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
>>>>>> Regards.
>>>>>>
>>>>>> Regards,
>>>>>> Rohit Yadav
>>>>>>
>>>>>> ________________________________
>>>>>> From: Eric Lee Green <er...@gmail.com>
>>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
>>>>>> To: users@cloudstack.apache.org
>>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>>>>>>
>>>>>> You may remember me as the person who had to roll back to Cloudstack
>>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines
>>> once
>>>>>> I upgraded to it, claiming that there were inadequate resources even
>>>>>> though I had over 150 gigabytes of memory free in my cluster and oodles
>>>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a
>>> 512mb
>>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
>>>>>> *again* it's misbehaving.
>>>>>>
>>>>>> The symptom is that my virtual routers when I log into their console
>>>>>> show 4.11.2 but when I look at them in the console they say 'Version:
>>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or link
>>>>>> local IP address it fails. And when I try to start up a virtual machine
>>>>>> that uses that virtual network, it says "Network unavailable', even
>>>>>> though the router for that network is showing up and running.
>>>>>>
>>>>>> Clearly something's broken in the virtual routers but I don't know what
>>>>>> because I can't get into the router virtual machines. How do I get the
>>>>>> console password to get into the router virtual machines? It's
>>> encrypted
>>>>>> in the database (duh), how do I decrypt it?
>>>>>>
>>>>>>
>>>>>>
>>>>>> rohit.yadav@shapeblue.com
>>>>>> www.shapeblue.com
>>>>>> Amadeus House, Floral Street, London WC2E 9DPUK
>>>>>> @shapeblue
>>>>>>
>>>>>>
>>>>>>
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Eric Lee Green <er...@gmail.com>.
Okay. This makes sense.
And people wonder why Amazon decided to make their own Linux rather than
use Centos and why Ubuntu has seized huge market share from Red Hat in
the past few years. SIGH.
Downgrading my CentOS is not going to be easy. There were security
patches in the latest CentOS that I am required to have. It would
actually be easier to just switch to Ubuntu at this point since we're
talking a complete re-install anyhow. SIGH. Thank you, Red Hat Software.
I hope IBM fixes them, but I have my doubts.
On 5/22/2019 1:05 AM, Andrija Panic wrote:
> Hi Eric, all,
>
> I believe you might be hitting this one - issues on latest CentOS 7.6:
> https://github.com/apache/cloudstack/pull/3333 due to changes in the OS
> itself...
>
> If you believe that is the case, please try with CentOS 7.4 (can confirm
> works fine) and/or CentOS 7.5.
>
> Best,
> Andrija
>
>
>
> On Wed, 22 May 2019 at 09:04, Eric Lee Green <er...@gmail.com>
> wrote:
>
>> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
>>> Hello Eric,
>>>
>>> If the router version is displayed as UNKNOWN in the portal, it
>> would be
>>> a connectivity issue. Check your connections related to firewall rules
>>> between the ACP management hosts, hypervisor and VR. Is your VR
>> management
>>> network setup correctly?
>> Communications between the hosts is working fine and I have not changed
>> the VR management network between the running 4.9 and the not-running
>> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
>> should properly forward VR traffic.
>>
>> More tomorrow when I have some sleep since it is now midnight here in
>> the SF Bay area.
>>
>>
>>> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <er...@gmail.com>
>>> wrote:
>>>
>>>> Thanks for the response, sorry if I sound frustrated, but this is
>>>> supposed to be a simple easy process and it's been horrible all the way
>>>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2
>>>> has failed to upgrade too. I've thus far spent around 16 hours of my
>>>> time on what should have taken an hour at most. I'm frustrated and
>> bummed.
>>>> [root@cloud2 ~]# rpm -q centos-release
>>>> centos-release-7-6.1810.2.el7.centos.x86_64
>>>> [root@cloud2 ~]# rpm -q libvirt
>>>> libvirt-4.5.0-10.el7_6.9.x86_64
>>>> [root@cloud1 ~]# rpm -qa | grep kvm
>>>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
>>>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
>>>> [root@cloud2 ~]# cat /proc/version
>>>> Linux version 3.10.0-862.6.3.el7.x86_64
>>>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
>>>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
>>>> [root@cloud2 ~]# cat /proc/cpuinfo
>>>> ...
>>>> processor : 23
>>>> vendor_id : GenuineIntel
>>>> cpu family : 6
>>>> model : 44
>>>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
>>>> stepping : 2
>>>> microcode : 0x1f
>>>> cpu MHz : 3068.000
>>>> cache size : 12288 KB
>>>> physical id : 1
>>>> siblings : 12
>>>> core id : 10
>>>> cpu cores : 6
>>>> apicid : 53
>>>> initial apicid : 53
>>>> fpu : yes
>>>> fpu_exception : yes
>>>> cpuid level : 11
>>>> wp : yes
>>>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>>>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>>>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
>>>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl
>>>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
>>>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm
>>>> ida arat
>>>> bogomips : 6133.21
>>>> clflush size : 64
>>>> cache_alignment : 64
>>>> address sizes : 40 bits physical, 48 bits virtual
>>>> [root@cloud1 ~]# free
>>>> total used free shared buff/cache
>>>> available
>>>> Mem: 123596388 79795792 34047696 632116 9752900
>> 39999332
>>>> Swap: 4194300 1956824 2237476
>>>> [root@cloud1 ~]# yum update
>>>>
>>>>
>> =========================================================================================================================================================================
>>>> Package Arch Version Repository Size
>>>>
>>>>
>> =========================================================================================================================================================================
>>>> Updating:
>>>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
>>>> cloudstack411 47 M
>>>> cloudstack-common x86_64 4.11.2.0-1.el7.centos
>>>> cloudstack411 58 M
>>>> cloudstack-management x86_64 4.11.2.0-1.el7.centos
>>>> cloudstack411 82 M
>>>>
>>>> Three compute servers running the above processor averaging 128gb of
>>>> memory apiece, two data servers running NFS for primary and secondary
>>>> storage. Running the 4.11.2 community RPM's above.
>>>>
>>>> Yes, I did register the 4.11.2 systemvmtemplate before updating. The
>>>> routers in fact started with the template, it said 4.11.2 when I opened
>>>> up the console window and looked at the login prompt. But they never got
>>>> configured, when I logged in with root/password at the console prompt
>>>> they had no networking set up and no configuration provided, the
>>>> cloud-init output in /var/log was essentially blank.
>>>>
>>>> Do I recall correctly that there is an issue with a particular version
>>>> of the hypervisor on the latest Centos 7? Any other information that you
>>>> need? I think I provided the complete log file for the cloud-init of the
>>>> router in another email...
>>>>
>>>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
>>>>> Hi Eric,
>>>>>
>>>>> Can you describe your environment in detail such as management server
>>>> host distro, hypervisor version, current CloudStack version, did you
>>>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
>>>>> Regards.
>>>>>
>>>>> Regards,
>>>>> Rohit Yadav
>>>>>
>>>>> ________________________________
>>>>> From: Eric Lee Green <er...@gmail.com>
>>>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
>>>>> To: users@cloudstack.apache.org
>>>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>>>>>
>>>>> You may remember me as the person who had to roll back to Cloudstack
>>>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines
>> once
>>>>> I upgraded to it, claiming that there were inadequate resources even
>>>>> though I had over 150 gigabytes of memory free in my cluster and oodles
>>>>> of CPU free (and a minimum of 40gb on each node, plenty to start a
>> 512mb
>>>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
>>>>> *again* it's misbehaving.
>>>>>
>>>>> The symptom is that my virtual routers when I log into their console
>>>>> show 4.11.2 but when I look at them in the console they say 'Version:
>>>>> UNKNOWN'. Also when I try to ssh into their guest IP address or link
>>>>> local IP address it fails. And when I try to start up a virtual machine
>>>>> that uses that virtual network, it says "Network unavailable', even
>>>>> though the router for that network is showing up and running.
>>>>>
>>>>> Clearly something's broken in the virtual routers but I don't know what
>>>>> because I can't get into the router virtual machines. How do I get the
>>>>> console password to get into the router virtual machines? It's
>> encrypted
>>>>> in the database (duh), how do I decrypt it?
>>>>>
>>>>>
>>>>>
>>>>> rohit.yadav@shapeblue.com
>>>>> www.shapeblue.com
>>>>> Amadeus House, Floral Street, London WC2E 9DPUK
>>>>> @shapeblue
>>>>>
>>>>>
>>>>>
>
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Andrija Panic <an...@gmail.com>.
Hi Eric, all,
I believe you might be hitting this one - issues on latest CentOS 7.6:
https://github.com/apache/cloudstack/pull/3333 due to changes in the OS
itself...
If you believe that is the case, please try with CentOS 7.4 (can confirm
works fine) and/or CentOS 7.5.
Best,
Andrija
On Wed, 22 May 2019 at 09:04, Eric Lee Green <er...@gmail.com>
wrote:
>
> On 5/21/2019 11:10 PM, Thomas Joseph wrote:
> > Hello Eric,
> >
> > If the router version is displayed as UNKNOWN in the portal, it
> would be
> > a connectivity issue. Check your connections related to firewall rules
> > between the ACP management hosts, hypervisor and VR. Is your VR
> management
> > network setup correctly?
>
> Communications between the hosts is working fine and I have not changed
> the VR management network between the running 4.9 and the not-running
> 4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
> should properly forward VR traffic.
>
> More tomorrow when I have some sleep since it is now midnight here in
> the SF Bay area.
>
>
> >
> > On Wed, 22 May 2019, 6:10 am Eric Lee Green, <er...@gmail.com>
> > wrote:
> >
> >> Thanks for the response, sorry if I sound frustrated, but this is
> >> supposed to be a simple easy process and it's been horrible all the way
> >> through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2
> >> has failed to upgrade too. I've thus far spent around 16 hours of my
> >> time on what should have taken an hour at most. I'm frustrated and
> bummed.
> >>
> >> [root@cloud2 ~]# rpm -q centos-release
> >> centos-release-7-6.1810.2.el7.centos.x86_64
> >> [root@cloud2 ~]# rpm -q libvirt
> >> libvirt-4.5.0-10.el7_6.9.x86_64
> >> [root@cloud1 ~]# rpm -qa | grep kvm
> >> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
> >> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
> >> [root@cloud2 ~]# cat /proc/version
> >> Linux version 3.10.0-862.6.3.el7.x86_64
> >> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
> >> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
> >> [root@cloud2 ~]# cat /proc/cpuinfo
> >> ...
> >> processor : 23
> >> vendor_id : GenuineIntel
> >> cpu family : 6
> >> model : 44
> >> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
> >> stepping : 2
> >> microcode : 0x1f
> >> cpu MHz : 3068.000
> >> cache size : 12288 KB
> >> physical id : 1
> >> siblings : 12
> >> core id : 10
> >> cpu cores : 6
> >> apicid : 53
> >> initial apicid : 53
> >> fpu : yes
> >> fpu_exception : yes
> >> cpuid level : 11
> >> wp : yes
> >> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> >> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> >> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
> >> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl
> >> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
> >> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm
> >> ida arat
> >> bogomips : 6133.21
> >> clflush size : 64
> >> cache_alignment : 64
> >> address sizes : 40 bits physical, 48 bits virtual
> >> [root@cloud1 ~]# free
> >> total used free shared buff/cache
> >> available
> >> Mem: 123596388 79795792 34047696 632116 9752900
> 39999332
> >> Swap: 4194300 1956824 2237476
> >> [root@cloud1 ~]# yum update
> >>
> >>
> =========================================================================================================================================================================
> >> Package Arch Version Repository Size
> >>
> >>
> =========================================================================================================================================================================
> >> Updating:
> >> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
> >> cloudstack411 47 M
> >> cloudstack-common x86_64 4.11.2.0-1.el7.centos
> >> cloudstack411 58 M
> >> cloudstack-management x86_64 4.11.2.0-1.el7.centos
> >> cloudstack411 82 M
> >>
> >> Three compute servers running the above processor averaging 128gb of
> >> memory apiece, two data servers running NFS for primary and secondary
> >> storage. Running the 4.11.2 community RPM's above.
> >>
> >> Yes, I did register the 4.11.2 systemvmtemplate before updating. The
> >> routers in fact started with the template, it said 4.11.2 when I opened
> >> up the console window and looked at the login prompt. But they never got
> >> configured, when I logged in with root/password at the console prompt
> >> they had no networking set up and no configuration provided, the
> >> cloud-init output in /var/log was essentially blank.
> >>
> >> Do I recall correctly that there is an issue with a particular version
> >> of the hypervisor on the latest Centos 7? Any other information that you
> >> need? I think I provided the complete log file for the cloud-init of the
> >> router in another email...
> >>
> >> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
> >>> Hi Eric,
> >>>
> >>> Can you describe your environment in detail such as management server
> >> host distro, hypervisor version, current CloudStack version, did you
> >> register the 4.11.2 systemvmtemplate beforw upgrading etc.
> >>> Regards.
> >>>
> >>> Regards,
> >>> Rohit Yadav
> >>>
> >>> ________________________________
> >>> From: Eric Lee Green <er...@gmail.com>
> >>> Sent: Wednesday, May 22, 2019 6:21:16 AM
> >>> To: users@cloudstack.apache.org
> >>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> >>>
> >>> You may remember me as the person who had to roll back to Cloudstack
> >>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines
> once
> >>> I upgraded to it, claiming that there were inadequate resources even
> >>> though I had over 150 gigabytes of memory free in my cluster and oodles
> >>> of CPU free (and a minimum of 40gb on each node, plenty to start a
> 512mb
> >>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
> >>> *again* it's misbehaving.
> >>>
> >>> The symptom is that my virtual routers when I log into their console
> >>> show 4.11.2 but when I look at them in the console they say 'Version:
> >>> UNKNOWN'. Also when I try to ssh into their guest IP address or link
> >>> local IP address it fails. And when I try to start up a virtual machine
> >>> that uses that virtual network, it says "Network unavailable', even
> >>> though the router for that network is showing up and running.
> >>>
> >>> Clearly something's broken in the virtual routers but I don't know what
> >>> because I can't get into the router virtual machines. How do I get the
> >>> console password to get into the router virtual machines? It's
> encrypted
> >>> in the database (duh), how do I decrypt it?
> >>>
> >>>
> >>>
> >>> rohit.yadav@shapeblue.com
> >>> www.shapeblue.com
> >>> Amadeus House, Floral Street, London WC2E 9DPUK
> >>> @shapeblue
> >>>
> >>>
> >>>
>
--
Andrija Panić
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Eric Lee Green <er...@gmail.com>.
On 5/21/2019 11:10 PM, Thomas Joseph wrote:
> Hello Eric,
>
> If the router version is displayed as UNKNOWN in the portal, it would be
> a connectivity issue. Check your connections related to firewall rules
> between the ACP management hosts, hypervisor and VR. Is your VR management
> network setup correctly?
Communications between the hosts is working fine and I have not changed
the VR management network between the running 4.9 and the not-running
4.11. FIrewall rules appear to be the defaults set up by Cloudstack and
should properly forward VR traffic.
More tomorrow when I have some sleep since it is now midnight here in
the SF Bay area.
>
> On Wed, 22 May 2019, 6:10 am Eric Lee Green, <er...@gmail.com>
> wrote:
>
>> Thanks for the response, sorry if I sound frustrated, but this is
>> supposed to be a simple easy process and it's been horrible all the way
>> through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2
>> has failed to upgrade too. I've thus far spent around 16 hours of my
>> time on what should have taken an hour at most. I'm frustrated and bummed.
>>
>> [root@cloud2 ~]# rpm -q centos-release
>> centos-release-7-6.1810.2.el7.centos.x86_64
>> [root@cloud2 ~]# rpm -q libvirt
>> libvirt-4.5.0-10.el7_6.9.x86_64
>> [root@cloud1 ~]# rpm -qa | grep kvm
>> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
>> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
>> [root@cloud2 ~]# cat /proc/version
>> Linux version 3.10.0-862.6.3.el7.x86_64
>> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
>> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
>> [root@cloud2 ~]# cat /proc/cpuinfo
>> ...
>> processor : 23
>> vendor_id : GenuineIntel
>> cpu family : 6
>> model : 44
>> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
>> stepping : 2
>> microcode : 0x1f
>> cpu MHz : 3068.000
>> cache size : 12288 KB
>> physical id : 1
>> siblings : 12
>> core id : 10
>> cpu cores : 6
>> apicid : 53
>> initial apicid : 53
>> fpu : yes
>> fpu_exception : yes
>> cpuid level : 11
>> wp : yes
>> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
>> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
>> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
>> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl
>> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
>> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm
>> ida arat
>> bogomips : 6133.21
>> clflush size : 64
>> cache_alignment : 64
>> address sizes : 40 bits physical, 48 bits virtual
>> [root@cloud1 ~]# free
>> total used free shared buff/cache
>> available
>> Mem: 123596388 79795792 34047696 632116 9752900 39999332
>> Swap: 4194300 1956824 2237476
>> [root@cloud1 ~]# yum update
>>
>> =========================================================================================================================================================================
>> Package Arch Version Repository Size
>>
>> =========================================================================================================================================================================
>> Updating:
>> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
>> cloudstack411 47 M
>> cloudstack-common x86_64 4.11.2.0-1.el7.centos
>> cloudstack411 58 M
>> cloudstack-management x86_64 4.11.2.0-1.el7.centos
>> cloudstack411 82 M
>>
>> Three compute servers running the above processor averaging 128gb of
>> memory apiece, two data servers running NFS for primary and secondary
>> storage. Running the 4.11.2 community RPM's above.
>>
>> Yes, I did register the 4.11.2 systemvmtemplate before updating. The
>> routers in fact started with the template, it said 4.11.2 when I opened
>> up the console window and looked at the login prompt. But they never got
>> configured, when I logged in with root/password at the console prompt
>> they had no networking set up and no configuration provided, the
>> cloud-init output in /var/log was essentially blank.
>>
>> Do I recall correctly that there is an issue with a particular version
>> of the hypervisor on the latest Centos 7? Any other information that you
>> need? I think I provided the complete log file for the cloud-init of the
>> router in another email...
>>
>> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
>>> Hi Eric,
>>>
>>> Can you describe your environment in detail such as management server
>> host distro, hypervisor version, current CloudStack version, did you
>> register the 4.11.2 systemvmtemplate beforw upgrading etc.
>>> Regards.
>>>
>>> Regards,
>>> Rohit Yadav
>>>
>>> ________________________________
>>> From: Eric Lee Green <er...@gmail.com>
>>> Sent: Wednesday, May 22, 2019 6:21:16 AM
>>> To: users@cloudstack.apache.org
>>> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>>>
>>> You may remember me as the person who had to roll back to Cloudstack
>>> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once
>>> I upgraded to it, claiming that there were inadequate resources even
>>> though I had over 150 gigabytes of memory free in my cluster and oodles
>>> of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb
>>> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
>>> *again* it's misbehaving.
>>>
>>> The symptom is that my virtual routers when I log into their console
>>> show 4.11.2 but when I look at them in the console they say 'Version:
>>> UNKNOWN'. Also when I try to ssh into their guest IP address or link
>>> local IP address it fails. And when I try to start up a virtual machine
>>> that uses that virtual network, it says "Network unavailable', even
>>> though the router for that network is showing up and running.
>>>
>>> Clearly something's broken in the virtual routers but I don't know what
>>> because I can't get into the router virtual machines. How do I get the
>>> console password to get into the router virtual machines? It's encrypted
>>> in the database (duh), how do I decrypt it?
>>>
>>>
>>>
>>> rohit.yadav@shapeblue.com
>>> www.shapeblue.com
>>> Amadeus House, Floral Street, London WC2E 9DPUK
>>> @shapeblue
>>>
>>>
>>>
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Thomas Joseph <th...@gmail.com>.
Hello Eric,
If the router version is displayed as UNKNOWN in the portal, it would be
a connectivity issue. Check your connections related to firewall rules
between the ACP management hosts, hypervisor and VR. Is your VR management
network setup correctly?
With regards
Thomas
On Wed, 22 May 2019, 6:10 am Eric Lee Green, <er...@gmail.com>
wrote:
> Thanks for the response, sorry if I sound frustrated, but this is
> supposed to be a simple easy process and it's been horrible all the way
> through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2
> has failed to upgrade too. I've thus far spent around 16 hours of my
> time on what should have taken an hour at most. I'm frustrated and bummed.
>
> [root@cloud2 ~]# rpm -q centos-release
> centos-release-7-6.1810.2.el7.centos.x86_64
> [root@cloud2 ~]# rpm -q libvirt
> libvirt-4.5.0-10.el7_6.9.x86_64
> [root@cloud1 ~]# rpm -qa | grep kvm
> qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
> qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
> [root@cloud2 ~]# cat /proc/version
> Linux version 3.10.0-862.6.3.el7.x86_64
> (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
> 4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
> [root@cloud2 ~]# cat /proc/cpuinfo
> ...
> processor : 23
> vendor_id : GenuineIntel
> cpu family : 6
> model : 44
> model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
> stepping : 2
> microcode : 0x1f
> cpu MHz : 3068.000
> cache size : 12288 KB
> physical id : 1
> siblings : 12
> core id : 10
> cpu cores : 6
> apicid : 53
> initial apicid : 53
> fpu : yes
> fpu_exception : yes
> cpuid level : 11
> wp : yes
> flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
> mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
> syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
> nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl
> vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
> lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm
> ida arat
> bogomips : 6133.21
> clflush size : 64
> cache_alignment : 64
> address sizes : 40 bits physical, 48 bits virtual
> [root@cloud1 ~]# free
> total used free shared buff/cache
> available
> Mem: 123596388 79795792 34047696 632116 9752900 39999332
> Swap: 4194300 1956824 2237476
> [root@cloud1 ~]# yum update
>
> =========================================================================================================================================================================
> Package Arch Version Repository Size
>
> =========================================================================================================================================================================
> Updating:
> cloudstack-agent x86_64 4.11.2.0-1.el7.centos
> cloudstack411 47 M
> cloudstack-common x86_64 4.11.2.0-1.el7.centos
> cloudstack411 58 M
> cloudstack-management x86_64 4.11.2.0-1.el7.centos
> cloudstack411 82 M
>
> Three compute servers running the above processor averaging 128gb of
> memory apiece, two data servers running NFS for primary and secondary
> storage. Running the 4.11.2 community RPM's above.
>
> Yes, I did register the 4.11.2 systemvmtemplate before updating. The
> routers in fact started with the template, it said 4.11.2 when I opened
> up the console window and looked at the login prompt. But they never got
> configured, when I logged in with root/password at the console prompt
> they had no networking set up and no configuration provided, the
> cloud-init output in /var/log was essentially blank.
>
> Do I recall correctly that there is an issue with a particular version
> of the hypervisor on the latest Centos 7? Any other information that you
> need? I think I provided the complete log file for the cloud-init of the
> router in another email...
>
> On 5/21/2019 9:38 PM, Rohit Yadav wrote:
> > Hi Eric,
> >
> > Can you describe your environment in detail such as management server
> host distro, hypervisor version, current CloudStack version, did you
> register the 4.11.2 systemvmtemplate beforw upgrading etc.
> >
> > Regards.
> >
> > Regards,
> > Rohit Yadav
> >
> > ________________________________
> > From: Eric Lee Green <er...@gmail.com>
> > Sent: Wednesday, May 22, 2019 6:21:16 AM
> > To: users@cloudstack.apache.org
> > Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
> >
> > You may remember me as the person who had to roll back to Cloudstack
> > 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once
> > I upgraded to it, claiming that there were inadequate resources even
> > though I had over 150 gigabytes of memory free in my cluster and oodles
> > of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb
> > router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
> > *again* it's misbehaving.
> >
> > The symptom is that my virtual routers when I log into their console
> > show 4.11.2 but when I look at them in the console they say 'Version:
> > UNKNOWN'. Also when I try to ssh into their guest IP address or link
> > local IP address it fails. And when I try to start up a virtual machine
> > that uses that virtual network, it says "Network unavailable', even
> > though the router for that network is showing up and running.
> >
> > Clearly something's broken in the virtual routers but I don't know what
> > because I can't get into the router virtual machines. How do I get the
> > console password to get into the router virtual machines? It's encrypted
> > in the database (duh), how do I decrypt it?
> >
> >
> >
> > rohit.yadav@shapeblue.com
> > www.shapeblue.com
> > Amadeus House, Floral Street, London WC2E 9DPUK
> > @shapeblue
> >
> >
> >
>
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Eric,
Based on your shared version details, the specific issue and why it does not work for you has to do with usage of qemu-ev 2.12, which was only recently supported:
https://github.com/apache/cloudstack/pull/3278
The issue has to do with how systemvms are patched by cloudstack-agent using a patching script that no longer works well with qemu-ev 2.12. You can test latest 4.11 or wait until 4.11.3.0 is out that will have support for qemu-ev 2.12 with a new patching mechanism that uses qemu-guest-agent.
Regards,
Rohit Yadav
Software Architect, ShapeBlue
https://www.shapeblue.com
________________________________
From: Eric Lee Green <er...@gmail.com>
Sent: Wednesday, May 22, 2019 10:40:08 AM
To: users@cloudstack.apache.org; Rohit Yadav
Subject: Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Thanks for the response, sorry if I sound frustrated, but this is
supposed to be a simple easy process and it's been horrible all the way
through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2
has failed to upgrade too. I've thus far spent around 16 hours of my
time on what should have taken an hour at most. I'm frustrated and bummed.
[root@cloud2 ~]# rpm -q centos-release
centos-release-7-6.1810.2.el7.centos.x86_64
[root@cloud2 ~]# rpm -q libvirt
libvirt-4.5.0-10.el7_6.9.x86_64
[root@cloud1 ~]# rpm -qa | grep kvm
qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
[root@cloud2 ~]# cat /proc/version
Linux version 3.10.0-862.6.3.el7.x86_64
(builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
[root@cloud2 ~]# cat /proc/cpuinfo
...
processor : 23
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
stepping : 2
microcode : 0x1f
cpu MHz : 3068.000
cache size : 12288 KB
physical id : 1
siblings : 12
core id : 10
cpu cores : 6
apicid : 53
initial apicid : 53
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl
vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm
ida arat
bogomips : 6133.21
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
[root@cloud1 ~]# free
total used free shared buff/cache
available
Mem: 123596388 79795792 34047696 632116 9752900 39999332
Swap: 4194300 1956824 2237476
[root@cloud1 ~]# yum update
=========================================================================================================================================================================
Package Arch Version Repository Size
=========================================================================================================================================================================
Updating:
cloudstack-agent x86_64 4.11.2.0-1.el7.centos
cloudstack411 47 M
cloudstack-common x86_64 4.11.2.0-1.el7.centos
cloudstack411 58 M
cloudstack-management x86_64 4.11.2.0-1.el7.centos
cloudstack411 82 M
Three compute servers running the above processor averaging 128gb of
memory apiece, two data servers running NFS for primary and secondary
storage. Running the 4.11.2 community RPM's above.
Yes, I did register the 4.11.2 systemvmtemplate before updating. The
routers in fact started with the template, it said 4.11.2 when I opened
up the console window and looked at the login prompt. But they never got
configured, when I logged in with root/password at the console prompt
they had no networking set up and no configuration provided, the
cloud-init output in /var/log was essentially blank.
Do I recall correctly that there is an issue with a particular version
of the hypervisor on the latest Centos 7? Any other information that you
need? I think I provided the complete log file for the cloud-init of the
router in another email...
On 5/21/2019 9:38 PM, Rohit Yadav wrote:
> Hi Eric,
>
> Can you describe your environment in detail such as management server host distro, hypervisor version, current CloudStack version, did you register the 4.11.2 systemvmtemplate beforw upgrading etc.
>
> Regards.
>
> Regards,
> Rohit Yadav
>
> ________________________________
> From: Eric Lee Green <er...@gmail.com>
> Sent: Wednesday, May 22, 2019 6:21:16 AM
> To: users@cloudstack.apache.org
> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>
> You may remember me as the person who had to roll back to Cloudstack
> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once
> I upgraded to it, claiming that there were inadequate resources even
> though I had over 150 gigabytes of memory free in my cluster and oodles
> of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb
> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
> *again* it's misbehaving.
>
> The symptom is that my virtual routers when I log into their console
> show 4.11.2 but when I look at them in the console they say 'Version:
> UNKNOWN'. Also when I try to ssh into their guest IP address or link
> local IP address it fails. And when I try to start up a virtual machine
> that uses that virtual network, it says "Network unavailable', even
> though the router for that network is showing up and running.
>
> Clearly something's broken in the virtual routers but I don't know what
> because I can't get into the router virtual machines. How do I get the
> console password to get into the router virtual machines? It's encrypted
> in the database (duh), how do I decrypt it?
>
>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com<http://www.shapeblue.com>
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
rohit.yadav@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Eric Lee Green <er...@gmail.com>.
Thanks for the response, sorry if I sound frustrated, but this is
supposed to be a simple easy process and it's been horrible all the way
through. 4.11.1 failed so I had to downgrade back to 4.9, and now 4.11.2
has failed to upgrade too. I've thus far spent around 16 hours of my
time on what should have taken an hour at most. I'm frustrated and bummed.
[root@cloud2 ~]# rpm -q centos-release
centos-release-7-6.1810.2.el7.centos.x86_64
[root@cloud2 ~]# rpm -q libvirt
libvirt-4.5.0-10.el7_6.9.x86_64
[root@cloud1 ~]# rpm -qa | grep kvm
qemu-kvm-ev-2.12.0-18.el7_6.5.1.x86_64
qemu-kvm-common-ev-2.12.0-18.el7_6.5.1.x86_64
[root@cloud2 ~]# cat /proc/version
Linux version 3.10.0-862.6.3.el7.x86_64
(builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat
4.8.5-28) (GCC) ) #1 SMP Tue Jun 26 16:32:21 UTC 2018
[root@cloud2 ~]# cat /proc/cpuinfo
...
processor : 23
vendor_id : GenuineIntel
cpu family : 6
model : 44
model name : Intel(R) Xeon(R) CPU X5675 @ 3.07GHz
stepping : 2
microcode : 0x1f
cpu MHz : 3068.000
cache size : 12288 KB
physical id : 1
siblings : 12
core id : 10
cpu cores : 6
apicid : 53
initial apicid : 53
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe
syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good
nopl xtopology nonstop_tsc aperfmperf eagerfpu pni dtes64 monitor ds_cpl
vmx smx est tm2 ssse3 cx16 xtpr pdcm pcid dca sse4_1 sse4_2 popcnt
lahf_lm epb ssbd ibrs ibpb tpr_shadow vnmi flexpriority ept vpid dtherm
ida arat
bogomips : 6133.21
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
[root@cloud1 ~]# free
total used free shared buff/cache
available
Mem: 123596388 79795792 34047696 632116 9752900 39999332
Swap: 4194300 1956824 2237476
[root@cloud1 ~]# yum update
=========================================================================================================================================================================
Package Arch Version Repository Size
=========================================================================================================================================================================
Updating:
cloudstack-agent x86_64 4.11.2.0-1.el7.centos
cloudstack411 47 M
cloudstack-common x86_64 4.11.2.0-1.el7.centos
cloudstack411 58 M
cloudstack-management x86_64 4.11.2.0-1.el7.centos
cloudstack411 82 M
Three compute servers running the above processor averaging 128gb of
memory apiece, two data servers running NFS for primary and secondary
storage. Running the 4.11.2 community RPM's above.
Yes, I did register the 4.11.2 systemvmtemplate before updating. The
routers in fact started with the template, it said 4.11.2 when I opened
up the console window and looked at the login prompt. But they never got
configured, when I logged in with root/password at the console prompt
they had no networking set up and no configuration provided, the
cloud-init output in /var/log was essentially blank.
Do I recall correctly that there is an issue with a particular version
of the hypervisor on the latest Centos 7? Any other information that you
need? I think I provided the complete log file for the cloud-init of the
router in another email...
On 5/21/2019 9:38 PM, Rohit Yadav wrote:
> Hi Eric,
>
> Can you describe your environment in detail such as management server host distro, hypervisor version, current CloudStack version, did you register the 4.11.2 systemvmtemplate beforw upgrading etc.
>
> Regards.
>
> Regards,
> Rohit Yadav
>
> ________________________________
> From: Eric Lee Green <er...@gmail.com>
> Sent: Wednesday, May 22, 2019 6:21:16 AM
> To: users@cloudstack.apache.org
> Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
>
> You may remember me as the person who had to roll back to Cloudstack
> 4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once
> I upgraded to it, claiming that there were inadequate resources even
> though I had over 150 gigabytes of memory free in my cluster and oodles
> of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb
> router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
> *again* it's misbehaving.
>
> The symptom is that my virtual routers when I log into their console
> show 4.11.2 but when I look at them in the console they say 'Version:
> UNKNOWN'. Also when I try to ssh into their guest IP address or link
> local IP address it fails. And when I try to start up a virtual machine
> that uses that virtual network, it says "Network unavailable', even
> though the router for that network is showing up and running.
>
> Clearly something's broken in the virtual routers but I don't know what
> because I can't get into the router virtual machines. How do I get the
> console password to get into the router virtual machines? It's encrypted
> in the database (duh), how do I decrypt it?
>
>
>
> rohit.yadav@shapeblue.com
> www.shapeblue.com
> Amadeus House, Floral Street, London WC2E 9DPUK
> @shapeblue
>
>
>
Re: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
Posted by Rohit Yadav <ro...@shapeblue.com>.
Hi Eric,
Can you describe your environment in detail such as management server host distro, hypervisor version, current CloudStack version, did you register the 4.11.2 systemvmtemplate beforw upgrading etc.
Regards.
Regards,
Rohit Yadav
________________________________
From: Eric Lee Green <er...@gmail.com>
Sent: Wednesday, May 22, 2019 6:21:16 AM
To: users@cloudstack.apache.org
Subject: Upgrade to Cloudstack 4.11.2 fails *AGAIN*
You may remember me as the person who had to roll back to Cloudstack
4.9.x because Cloudstack 4.11.1 wouldn't start any virtual machines once
I upgraded to it, claiming that there were inadequate resources even
though I had over 150 gigabytes of memory free in my cluster and oodles
of CPU free (and a minimum of 40gb on each node, plenty to start a 512mb
router VM). So now I'm trying to upgrade to Cloudstack 4.11.2 and
*again* it's misbehaving.
The symptom is that my virtual routers when I log into their console
show 4.11.2 but when I look at them in the console they say 'Version:
UNKNOWN'. Also when I try to ssh into their guest IP address or link
local IP address it fails. And when I try to start up a virtual machine
that uses that virtual network, it says "Network unavailable', even
though the router for that network is showing up and running.
Clearly something's broken in the virtual routers but I don't know what
because I can't get into the router virtual machines. How do I get the
console password to get into the router virtual machines? It's encrypted
in the database (duh), how do I decrypt it?
rohit.yadav@shapeblue.com
www.shapeblue.com
Amadeus House, Floral Street, London WC2E 9DPUK
@shapeblue