You are viewing a plain text version of this content. The canonical link for it is here.

Posted to users@cloudstack.apache.org by Trevor Francis <tr...@46labs.com> on 2012/12/02 16:26:31 UTC

BAD System VMs in Ubuntu 12.04

I have recently completely rebuilding cloudstack and a kvm hypervisor from scratch to determine if there is an issue with my system or with the latest Ubuntu cloudstack build.

Currently, cloudstack builds bad systems VMs on Ubuntu 12.04 Hosts and Management server. The system VM is pingable, but I am getting "Connection Refused" on CPVM and the SSVM will not respond to ISO uploads.

Plus:

I run a: ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.3.122
and receive a: Permission denied (public key)

I looked at the mysql database for the keys and compared them against the keys on the hypervisors. They do not match.

There has to be a problem with the builds, as I have been able to replicate this multiple times on completely fresh builds. 

UB 12.04 on both HV and Cloustack hosts. Running normal apt-get install cloud-agent and cloud-client respectively.

Help!

Trevor Francis
Partner
46 Labs | PeerEdge Cloud Switch (PeCS)
http://www.46labs.com | http://www.peeredge.net
720-214-3643- Voice
trevor@46labs.com
 
Solutions Provider for the Telecom Industry

RE: BAD System VMs in Ubuntu 12.04

Posted by Rohit Yadav <ro...@citrix.com>.

Trevor, when you say you completely rebuilt cloudstack, do you mean that by source code? If you're building using asf master, systemvm.iso won't be built unless you enable the systemvm profile:
mvn clean install -P systemvm,developer

If not, you can go ahead and destroy the systemvm and retry? The other thing you can do is use wine or natively, run XenCenter and debug from the console, see if you see any errors etc.?
Also, after installation (deployment and not developer setup) you need to setup databases and management server (see docs for scripts names and args).

Regards.
________________________________________
From: Trevor Francis [trevor@46labs.com]
Sent: Sunday, December 02, 2012 8:56 PM
To: cloudstack-users@incubator.apache.org
Subject: BAD System VMs in Ubuntu 12.04

I have recently completely rebuilding cloudstack and a kvm hypervisor from scratch to determine if there is an issue with my system or with the latest Ubuntu cloudstack build.

Currently, cloudstack builds bad systems VMs on Ubuntu 12.04 Hosts and Management server. The system VM is pingable, but I am getting "Connection Refused" on CPVM and the SSVM will not respond to ISO uploads.

Plus:

I run a: ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.3.122<ma...@169.254.3.122>
and receive a: Permission denied (public key)

I looked at the mysql database for the keys and compared them against the keys on the hypervisors. They do not match.

There has to be a problem with the builds, as I have been able to replicate this multiple times on completely fresh builds.

UB 12.04 on both HV and Cloustack hosts. Running normal apt-get install cloud-agent and cloud-client respectively.

Help!

Trevor Francis
Partner
46 Labs | PeerEdge Cloud Switch (PeCS)
http://www.46labs.com<http://www.46labs.com/> | http://www.peeredge.net<http://www.peeredge.net/>
720-214-3643- Voice
trevor@46labs.com<ma...@46labs.com>

Solutions Provider for the Telecom Industry

[cid:image001.jpg@01CD7EB2.A839EF60][cid:image002.jpg@01CD7EB2.A839EF60]

Re: BAD System VMs in Ubuntu 12.04

Posted by Trevor Francis <tr...@tgrahamcapital.com>.

resolved by a apt-get install cloud-system-iso and an cloud-agent restart.

This should be part of the cloud-agent installation and included in 4.0.1.

Trevor Francis
Partner
46 Labs | PeerEdge Cloud Switch (PeCS)
http://www.46labs.com | http://www.peeredge.net
720-214-3643- Voice
trevor@46labs.com
 
Solutions Provider for the Telecom Industry

 

On Dec 2, 2012, at 2:13 PM, Trevor Francis <tr...@46labs.com> wrote:

> Bug Filed. 
> 
> Can any of my kind KVM brethren weigh in on this?
> 
> Trevor Francis
> Partner
> 46 Labs | PeerEdge Cloud Switch (PeCS)
> http://www.46labs.com | http://www.peeredge.net
> 720-214-3643- Voice
> trevor@46labs.com
>  
> Solutions Provider for the Telecom Industry
> 
> <image001.jpg><image002.jpg>
> 
> On Dec 2, 2012, at 1:59 PM, Trevor Francis <tr...@46labs.com> wrote:
> 
>> Yeap! I thought so. 
>> 
>> I was registered for Vegas, but was dragged down by…..Cloud stuff… I hope, for the KVM Agent's sake that they are enjoying in the festivities.
>> 
>> root@compute0:~# libvirtd --version
>> libvirtd (libvirt) 0.10.2
>> root@compute0:~# kvm --version
>> QEMU emulator version 1.0 (qemu-kvm-1.0), Copyright (c) 2003-2008 Fabrice Bellard
>> 
>> I compiled in Libvirtd 0.10.2 to enable RBD for ceph storage and the native KVM client in 12.04.1 Ubuntu, as of late,  has RBD baked in.
>> 
>> Regards,
>> 
>> 
>> Trevor Francis
>> Partner
>> 46 Labs | PeerEdge Cloud Switch (PeCS)
>> http://www.46labs.com | http://www.peeredge.net
>> 720-214-3643- Voice
>> trevor@46labs.com
>>  
>> Solutions Provider for the Telecom Industry
>> 
>> <image001.jpg><image002.jpg>
>> 
>> On Dec 2, 2012, at 1:49 PM, Rohit Yadav <ro...@citrix.com> wrote:
>> 
>>> Woah, that's is a bug, please file it? Can you also share your libvirt version on your kvm host? With virtsh manager or VMM can you debug using gui? I'm suspecting for some reason, the system vms are not getting patched when they boot, so you can ping them but don't see any agent on them connecting back to the mgmt server.
>>> 
>>> Regards.
>>> PS. you're not in vegas are you? the kvm agent must be high on alcohol :)
>>> ________________________________________
>>> From: Trevor Francis [trevor@46labs.com]
>>> Sent: Monday, December 03, 2012 1:01 AM
>>> To: cloudstack-users@incubator.apache.org
>>> Subject: Re: BAD System VMs in Ubuntu 12.04
>>> 
>>> 1. Check.
>>> 2. Not sure how that works, but running KVM on Ubuntu 12.04
>>> 3. Check.
>>> 4. Check.
>>> 
>>> The SSVM and CPVM spin up fine and I can ping them. Virsh list shows both running. However, neither are accessible via ssh, CPVM is showing "Connection Refused" when I attempt to pull up a console and SSVM isn't downloading templates or ISO.
>>> 
>>> The strange thing is that the cloud-agent logs aren't showing anything strange and the management server isn't showing anything other than:
>>> 
>>> [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
>>> 2012-12-02 19:28:33,167 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
>>> 2012-12-02 19:28:33,453 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers.
>>> 2012-12-02 19:28:49,702 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-8:null) Ping from 1
>>> 2012-12-02 19:29:03,065 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
>>> 2012-12-02 19:29:03,167 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
>>> 2012-12-02 19:29:03,453 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers.
>>> 2012-12-02 19:29:19,290 DEBUG [cloud.server.StatsCollector] (StatsCollector-3:null) VmStatsCollector is running...
>>> 
>>> 
>>> It seems that management server is "ready to launch" CPVM and SSVM, even though both are already showing "running."
>>> 
>>> I have built cloudstack probably 200 times and this is the first time I have every seen this.
>>> 
>>> 
>>> 
>>> Trevor Francis
>>> Partner
>>> 46 Labs | PeerEdge Cloud Switch (PeCS)
>>> http://www.46labs.com<http://www.46labs.com/> | http://www.peeredge.net<http://www.peeredge.net/>
>>> 720-214-3643- Voice
>>> trevor@46labs.com<ma...@46labs.com>
>>> 
>>> Solutions Provider for the Telecom Industry
>>> 
>>> [cid:image001.jpg@01CD7EB2.A839EF60][cid:image002.jpg@01CD7EB2.A839EF60]
>>> 
>>> On Dec 2, 2012, at 12:28 PM, Rohit Yadav <ro...@citrix.com>> wrote:
>>> 
>>> Cannot help, but checklist;
>>> 
>>> 1. Global settings:
>>>   host is mgmt server IP and is reachable
>>>   mgmt server cidr is correct
>>> 2. Systemvm.iso is being sent to your xen host (assuming you're using xen host) etc.
>>> 3. You've ssh-client package install on your mgmt server, sshd on hosts.
>>> 4. You've suitable access to administrate mgmt server
>>> 
>>> Regards.
>>> ________________________________________
>>> From: Trevor Francis [trevor@46labs.com<ma...@46labs.com>]
>>> Sent: Sunday, December 02, 2012 10:36 PM
>>> To: cloudstack-users@incubator.apache.org<ma...@incubator.apache.org>
>>> Subject: Re: BAD System VMs in Ubuntu 12.04
>>> 
>>> Deleted Both SystemVM's and allowed them to rebuild….same issue.
>>> 
>>> Also, I am getting "Connection Refused" when I try to activate the CPVM.
>>> 
>>> 
>>> 
>>> Trevor Francis
>>> Partner
>>> 46 Labs | PeerEdge Cloud Switch (PeCS)
>>> http://www.46labs.com<http://www.46labs.com/> | http://www.peeredge.net<http://www.peeredge.net/>
>>> 720-214-3643- Voice
>>> trevor@46labs.com<ma...@46labs.com>
>>> 
>>> Solutions Provider for the Telecom Industry
>>> 
>>> [cid:image001.jpg@01CD7EB2.A839EF60<ma...@01CD7EB2.A839EF60>]
>>> 
>>> On Dec 2, 2012, at 11:02 AM, Trevor Francis <tr...@tgrahamcapital.com>> wrote:
>>> 
>>> /usr/lib/cloud/common/vms/systemvm.iso
>>> 5972 -rw-r--r-- 1 root 6113280 Dec  2 03:58 systemvm.iso
>>> 
>>> root@management1:/usr/lib/cloud/common/vms# mount -o loop systemvm.iso /media/iso
>>> mount: warning: /media/iso seems to be mounted read-only.
>>> root@management1:/usr/lib/cloud/common/vms# cd /media/iso
>>> root@management1:/media/iso# ls
>>> authorized_keys  cloud-scripts.tgz  systemvm.zip
>>> 
>>> 
>>> 
>>> Trevor Francis
>>> Partner
>>> 46 Labs | PeerEdge Cloud Switch (PeCS)
>>> http://www.46labs.com<http://www.46labs.com/> | http://www.peeredge.net<http://www.peeredge.net/>
>>> 720-214-3643- Voice
>>> trevor@46labs.com<ma...@46labs.com>
>>> 
>>> Solutions Provider for the Telecom Industry
>>> 
>>> <image001.jpg><image002.jpg>
>>> 
>>> On Dec 2, 2012, at 10:36 AM, Rohit Yadav <ro...@citrix.com>> wrote:
>>> 
>>> Just saw the version, so can you check if you have systemvm.iso somewhere in the install path /usr/share or /usr/bin (don't remember actual paths for debian builds) and it is readable and mountable.
>>> 
>>> Regards
>>> ________________________________________
>>> From: Trevor Francis [trevor@46labs.com<ma...@46labs.com>]
>>> Sent: Sunday, December 02, 2012 9:39 PM
>>> To: cloudstack-users@incubator.apache.org<ma...@incubator.apache.org>
>>> Subject: Re: BAD System VMs in Ubuntu 12.04
>>> 
>>> No dice. Same issue. There is something wrong with these VMs or CS.
>>> 
>>> Running Ubuntu 12.04 4.0.0-incubating.20121024195401
>>> 
>>> 
>>> 
>>> 
>>> Trevor Francis
>>> Partner
>>> 46 Labs | PeerEdge Cloud Switch (PeCS)
>>> http://www.46labs.com<http://www.46labs.com/> | http://www.peeredge.net<http://www.peeredge.net/>
>>> 720-214-3643- Voice
>>> trevor@46labs.com<ma...@46labs.com>
>>> 
>>> Solutions Provider for the Telecom Industry
>>> 
>>> [cid:image001.jpg@01CD7EB2.A839EF60<ma...@01CD7EB2.A839EF60>]
>>> 
>>> On Dec 2, 2012, at 9:46 AM, Ogiljae <og...@gmail.com>> wrote:
>>> 
>>> Why don't you try deleting ssh key called id_rsa.cloud under .ssh/ of host where SSVM is running on and do the force reconnection with the host on UI.
>>> 
>>> 
>>> James Oh
>>> Cloud QA Team
>>> KT
>>> Mobile : 010-9549-5588
>>> FAX : 0303-0993-2884
>>> Email : ogiljae@kt.com<ma...@kt.com>
>>> 
>>> On 2012. 12. 3., at 오전 12:26, Trevor Francis <tr...@46labs.com>> wrote:
>>> 
>>> I have recently completely rebuilding cloudstack and a kvm hypervisor from scratch to determine if there is an issue with my system or with the latest Ubuntu cloudstack build.
>>> 
>>> Currently, cloudstack builds bad systems VMs on Ubuntu 12.04 Hosts and Management server. The system VM is pingable, but I am getting "Connection Refused" on CPVM and the SSVM will not respond to ISO uploads.
>>> 
>>> Plus:
>>> 
>>> I run a: ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@169.254.3.122<ma...@169.254.3.122>
>>> and receive a: Permission denied (public key)
>>> 
>>> I looked at the mysql database for the keys and compared them against the keys on the hypervisors. They do not match
>>> There has to be a problem with the builds, as I have been able to replicate this multiple times on completely fresh builds.
>>> 
>>> UB 12.04 on both HV and Cloustack hosts. Running normal apt-get install cloud-agent and cloud-client respectively.
>>> 
>>> Help!
>>> 
>>> Trevor Francis
>>> Partner
>>> 46 Labs | PeerEdge Cloud Switch (PeCS)
>>> http://www.46labs.com<http://www.46labs.com/> | http://www.peeredge.net<http://www.peeredge.net/>
>>> 720-214-3643- Voice
>>> trevor@46labs.com<ma...@46labs.com>
>>> 
>>> Solutions Provider for the Telecom Industry
>>> 
>>> <image001.jpg><image002.jpg>
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
>