You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Rohit Yadav <ro...@citrix.com> on 2012/07/25 11:43:47 UTC
Development with multiple devclouds
Hi Edison and everyone,
Devcloud works fine as one hypervisor/xen host. But, in order to develop/test certain features of CS such as migration among hosts in a cluster, you would require at least two hypervisor hosts.
To solve this problem, using VBox's NAT and host-only adapter I tried various combinations, but none of them are working.
I tried VBox's; 1. host-only adapter in place of NAT, keeping the same gateway and network (my system crashed); 2. the default NAT + one host-only adapter; with another devcloud vm which is only used as a hypervisor/xen host.
@Edison: On the wiki page, you mention not to reinitialize the NAT. Is the NAT's mac and dom0/xen's IP values tightly coupled with the present devcloud image?
How can I run run multiple devclouds/vms on VBox such that only one is used for management/storage/xen and the others are just used as xen hosts (as within a cluster the hypervisors hosts should be homogenous, using the same devcloud vm should be fine).
To solve this problem, now I've two separate Citrix XenServer vms on VBox and trying to setup a management server/nfs-storage running on a third Ubuntu desktop VM on VBox.
Comment/suggestions on this issue and my approach now?
Thanks.
Best regards,
Rohit
Re: Development with multiple devclouds
Posted by Edison Su <Ed...@citrix.com>.
Sent from my iPhone
On Jul 30, 2012, at 10:14 PM, "Rohit Yadav" <ro...@citrix.com> wrote:
> Hi,
>
>>>
>>> I tried to import and use two devclouds, but for some reason I was not
>>> able to add the first host itself. Please help debug, if at all this
>>> idea seems worth exploring. Also, as Chiradeep suggested in an email
>>> previously, would n't be a good option to have a single XenServer/NFS
>>> as DevCloud and Mgmt server/MySQL on host os?
>> Yes, that's a good option. But there are potential issues:
>> 1. mgt server connects to hypervisor host using port 22.
>> 2. mgt server accesses XAPI using port 80.
>> Both port 22 and 80/443 can't be port-forwarded(at least under Linux host OS) when VirtualBox working in NAT mode.
>> To get around, need to change cloudstack source code to using port beyond 1024.
>
> Yes, that's true for NAT, but in my case I'm trying to use host-only adapter as primary interface and a second NAT adapter for DNS/Internet. Is it possible to use one as host-only (so both ssh and XAPI are reachable) and second one as NAT (so, DNS and Internet works) for DevCloud (this works for me in the decoupled setup as explained in the wiki but not using default DevCloud image)? Thanks.
That's a good idea, host-only network as the private/guest/management network, NAT as the public network.
>
> Best regards,
> Rohit Yadav
>
>>
>>>
>>> Thanks.
>>>
>>> Best regards,
>>> Rohit Yadav
>
Re: Development with multiple devclouds
Posted by Rohit Yadav <ro...@citrix.com>.
Hi,
>>
>> I tried to import and use two devclouds, but for some reason I was not
>> able to add the first host itself. Please help debug, if at all this
>> idea seems worth exploring. Also, as Chiradeep suggested in an email
>> previously, would n't be a good option to have a single XenServer/NFS
>> as DevCloud and Mgmt server/MySQL on host os?
> Yes, that's a good option. But there are potential issues:
> 1. mgt server connects to hypervisor host using port 22.
> 2. mgt server accesses XAPI using port 80.
> Both port 22 and 80/443 can't be port-forwarded(at least under Linux host OS) when VirtualBox working in NAT mode.
> To get around, need to change cloudstack source code to using port beyond 1024.
Yes, that's true for NAT, but in my case I'm trying to use host-only adapter as primary interface and a second NAT adapter for DNS/Internet. Is it possible to use one as host-only (so both ssh and XAPI are reachable) and second one as NAT (so, DNS and Internet works) for DevCloud (this works for me in the decoupled setup as explained in the wiki but not using default DevCloud image)? Thanks.
Best regards,
Rohit Yadav
>
>>
>> Thanks.
>>
>> Best regards,
>> Rohit Yadav
RE: Development with multiple devclouds
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Rohit Yadav
> Sent: Monday, July 30, 2012 5:12 AM
> To: Edison Su
> Cc: cloudstack-dev@incubator.apache.org
> Subject: Re: Development with multiple devclouds
>
> Hi Edison,
>
> > Thanks for your sharing. Could you help to write up a wiki page about
> multiple devcloud setup?
>
> Yes, wrote a page here:
> http://wiki.cloudstack.org/display/COMM/DIY+DevCloud+Setup
>
> Though the proof of concept setup works (with a lot of system crashes)
> and the idea of multiple devcloud seems interesting for trying out
> advance functions, this kind of setup requires a lot of RAM (in my case
> 8GB was inadequate), and may not be the best environment for new
> developers. If possible it would be better to get an extra
> computer/server.
>
> I tried to import and use two devclouds, but for some reason I was not
> able to add the first host itself. Please help debug, if at all this
> idea seems worth exploring. Also, as Chiradeep suggested in an email
> previously, would n't be a good option to have a single XenServer/NFS
> as DevCloud and Mgmt server/MySQL on host os?
Yes, that's a good option. But there are potential issues:
1. mgt server connects to hypervisor host using port 22.
2. mgt server accesses XAPI using port 80.
Both port 22 and 80/443 can't be port-forwarded(at least under Linux host OS) when VirtualBox working in NAT mode.
To get around, need to change cloudstack source code to using port beyond 1024.
>
> Thanks.
>
> Best regards,
> Rohit Yadav
Re: Development with multiple devclouds
Posted by Rohit Yadav <ro...@citrix.com>.
Hi Edison,
> Thanks for your sharing. Could you help to write up a wiki page about multiple devcloud setup?
Yes, wrote a page here: http://wiki.cloudstack.org/display/COMM/DIY+DevCloud+Setup
Though the proof of concept setup works (with a lot of system crashes) and the idea of multiple devcloud seems interesting for trying out advance functions, this kind of setup requires a lot of RAM (in my case 8GB was inadequate), and may not be the best environment for new developers. If possible it would be better to get an extra computer/server.
I tried to import and use two devclouds, but for some reason I was not able to add the first host itself. Please help debug, if at all this idea seems worth exploring. Also, as Chiradeep suggested in an email previously, would n't be a good option to have a single XenServer/NFS as DevCloud and Mgmt server/MySQL on host os?
Thanks.
Best regards,
Rohit Yadav
RE: Development with multiple devclouds
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Rohit Yadav
> Sent: Friday, July 27, 2012 4:57 AM
> To: Edison Su
> Cc: cloudstack-dev@incubator.apache.org
> Subject: Re: Development with multiple devclouds
>
> Hi,
>
> I found a solution:
>
> >> The setup seems to work and the system vms get deployed on one of
> the xenserver vm, but the SSVM fails to work. I ssh into the SSVM using
> it's link-local IP via its host xenserver's dom0, and find that the
> SSVM is unable to reach the gateway, 192.168.0.1. All routes, iptables
> look fine, comments?
> >
> > Which bridge is used by the system vm ?
>
> The system vm, specifically ssvm, uses four vifs, one as link-local
> eth0 with xapi1 interface and other three (eth1,eth2,eth3) with xenbr0.
>
> I'm able to ping on ssvm's eth1 from xenserver and vice-versa. And also
> from my host os to my xenserver and vice versa. When I ping from my
> ssvm to my gateway (192.168.0.1, host os), it fails. The tcpdump on
> these interfaces show that arp requests from ssvm reach the gateway and
> replies are sent and are very slow. And no icmp reaches the gateway
> from ssvm and vice-versa. The host-only adapter attached to XenServer
> vm was set by default in promiscuous: deny mode.
>
> So, we need to set the host-only adapter's promiscuous mode to 'allow
> all' and configure global settings (in my case local storage to 'true',
> mgmt host '192.168.0.1' otherwise ssvm fails) as needed. This should
> work with multiple devclouds as well.
Thanks for your sharing. Could you help to write up a wiki page about multiple devcloud setup? You are already ahead of the curve.
>
> > "brctl show" on each of your hyper visor vm
>
> Detailed log of xenserver vm, jfyi: http://pastebin.com/1htDq44J
>
> Regards,
> Rohit
Re: Development with multiple devclouds
Posted by Rohit Yadav <ro...@citrix.com>.
Hi,
I found a solution:
>> The setup seems to work and the system vms get deployed on one of the xenserver vm, but the SSVM fails to work. I ssh into the SSVM using it's link-local IP via its host xenserver's dom0, and find that the SSVM is unable to reach the gateway, 192.168.0.1. All routes, iptables look fine, comments?
>
> Which bridge is used by the system vm ?
The system vm, specifically ssvm, uses four vifs, one as link-local eth0 with xapi1 interface and other three (eth1,eth2,eth3) with xenbr0.
I'm able to ping on ssvm's eth1 from xenserver and vice-versa. And also from my host os to my xenserver and vice versa. When I ping from my ssvm to my gateway (192.168.0.1, host os), it fails. The tcpdump on these interfaces show that arp requests from ssvm reach the gateway and replies are sent and are very slow. And no icmp reaches the gateway from ssvm and vice-versa. The host-only adapter attached to XenServer vm was set by default in promiscuous: deny mode.
So, we need to set the host-only adapter's promiscuous mode to 'allow all' and configure global settings (in my case local storage to 'true', mgmt host '192.168.0.1' otherwise ssvm fails) as needed. This should work with multiple devclouds as well.
> "brctl show" on each of your hyper visor vm
Detailed log of xenserver vm, jfyi: http://pastebin.com/1htDq44J
Regards,
Rohit
Re: Development with multiple devclouds
Posted by Edison Su <Ed...@citrix.com>.
Sent from my iPhone
On Jul 26, 2012, at 7:23 AM, "Rohit Yadav" <ro...@citrix.com> wrote:
> Hi,
>
>>> The following combination should work:
>>> 1. one devcloud VM has one NAT nic + one host-only nic
>>> 2. another one devcloud VM has one host-only nic, which uses the same
>>> host-only network as above devcloud.
>>>
>>> The first devcloud VM acts as both mgt server and xen host, the second
>>> one just as a xen host.
>>> The changes need to make in both VMs:
>>> In first devcloud VM, put xenbr0 on eth1(I assume it's the host-only
>>> network), instead of eth0.
>>
>> One more thing...: Need to join both vm1 and vm2 into a cluster with "xe pool-join"
>> I'll have a try by myself at first, then give you a detailed instruction.
>
> Thanks, looking forward to your solution.
>
> I tried to do that, for some reason it failed to work for me. So, as a proof of concept I made a setup from scratch in which my development env and mgmt server/db is on host os and the hypervisor runs separately on VBox. This kind of setup may be preferable than a monolithic one?
>
> Just want to share my approach and problems I have with it. The setup:
>
> Two separate default citrix xenserver 6.0.2 (2GB RAM each) running inside VBox, one ubuntu linux vm (512MB RAM) to serve as NFS secondary storage, 'use.local.storage' and 'system.vm.local.storage' set to true in cloudstack's global settings. Only host-only network 192.168.0.x/vboxnet0, one NIC on each VMs, is used.
>
> My development environment and mgmt server/mysql runs on my host os, mac osx 10.7.4/8GB RAM using latest apache-cloudstack-git. In this setup, after setting up my CS mgmt server I'm able to configure a basic zone with basic shared network, using the same IP values as given on wiki, replacing 10.0.2.x with 192.168.0.x.
>
> The setup seems to work and the system vms get deployed on one of the xenserver vm, but the SSVM fails to work. I ssh into the SSVM using it's link-local IP via its host xenserver's dom0, and find that the SSVM is unable to reach the gateway, 192.168.0.1. All routes, iptables look fine, comments?
Which bridge is used by the system vm ?
"brctl show" on each of your hyper visor vm
>
> Thanks.
>
> Regards,
> Rohit Yadav
Re: Development with multiple devclouds
Posted by Rohit Yadav <ro...@citrix.com>.
Hi,
>> The following combination should work:
>> 1. one devcloud VM has one NAT nic + one host-only nic
>> 2. another one devcloud VM has one host-only nic, which uses the same
>> host-only network as above devcloud.
>>
>> The first devcloud VM acts as both mgt server and xen host, the second
>> one just as a xen host.
>> The changes need to make in both VMs:
>> In first devcloud VM, put xenbr0 on eth1(I assume it's the host-only
>> network), instead of eth0.
>
> One more thing...: Need to join both vm1 and vm2 into a cluster with "xe pool-join"
> I'll have a try by myself at first, then give you a detailed instruction.
Thanks, looking forward to your solution.
I tried to do that, for some reason it failed to work for me. So, as a proof of concept I made a setup from scratch in which my development env and mgmt server/db is on host os and the hypervisor runs separately on VBox. This kind of setup may be preferable than a monolithic one?
Just want to share my approach and problems I have with it. The setup:
Two separate default citrix xenserver 6.0.2 (2GB RAM each) running inside VBox, one ubuntu linux vm (512MB RAM) to serve as NFS secondary storage, 'use.local.storage' and 'system.vm.local.storage' set to true in cloudstack's global settings. Only host-only network 192.168.0.x/vboxnet0, one NIC on each VMs, is used.
My development environment and mgmt server/mysql runs on my host os, mac osx 10.7.4/8GB RAM using latest apache-cloudstack-git. In this setup, after setting up my CS mgmt server I'm able to configure a basic zone with basic shared network, using the same IP values as given on wiki, replacing 10.0.2.x with 192.168.0.x.
The setup seems to work and the system vms get deployed on one of the xenserver vm, but the SSVM fails to work. I ssh into the SSVM using it's link-local IP via its host xenserver's dom0, and find that the SSVM is unable to reach the gateway, 192.168.0.1. All routes, iptables look fine, comments?
Thanks.
Regards,
Rohit Yadav
Re: Development with multiple devclouds
Posted by Chiradeep Vittal <Ch...@citrix.com>.
I actually would prefer to continue developing/running on my laptop and
use the xen hosts only as hypervisors. So in the 2 dom0 case below, I feel
it is less awkward to use the native dev environment.
Where running the code on inside vBox helps is that developers always run
"ant run" or "ant debug" rather than do the full rpm deploy and install.
Being able to use DevCloud as the final verification environment for such
a deployment pre-check-in would help a lot.
--
Chiradeep
On 7/25/12 11:11 AM, "Edison Su" <Ed...@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: Edison Su [mailto:Edison.su@citrix.com]
>> Sent: Wednesday, July 25, 2012 10:51 AM
>> To: Rohit Yadav; cloudstack-dev@incubator.apache.org
>> Subject: RE: Development with multiple devclouds
>>
>>
>>
>> > -----Original Message-----
>> > From: Rohit Yadav
>> > Sent: Wednesday, July 25, 2012 2:44 AM
>> > To: Edison Su; cloudstack-dev@incubator.apache.org
>> > Subject: Development with multiple devclouds
>> >
>> > Hi Edison and everyone,
>> >
>> > Devcloud works fine as one hypervisor/xen host. But, in order to
>> > develop/test certain features of CS such as migration among hosts in
>> a
>> > cluster, you would require at least two hypervisor hosts.
>> >
>> > To solve this problem, using VBox's NAT and host-only adapter I tried
>> > various combinations, but none of them are working.
>> >
>> > I tried VBox's; 1. host-only adapter in place of NAT, keeping the
>> same
>> > gateway and network (my system crashed); 2. the default NAT + one
>> > host-only adapter; with another devcloud vm which is only used as a
>> > hypervisor/xen host.
>>
>> The last method(NAT + one host-only adapter) should work:
>>
>> >
>> > @Edison: On the wiki page, you mention not to reinitialize the NAT.
>> Is
>> > the NAT's mac and dom0/xen's IP values tightly coupled with the
>> present
>> > devcloud image?
>>
>> If the MAC address is changed, then the default eth0 will be disabled,
>> as there is a udev rule
>> In devcloud, which binds eth0 on a specific MAC address. I will remove
>> the rule in the next devcloud release.
>> FYI, the rule is in /etc/udev/rules.d/70-persistent-net.rules, you can
>> remove it by yourself, then restart devcloud,
>> Eth0 will be enabled again.
>>
>> >
>> > How can I run run multiple devclouds/vms on VBox such that only one
>> is
>> > used for management/storage/xen and the others are just used as xen
>> > hosts (as within a cluster the hypervisors hosts should be homogenous,
>> > using the same devcloud vm should be fine).
>> >
>> > To solve this problem, now I've two separate Citrix XenServer vms on
>> > VBox and trying to setup a management server/nfs-storage running on a
>> > third Ubuntu desktop VM on VBox.
>> > Comment/suggestions on this issue and my approach now?
>>
>> The following combination should work:
>> 1. one devcloud VM has one NAT nic + one host-only nic
>> 2. another one devcloud VM has one host-only nic, which uses the same
>> host-only network as above devcloud.
>>
>> The first devcloud VM acts as both mgt server and xen host, the second
>> one just as a xen host.
>> The changes need to make in both VMs:
>> In first devcloud VM, put xenbr0 on eth1(I assume it's the host-only
>> network), instead of eth0.
>
>One more thing...: Need to join both vm1 and vm2 into a cluster with "xe
>pool-join"
>I'll have a try by myself at first, then give you a detailed instruction.
>
>>
>> >
>> > Thanks.
>> >
>> > Best regards,
>> > Rohit
RE: Development with multiple devclouds
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Edison Su [mailto:Edison.su@citrix.com]
> Sent: Wednesday, July 25, 2012 10:51 AM
> To: Rohit Yadav; cloudstack-dev@incubator.apache.org
> Subject: RE: Development with multiple devclouds
>
>
>
> > -----Original Message-----
> > From: Rohit Yadav
> > Sent: Wednesday, July 25, 2012 2:44 AM
> > To: Edison Su; cloudstack-dev@incubator.apache.org
> > Subject: Development with multiple devclouds
> >
> > Hi Edison and everyone,
> >
> > Devcloud works fine as one hypervisor/xen host. But, in order to
> > develop/test certain features of CS such as migration among hosts in
> a
> > cluster, you would require at least two hypervisor hosts.
> >
> > To solve this problem, using VBox's NAT and host-only adapter I tried
> > various combinations, but none of them are working.
> >
> > I tried VBox's; 1. host-only adapter in place of NAT, keeping the
> same
> > gateway and network (my system crashed); 2. the default NAT + one
> > host-only adapter; with another devcloud vm which is only used as a
> > hypervisor/xen host.
>
> The last method(NAT + one host-only adapter) should work:
>
> >
> > @Edison: On the wiki page, you mention not to reinitialize the NAT.
> Is
> > the NAT's mac and dom0/xen's IP values tightly coupled with the
> present
> > devcloud image?
>
> If the MAC address is changed, then the default eth0 will be disabled,
> as there is a udev rule
> In devcloud, which binds eth0 on a specific MAC address. I will remove
> the rule in the next devcloud release.
> FYI, the rule is in /etc/udev/rules.d/70-persistent-net.rules, you can
> remove it by yourself, then restart devcloud,
> Eth0 will be enabled again.
>
> >
> > How can I run run multiple devclouds/vms on VBox such that only one
> is
> > used for management/storage/xen and the others are just used as xen
> > hosts (as within a cluster the hypervisors hosts should be homogenous,
> > using the same devcloud vm should be fine).
> >
> > To solve this problem, now I've two separate Citrix XenServer vms on
> > VBox and trying to setup a management server/nfs-storage running on a
> > third Ubuntu desktop VM on VBox.
> > Comment/suggestions on this issue and my approach now?
>
> The following combination should work:
> 1. one devcloud VM has one NAT nic + one host-only nic
> 2. another one devcloud VM has one host-only nic, which uses the same
> host-only network as above devcloud.
>
> The first devcloud VM acts as both mgt server and xen host, the second
> one just as a xen host.
> The changes need to make in both VMs:
> In first devcloud VM, put xenbr0 on eth1(I assume it's the host-only
> network), instead of eth0.
One more thing...: Need to join both vm1 and vm2 into a cluster with "xe pool-join"
I'll have a try by myself at first, then give you a detailed instruction.
>
> >
> > Thanks.
> >
> > Best regards,
> > Rohit
RE: Development with multiple devclouds
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Rohit Yadav
> Sent: Wednesday, July 25, 2012 2:44 AM
> To: Edison Su; cloudstack-dev@incubator.apache.org
> Subject: Development with multiple devclouds
>
> Hi Edison and everyone,
>
> Devcloud works fine as one hypervisor/xen host. But, in order to
> develop/test certain features of CS such as migration among hosts in a
> cluster, you would require at least two hypervisor hosts.
>
> To solve this problem, using VBox's NAT and host-only adapter I tried
> various combinations, but none of them are working.
>
> I tried VBox's; 1. host-only adapter in place of NAT, keeping the same
> gateway and network (my system crashed); 2. the default NAT + one
> host-only adapter; with another devcloud vm which is only used as a
> hypervisor/xen host.
The last method(NAT + one host-only adapter) should work:
>
> @Edison: On the wiki page, you mention not to reinitialize the NAT. Is
> the NAT's mac and dom0/xen's IP values tightly coupled with the present
> devcloud image?
If the MAC address is changed, then the default eth0 will be disabled, as there is a udev rule
In devcloud, which binds eth0 on a specific MAC address. I will remove the rule in the next devcloud release.
FYI, the rule is in /etc/udev/rules.d/70-persistent-net.rules, you can remove it by yourself, then restart devcloud,
Eth0 will be enabled again.
>
> How can I run run multiple devclouds/vms on VBox such that only one is
> used for management/storage/xen and the others are just used as xen
> hosts (as within a cluster the hypervisors hosts should be homogenous,
> using the same devcloud vm should be fine).
>
> To solve this problem, now I've two separate Citrix XenServer vms on
> VBox and trying to setup a management server/nfs-storage running on a
> third Ubuntu desktop VM on VBox.
> Comment/suggestions on this issue and my approach now?
The following combination should work:
1. one devcloud VM has one NAT nic + one host-only nic
2. another one devcloud VM has one host-only nic, which uses the same host-only network as above devcloud.
The first devcloud VM acts as both mgt server and xen host, the second one just as a xen host.
The changes need to make in both VMs:
In first devcloud VM, put xenbr0 on eth1(I assume it's the host-only network), instead of eth0.
>
> Thanks.
>
> Best regards,
> Rohit