You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Alex Jiang <al...@gmail.com> on 2012/11/13 06:32:58 UTC
[Help] Use Ceph RBD as primary storage in CloudStack
Hi, All
Has somebody used Ceph RBD in CloudStack as primary storage? I see that in
the new features of CS 4.0, RBD is supported for KVM. So I tried using RBD
as primary storage but met with some problems.
I use a CentOS6.3 server as host. First I erase the qemu-kvm(0.12.1) and
libvirt(0.9.10) because their versions are too low (Qemu on the Hypervisor
has to be compiled with RBD enabled .The libvirt version on the Hypervisor
has to be at least 0.10 with RBD enabled).Then I download the latest
qemu(1.2.0) and libvirt(1.0.0) source code and compile and install them.
But when compiling qemu source code,
#wget http://wiki.qemu-project.org/download/qemu-1.2.0.tar.bz2
#tar jxvf qemu-1.2.0.tar.bz2
# cd qemu-1.2.0
# ./configure --enable-rbd
the following errors occur:
ERROR: User requested feature rados block device
ERROR: configure was not able to find it
But on Ubuntu12.04 I tried compiling qemu source code and succeed.Now I am
very confused.How to use Ceph RBD as primary storage in CloudStack on
CentOS6.3?Anyone can help me?
Best Regards,
Alex
Re: [Help] Use Ceph RBD as primary storage in CloudStack
Posted by Alex Jiang <al...@gmail.com>.
Hi, Prasanna
I did not make any customizations to the procedure.You only need to
put the source code in the right path when building Ceph.
The errors I mentioned above occur because qemu is looking for the
Ceph libraries and not finding them.The library is ceph-devel.x86_64.
So I think for those users who don't want to build Ceph can just
install ceph-devel rpm packages(yum install ceph-devel.x86_64).
Best Regards,
Alex
2012/11/14 Prasanna Santhanam <pr...@citrix.com>:
> On Wed, Nov 14, 2012 at 02:39:41PM +0530, Alex Jiang wrote:
>> Hi, Marcus
>> Thank you for your reply.Now I can compile Qemu with RBD enable and
>> have added the host to CloudStack successfully.
>>
> Can the below steps be recorded as is on our wiki for setting up ceph?
> Or did you have to make any customizations to the procedure?
>
>> 2012/11/13 Marcus Sorensen <sh...@gmail.com>:
>> > Ok, doesn't seem to be built with RBD, but here's a quick rundown on how to
>> > do it. I just ran through this and then tried to make sense of my history,
>> > so hopefully you can adjust it to work for your environment. I built ceph
>> > rpms since I imagine you'll want them if you deploy it elsewhere.
>> >
>> > get dependencies
>> >
>> > - from yum:
>> > - yum install automake libtool nss nss-devel fuse-devel gcc-c++
>> > libatomic_ops-devel libaio-devel boost* libcurl-devel libxml2-devel
>> > expat-devel rpm-build gcc
>> > - You may need to do more packages depending on your install
>> > - from epel:
>> > - rpm -ivh
>> > http://dl.fedoraproject.org/pub/epel/6/x86_64/fcgi-2.4.0-10.el6.x86_64.rpm
>> > http://dl.fedoraproject.org/pub/epel/6/x86_64/fcgi-devel-2.4.0-10.el6.x86_64.rpm
>> > - from source:
>> > - wget
>> > http://download.savannah.gnu.org/releases/libunwind/libunwind-0.99-beta.tar.gz
>> > - tar xzf libunwind-0.99-beta.tar.gz
>> > - cd libunwind-0.99-beta
>> > - ./configure; make; make install
>> > - wget http://gperftools.googlecode.com/files/gperftools-2.0.tar.gz
>> > - tar xzf gperftools-2.0.tar.gz
>> > - cd gperftools-2.0
>> > - ./configure; make; make install
>> >
>> >
>> > build ceph:
>> >
>> > - git clone --recursive https://github.com/ceph/ceph.git
>> > - cd ceph
>> > - ./autogen.sh
>> > - ./configure
>> > - cd ..
>> > - mv ceph ceph-0.53
>> > - tar cjf ceph-0.53.tar.bz2 /root/rpmbuild/SOURCES
>> > - rpmbuild -ba ceph-0.53/ceph.spec
>> > - rpm -ivh /root/rpmbuild/RPMS/x86_64/ceph-devel*
>> >
>> > Now your ./configure of qemu should work. Some notes... you probably don't
>> > want to build RPMs as root. Also, this builds ceph with NSS, which from
>> > what I understand isn't preferable to cryptopp. I installed cryptopp and
>> > cryptopp-devel from EPEL, but it didn't automatically find it so I moved
>> > along in the interest of time.
>> >
>> >
>> > On Mon, Nov 12, 2012 at 10:37 PM, Marcus Sorensen <sh...@gmail.com>wrote:
>> >
>> >> Keep in mind that the version numbers on the Cent 6.3 packages mean
>> >> nothing. They are far ahead and patched beyond what they say. For example
>> >> qemu-kvm version is in reality built off of 1.0 I will look and see if the
>> >> stock ones are built with rbd.
>> >> On Nov 12, 2012 10:33 PM, "Alex Jiang" <al...@gmail.com> wrote:
>> >>
>> >>> Hi, All
>> >>>
>> >>> Has somebody used Ceph RBD in CloudStack as primary storage? I see that in
>> >>> the new features of CS 4.0, RBD is supported for KVM. So I tried using RBD
>> >>> as primary storage but met with some problems.
>> >>>
>> >>> I use a CentOS6.3 server as host. First I erase the qemu-kvm(0.12.1) and
>> >>> libvirt(0.9.10) because their versions are too low (Qemu on the Hypervisor
>> >>> has to be compiled with RBD enabled .The libvirt version on the Hypervisor
>> >>> has to be at least 0.10 with RBD enabled).Then I download the latest
>> >>> qemu(1.2.0) and libvirt(1.0.0) source code and compile and install them.
>> >>> But when compiling qemu source code,
>> >>> #wget http://wiki.qemu-project.org/download/qemu-1.2.0.tar.bz2
>> >>> #tar jxvf qemu-1.2.0.tar.bz2
>> >>> # cd qemu-1.2.0
>> >>> # ./configure --enable-rbd
>> >>>
>> >>> the following errors occur:
>> >>> ERROR: User requested feature rados block device
>> >>> ERROR: configure was not able to find it
>> >>>
>> >>> But on Ubuntu12.04 I tried compiling qemu source code and succeed.Now I am
>> >>> very confused.How to use Ceph RBD as primary storage in CloudStack on
>> >>> CentOS6.3?Anyone can help me?
>> >>>
>> >>> Best Regards,
>> >>>
>> >>> Alex
>> >>>
>> >>
>
> --
> Prasanna.,
Re: [Help] Use Ceph RBD as primary storage in CloudStack
Posted by Prasanna Santhanam <pr...@citrix.com>.
On Wed, Nov 14, 2012 at 02:39:41PM +0530, Alex Jiang wrote:
> Hi, Marcus
> Thank you for your reply.Now I can compile Qemu with RBD enable and
> have added the host to CloudStack successfully.
>
Can the below steps be recorded as is on our wiki for setting up ceph?
Or did you have to make any customizations to the procedure?
> 2012/11/13 Marcus Sorensen <sh...@gmail.com>:
> > Ok, doesn't seem to be built with RBD, but here's a quick rundown on how to
> > do it. I just ran through this and then tried to make sense of my history,
> > so hopefully you can adjust it to work for your environment. I built ceph
> > rpms since I imagine you'll want them if you deploy it elsewhere.
> >
> > get dependencies
> >
> > - from yum:
> > - yum install automake libtool nss nss-devel fuse-devel gcc-c++
> > libatomic_ops-devel libaio-devel boost* libcurl-devel libxml2-devel
> > expat-devel rpm-build gcc
> > - You may need to do more packages depending on your install
> > - from epel:
> > - rpm -ivh
> > http://dl.fedoraproject.org/pub/epel/6/x86_64/fcgi-2.4.0-10.el6.x86_64.rpm
> > http://dl.fedoraproject.org/pub/epel/6/x86_64/fcgi-devel-2.4.0-10.el6.x86_64.rpm
> > - from source:
> > - wget
> > http://download.savannah.gnu.org/releases/libunwind/libunwind-0.99-beta.tar.gz
> > - tar xzf libunwind-0.99-beta.tar.gz
> > - cd libunwind-0.99-beta
> > - ./configure; make; make install
> > - wget http://gperftools.googlecode.com/files/gperftools-2.0.tar.gz
> > - tar xzf gperftools-2.0.tar.gz
> > - cd gperftools-2.0
> > - ./configure; make; make install
> >
> >
> > build ceph:
> >
> > - git clone --recursive https://github.com/ceph/ceph.git
> > - cd ceph
> > - ./autogen.sh
> > - ./configure
> > - cd ..
> > - mv ceph ceph-0.53
> > - tar cjf ceph-0.53.tar.bz2 /root/rpmbuild/SOURCES
> > - rpmbuild -ba ceph-0.53/ceph.spec
> > - rpm -ivh /root/rpmbuild/RPMS/x86_64/ceph-devel*
> >
> > Now your ./configure of qemu should work. Some notes... you probably don't
> > want to build RPMs as root. Also, this builds ceph with NSS, which from
> > what I understand isn't preferable to cryptopp. I installed cryptopp and
> > cryptopp-devel from EPEL, but it didn't automatically find it so I moved
> > along in the interest of time.
> >
> >
> > On Mon, Nov 12, 2012 at 10:37 PM, Marcus Sorensen <sh...@gmail.com>wrote:
> >
> >> Keep in mind that the version numbers on the Cent 6.3 packages mean
> >> nothing. They are far ahead and patched beyond what they say. For example
> >> qemu-kvm version is in reality built off of 1.0 I will look and see if the
> >> stock ones are built with rbd.
> >> On Nov 12, 2012 10:33 PM, "Alex Jiang" <al...@gmail.com> wrote:
> >>
> >>> Hi, All
> >>>
> >>> Has somebody used Ceph RBD in CloudStack as primary storage? I see that in
> >>> the new features of CS 4.0, RBD is supported for KVM. So I tried using RBD
> >>> as primary storage but met with some problems.
> >>>
> >>> I use a CentOS6.3 server as host. First I erase the qemu-kvm(0.12.1) and
> >>> libvirt(0.9.10) because their versions are too low (Qemu on the Hypervisor
> >>> has to be compiled with RBD enabled .The libvirt version on the Hypervisor
> >>> has to be at least 0.10 with RBD enabled).Then I download the latest
> >>> qemu(1.2.0) and libvirt(1.0.0) source code and compile and install them.
> >>> But when compiling qemu source code,
> >>> #wget http://wiki.qemu-project.org/download/qemu-1.2.0.tar.bz2
> >>> #tar jxvf qemu-1.2.0.tar.bz2
> >>> # cd qemu-1.2.0
> >>> # ./configure --enable-rbd
> >>>
> >>> the following errors occur:
> >>> ERROR: User requested feature rados block device
> >>> ERROR: configure was not able to find it
> >>>
> >>> But on Ubuntu12.04 I tried compiling qemu source code and succeed.Now I am
> >>> very confused.How to use Ceph RBD as primary storage in CloudStack on
> >>> CentOS6.3?Anyone can help me?
> >>>
> >>> Best Regards,
> >>>
> >>> Alex
> >>>
> >>
--
Prasanna.,
Re: [Help] Use Ceph RBD as primary storage in CloudStack
Posted by Alex Jiang <al...@gmail.com>.
Hi, Marcus
Thank you for your reply.Now I can compile Qemu with RBD enable and
have added the host to CloudStack successfully.
2012/11/13 Marcus Sorensen <sh...@gmail.com>:
> Ok, doesn't seem to be built with RBD, but here's a quick rundown on how to
> do it. I just ran through this and then tried to make sense of my history,
> so hopefully you can adjust it to work for your environment. I built ceph
> rpms since I imagine you'll want them if you deploy it elsewhere.
>
> get dependencies
>
> - from yum:
> - yum install automake libtool nss nss-devel fuse-devel gcc-c++
> libatomic_ops-devel libaio-devel boost* libcurl-devel libxml2-devel
> expat-devel rpm-build gcc
> - You may need to do more packages depending on your install
> - from epel:
> - rpm -ivh
> http://dl.fedoraproject.org/pub/epel/6/x86_64/fcgi-2.4.0-10.el6.x86_64.rpm
> http://dl.fedoraproject.org/pub/epel/6/x86_64/fcgi-devel-2.4.0-10.el6.x86_64.rpm
> - from source:
> - wget
> http://download.savannah.gnu.org/releases/libunwind/libunwind-0.99-beta.tar.gz
> - tar xzf libunwind-0.99-beta.tar.gz
> - cd libunwind-0.99-beta
> - ./configure; make; make install
> - wget http://gperftools.googlecode.com/files/gperftools-2.0.tar.gz
> - tar xzf gperftools-2.0.tar.gz
> - cd gperftools-2.0
> - ./configure; make; make install
>
>
> build ceph:
>
> - git clone --recursive https://github.com/ceph/ceph.git
> - cd ceph
> - ./autogen.sh
> - ./configure
> - cd ..
> - mv ceph ceph-0.53
> - tar cjf ceph-0.53.tar.bz2 /root/rpmbuild/SOURCES
> - rpmbuild -ba ceph-0.53/ceph.spec
> - rpm -ivh /root/rpmbuild/RPMS/x86_64/ceph-devel*
>
> Now your ./configure of qemu should work. Some notes... you probably don't
> want to build RPMs as root. Also, this builds ceph with NSS, which from
> what I understand isn't preferable to cryptopp. I installed cryptopp and
> cryptopp-devel from EPEL, but it didn't automatically find it so I moved
> along in the interest of time.
>
>
> On Mon, Nov 12, 2012 at 10:37 PM, Marcus Sorensen <sh...@gmail.com>wrote:
>
>> Keep in mind that the version numbers on the Cent 6.3 packages mean
>> nothing. They are far ahead and patched beyond what they say. For example
>> qemu-kvm version is in reality built off of 1.0 I will look and see if the
>> stock ones are built with rbd.
>> On Nov 12, 2012 10:33 PM, "Alex Jiang" <al...@gmail.com> wrote:
>>
>>> Hi, All
>>>
>>> Has somebody used Ceph RBD in CloudStack as primary storage? I see that in
>>> the new features of CS 4.0, RBD is supported for KVM. So I tried using RBD
>>> as primary storage but met with some problems.
>>>
>>> I use a CentOS6.3 server as host. First I erase the qemu-kvm(0.12.1) and
>>> libvirt(0.9.10) because their versions are too low (Qemu on the Hypervisor
>>> has to be compiled with RBD enabled .The libvirt version on the Hypervisor
>>> has to be at least 0.10 with RBD enabled).Then I download the latest
>>> qemu(1.2.0) and libvirt(1.0.0) source code and compile and install them.
>>> But when compiling qemu source code,
>>> #wget http://wiki.qemu-project.org/download/qemu-1.2.0.tar.bz2
>>> #tar jxvf qemu-1.2.0.tar.bz2
>>> # cd qemu-1.2.0
>>> # ./configure --enable-rbd
>>>
>>> the following errors occur:
>>> ERROR: User requested feature rados block device
>>> ERROR: configure was not able to find it
>>>
>>> But on Ubuntu12.04 I tried compiling qemu source code and succeed.Now I am
>>> very confused.How to use Ceph RBD as primary storage in CloudStack on
>>> CentOS6.3?Anyone can help me?
>>>
>>> Best Regards,
>>>
>>> Alex
>>>
>>
Re: [Help] Use Ceph RBD as primary storage in CloudStack
Posted by Marcus Sorensen <sh...@gmail.com>.
Ok, doesn't seem to be built with RBD, but here's a quick rundown on how to
do it. I just ran through this and then tried to make sense of my history,
so hopefully you can adjust it to work for your environment. I built ceph
rpms since I imagine you'll want them if you deploy it elsewhere.
get dependencies
- from yum:
- yum install automake libtool nss nss-devel fuse-devel gcc-c++
libatomic_ops-devel libaio-devel boost* libcurl-devel libxml2-devel
expat-devel rpm-build gcc
- You may need to do more packages depending on your install
- from epel:
- rpm -ivh
http://dl.fedoraproject.org/pub/epel/6/x86_64/fcgi-2.4.0-10.el6.x86_64.rpm
http://dl.fedoraproject.org/pub/epel/6/x86_64/fcgi-devel-2.4.0-10.el6.x86_64.rpm
- from source:
- wget
http://download.savannah.gnu.org/releases/libunwind/libunwind-0.99-beta.tar.gz
- tar xzf libunwind-0.99-beta.tar.gz
- cd libunwind-0.99-beta
- ./configure; make; make install
- wget http://gperftools.googlecode.com/files/gperftools-2.0.tar.gz
- tar xzf gperftools-2.0.tar.gz
- cd gperftools-2.0
- ./configure; make; make install
build ceph:
- git clone --recursive https://github.com/ceph/ceph.git
- cd ceph
- ./autogen.sh
- ./configure
- cd ..
- mv ceph ceph-0.53
- tar cjf ceph-0.53.tar.bz2 /root/rpmbuild/SOURCES
- rpmbuild -ba ceph-0.53/ceph.spec
- rpm -ivh /root/rpmbuild/RPMS/x86_64/ceph-devel*
Now your ./configure of qemu should work. Some notes... you probably don't
want to build RPMs as root. Also, this builds ceph with NSS, which from
what I understand isn't preferable to cryptopp. I installed cryptopp and
cryptopp-devel from EPEL, but it didn't automatically find it so I moved
along in the interest of time.
On Mon, Nov 12, 2012 at 10:37 PM, Marcus Sorensen <sh...@gmail.com>wrote:
> Keep in mind that the version numbers on the Cent 6.3 packages mean
> nothing. They are far ahead and patched beyond what they say. For example
> qemu-kvm version is in reality built off of 1.0 I will look and see if the
> stock ones are built with rbd.
> On Nov 12, 2012 10:33 PM, "Alex Jiang" <al...@gmail.com> wrote:
>
>> Hi, All
>>
>> Has somebody used Ceph RBD in CloudStack as primary storage? I see that in
>> the new features of CS 4.0, RBD is supported for KVM. So I tried using RBD
>> as primary storage but met with some problems.
>>
>> I use a CentOS6.3 server as host. First I erase the qemu-kvm(0.12.1) and
>> libvirt(0.9.10) because their versions are too low (Qemu on the Hypervisor
>> has to be compiled with RBD enabled .The libvirt version on the Hypervisor
>> has to be at least 0.10 with RBD enabled).Then I download the latest
>> qemu(1.2.0) and libvirt(1.0.0) source code and compile and install them.
>> But when compiling qemu source code,
>> #wget http://wiki.qemu-project.org/download/qemu-1.2.0.tar.bz2
>> #tar jxvf qemu-1.2.0.tar.bz2
>> # cd qemu-1.2.0
>> # ./configure --enable-rbd
>>
>> the following errors occur:
>> ERROR: User requested feature rados block device
>> ERROR: configure was not able to find it
>>
>> But on Ubuntu12.04 I tried compiling qemu source code and succeed.Now I am
>> very confused.How to use Ceph RBD as primary storage in CloudStack on
>> CentOS6.3?Anyone can help me?
>>
>> Best Regards,
>>
>> Alex
>>
>
Re: [Help] Use Ceph RBD as primary storage in CloudStack
Posted by Marcus Sorensen <sh...@gmail.com>.
Keep in mind that the version numbers on the Cent 6.3 packages mean
nothing. They are far ahead and patched beyond what they say. For example
qemu-kvm version is in reality built off of 1.0 I will look and see if the
stock ones are built with rbd.
On Nov 12, 2012 10:33 PM, "Alex Jiang" <al...@gmail.com> wrote:
> Hi, All
>
> Has somebody used Ceph RBD in CloudStack as primary storage? I see that in
> the new features of CS 4.0, RBD is supported for KVM. So I tried using RBD
> as primary storage but met with some problems.
>
> I use a CentOS6.3 server as host. First I erase the qemu-kvm(0.12.1) and
> libvirt(0.9.10) because their versions are too low (Qemu on the Hypervisor
> has to be compiled with RBD enabled .The libvirt version on the Hypervisor
> has to be at least 0.10 with RBD enabled).Then I download the latest
> qemu(1.2.0) and libvirt(1.0.0) source code and compile and install them.
> But when compiling qemu source code,
> #wget http://wiki.qemu-project.org/download/qemu-1.2.0.tar.bz2
> #tar jxvf qemu-1.2.0.tar.bz2
> # cd qemu-1.2.0
> # ./configure --enable-rbd
>
> the following errors occur:
> ERROR: User requested feature rados block device
> ERROR: configure was not able to find it
>
> But on Ubuntu12.04 I tried compiling qemu source code and succeed.Now I am
> very confused.How to use Ceph RBD as primary storage in CloudStack on
> CentOS6.3?Anyone can help me?
>
> Best Regards,
>
> Alex
>