You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Wido den Hollander <wi...@widodh.nl> on 2012/06/29 17:59:34 UTC
First review of RBD support for primary storage
Hi,
After a couple of months worth of work I'm happy to announce that the
RBD support for primary storage in CloudStack seems to be reaching a
point where it's good enough to be reviewed.
If you are planning to test RBD, please do read this e-mail carefully
since there are still some catches.
Although the code inside CloudStack doesn't seem like a lot of code, I
had to modify code outside CloudStack to get RBD support working:
1. RBD storage pool support in libvirt. [0] [1]
2. Fix a couple of bugs in the libvirt-java bindings. [2]
With those issues addressed I could implement RBD inside CloudStack.
While doing so I ran into multiple issues inside CloudStack which
delayed everything a bit.
Now, the RBD support for primary storage knows limitations:
- It only works with KVM
- You are NOT able to snapshot RBD volumes. This is due to CloudStack
wanting to backup snapshots to the secondary storage and uses 'qemu-img
convert' for this. That doesn't work with RBD, but it's also very
inefficient.
RBD supports native snapshots inside the Ceph cluster. RBD disks also
have the potential to reach very large sizes. Disks of 1TB won't be the
exception. It would stress your network heavily. I'm thinking about
implementing "internal snapshots", but that is step #2. For now no
snapshots.
- You are able create a template from a RBD volume, but creating a new
instance with RBD storage from a template is still a hit-and-miss.
Working on that one.
Other than these limitations, everything works. You can create instances
and attach RBD disks. It also supports cephx authorization, so no
problem there!
What do you need to run this patch?
- A Ceph cluster
- libvirt with RBD storage pool support (>0.9.12)
- Modified libvirt-java bindings (jar is in the patch)
- Qemu with RBD support (>0.14)
- A extra field "user_info" in the storage pool table, see the SQL
change in the patch
You can fetch the code on my Github account [3].
Warning: I'll be rebasing against the master branch regularly, so be
aware of git pull not always working nicely.
I'd like to see this code reviewed while I'm working on the latest stuff
and getting all the patches upstream in other projects (mainly the
libvirt Java bindings).
Any suggestions or comments?
Thank you!
Wido
[0]:
http://libvirt.org/git/?p=libvirt.git;a=commit;h=74951eadef85e2d100c7dc7bd9ae1093fbda722f
[1]:
http://libvirt.org/git/?p=libvirt.git;a=commit;h=122fa379de44a2fd0a6d5fbcb634535d647ada17
[2]: https://github.com/wido/libvirt-java/commits/cloudstack
[3]: https://github.com/wido/CloudStack/commits/rbd
RE: First review of RBD support for primary storage
Posted by Frank Zhang <Fr...@citrix.com>.
> > -----Original Message-----
> > From: David Nalley [mailto:david@gnsa.us]
> > Sent: Thursday, July 05, 2012 12:15 PM
> > To: cloudstack-dev@incubator.apache.org
> > Subject: Re: First review of RBD support for primary storage
> >
> > > Excellent! Will you update the libvirt-java binary in CloudStack to
> > the latest upstream after it's released?
> > >
> >
> >
> > Please do not do this. libvirt-java is licensed under the LGPL which
> > makes it a prohibited item. We actually need to get rid of
> > libvirt-java
>
> But KVM depends on it. What we are going to do? Re-write a libvirt java
> binding licensed under ASF?
Is this library in our tarball? My understanding is we can use it as long as it is pulled from distribution's repo
> >
> > --David
RE: First review of RBD support for primary storage
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Thursday, July 05, 2012 2:34 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: First review of RBD support for primary storage
>
> On Thu, Jul 5, 2012 at 5:24 PM, Edison Su <Ed...@citrix.com> wrote:
> >
> >
> >> -----Original Message-----
> >> From: David Nalley [mailto:david@gnsa.us]
> >> Sent: Thursday, July 05, 2012 12:15 PM
> >> To: cloudstack-dev@incubator.apache.org
> >> Subject: Re: First review of RBD support for primary storage
> >>
> >> > Excellent! Will you update the libvirt-java binary in CloudStack
> to
> >> the latest upstream after it's released?
> >> >
> >>
> >>
> >> Please do not do this. libvirt-java is licensed under the LGPL which
> >> makes it a prohibited item. We actually need to get rid of
> >> libvirt-java
> >
> > But KVM depends on it. What we are going to do? Re-write a libvirt
> java binding licensed under ASF?
> >>
> >> --David
>
> Probably make it a system requirement, though there are some other
> alternatives. However we can't have libvirt-java in the repo.
System requirement doesn't work, as we will use the latest libvirt-java which haven't released yet, which means most likely all the distributions will not have the latest release. Maybe we can add it as a system requirement, then in the installation guide, tells user to download and install the latest libvirt-java from somewhere?
Re: First review of RBD support for primary storage
Posted by David Nalley <da...@gnsa.us>.
On Thu, Jul 5, 2012 at 5:24 PM, Edison Su <Ed...@citrix.com> wrote:
>
>
>> -----Original Message-----
>> From: David Nalley [mailto:david@gnsa.us]
>> Sent: Thursday, July 05, 2012 12:15 PM
>> To: cloudstack-dev@incubator.apache.org
>> Subject: Re: First review of RBD support for primary storage
>>
>> > Excellent! Will you update the libvirt-java binary in CloudStack to
>> the latest upstream after it's released?
>> >
>>
>>
>> Please do not do this. libvirt-java is licensed under the LGPL which
>> makes it a prohibited item. We actually need to get rid of
>> libvirt-java
>
> But KVM depends on it. What we are going to do? Re-write a libvirt java binding licensed under ASF?
>>
>> --David
Probably make it a system requirement, though there are some other
alternatives. However we can't have libvirt-java in the repo.
RE: First review of RBD support for primary storage
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: David Nalley [mailto:david@gnsa.us]
> Sent: Thursday, July 05, 2012 12:15 PM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: First review of RBD support for primary storage
>
> > Excellent! Will you update the libvirt-java binary in CloudStack to
> the latest upstream after it's released?
> >
>
>
> Please do not do this. libvirt-java is licensed under the LGPL which
> makes it a prohibited item. We actually need to get rid of
> libvirt-java
But KVM depends on it. What we are going to do? Re-write a libvirt java binding licensed under ASF?
>
> --David
Re: First review of RBD support for primary storage
Posted by David Nalley <da...@gnsa.us>.
> Excellent! Will you update the libvirt-java binary in CloudStack to the latest upstream after it's released?
>
Please do not do this. libvirt-java is licensed under the LGPL which
makes it a prohibited item. We actually need to get rid of
libvirt-java
--David
RE: First review of RBD support for primary storage
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Wido den Hollander [mailto:wido@widodh.nl]
> Sent: Thursday, July 05, 2012 6:50 AM
> To: cloudstack-dev@incubator.apache.org
> Subject: Re: First review of RBD support for primary storage
>
> >
> > Other than these limitations, everything works. You can create
> instances
> > and attach RBD disks. It also supports cephx authorization, so no
> > problem there!
>
> I found a bug in libvirt under Ubuntu 12.04. In short, the base64
> encoding/decoding inside libvirt is broken due to a third party library.
>
> For more information:
> https://www.redhat.com/archives/libvir-list/2012-July/msg00058.html
>
> >
> > What do you need to run this patch?
> > - A Ceph cluster
> > - libvirt with RBD storage pool support (>0.9.12)
>
> I recommend running 0.9.13 (just got out) since it contains RBD support.
> But there is a bug if you're not running with cephx, this just got
> fixed:
> http://libvirt.org/git/?p=libvirt.git;a=commit;h=ccb94785007d33365d49dd
> 566e194eb0a022148d
>
> In a couple of weeks libvirt 0.9.14 will be released and that will
> contain everything you need and will probably fix the base64/secret
> problem as well.
>
> > - Modified libvirt-java bindings (jar is in the patch)
>
> Tomorrow there will be a release of libvirt-java 0.4.8 which will
> contain everything you need. No more need for a homebrew version of the
> libvirt Java bindings, we can use the upstream ones!
Excellent! Will you update the libvirt-java binary in CloudStack to the latest upstream after it's released?
>
> http://www.libvirt.org/git/?p=libvirt-java.git;a=summary
>
> > - Qemu with RBD support (>0.14)
> > - A extra field "user_info" in the storage pool table, see the SQL
> > change in the patch
> >
> > You can fetch the code on my Github account [3].
>
> Not true anymore, I'm now pushing to the "rbd" feature branch at the
> Apache CloudStack repository.
>
> >
> > Warning: I'll be rebasing against the master branch regularly, so be
> > aware of git pull not always working nicely.
> >
> > I'd like to see this code reviewed while I'm working on the latest
> stuff
> > and getting all the patches upstream in other projects (mainly the
> > libvirt Java bindings).
>
> Like said, the libvirt Java bindings have gone upstream and that should
> be settled by tomorrow.
>
> Wido
Re: First review of RBD support for primary storage
Posted by David Nalley <da...@gnsa.us>.
Wido: perhaps you could produce some packages for testing? at least .debs?
--David
On Thu, Jul 5, 2012 at 5:34 PM, Senner, Talin <se...@wildcardcorp.com> wrote:
> Awesomeness Wido. +10. I'd be happy to do any testing on my Ubuntu
> 12.04 cluster + ceph .48...
>
> Talin
>
> On Thu, Jul 5, 2012 at 8:50 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>>>
>>> Other than these limitations, everything works. You can create instances
>>> and attach RBD disks. It also supports cephx authorization, so no
>>> problem there!
>>
>>
>> I found a bug in libvirt under Ubuntu 12.04. In short, the base64
>> encoding/decoding inside libvirt is broken due to a third party library.
>>
>> For more information:
>> https://www.redhat.com/archives/libvir-list/2012-July/msg00058.html
>>
>>
>>>
>>> What do you need to run this patch?
>>> - A Ceph cluster
>>> - libvirt with RBD storage pool support (>0.9.12)
>>
>>
>> I recommend running 0.9.13 (just got out) since it contains RBD support. But
>> there is a bug if you're not running with cephx, this just got fixed:
>> http://libvirt.org/git/?p=libvirt.git;a=commit;h=ccb94785007d33365d49dd566e194eb0a022148d
>>
>> In a couple of weeks libvirt 0.9.14 will be released and that will contain
>> everything you need and will probably fix the base64/secret problem as well.
>>
>>
>>> - Modified libvirt-java bindings (jar is in the patch)
>>
>>
>> Tomorrow there will be a release of libvirt-java 0.4.8 which will contain
>> everything you need. No more need for a homebrew version of the libvirt Java
>> bindings, we can use the upstream ones!
>>
>> http://www.libvirt.org/git/?p=libvirt-java.git;a=summary
>>
>>
>>> - Qemu with RBD support (>0.14)
>>> - A extra field "user_info" in the storage pool table, see the SQL
>>> change in the patch
>>>
>>> You can fetch the code on my Github account [3].
>>
>>
>> Not true anymore, I'm now pushing to the "rbd" feature branch at the Apache
>> CloudStack repository.
>>
>>
>>>
>>> Warning: I'll be rebasing against the master branch regularly, so be
>>> aware of git pull not always working nicely.
>>>
>>> I'd like to see this code reviewed while I'm working on the latest stuff
>>> and getting all the patches upstream in other projects (mainly the
>>> libvirt Java bindings).
>>
>>
>> Like said, the libvirt Java bindings have gone upstream and that should be
>> settled by tomorrow.
>>
>> Wido
Re: fail to add rbd primary storage
Posted by Wido den Hollander <wi...@widodh.nl>.
Hi,
As requested, could you try to define the RBD storage pool manually on
the Hypervisor first?
Create a file "secret.xml"
<secret ephemeral='no' private='no'>
<uuid>7a91dc24-b072-43c4-98fb-4b2415322b0f</uuid>
<usage type='ceph'>
<name>admin</name>
</usage>
</secret>
Then run:
$ virsh secret-define secret.xml
$ virsh secret-set-value 7a91dc24-b072-43c4-98fb-4b2415322b0f <key>
Where <key> is your cephx key.
Now, create a file "rbd-pool.xml"
<pool type='rbd'>
<name>mycephpool</name>
<uuid>f959641f-f518-4505-9e85-17d994e2a398</uuid>
<source>
<host name='1.2.3.4' port='6789'/>
<name>rbd</name>
<auth username='admin' type='ceph'>
<secret uuid='7a91dc24-b072-43c4-98fb-4b2415322b0f'/>
</auth>
</source>
</pool>
Obviously, replace 1.2.3.4 by the IP/hostname of your monitor.
Then define the pool:
$ virsh define-pool rbd-pool.xml
Let me know how that works out. It's just to rule out it's not a problem
with your libvirt or Ceph cluster.
Wido
On 09/27/2012 08:52 AM, coudstacks wrote:
> if this step is documented.
> RADOS user = client.admin
> RADOS secret = key correspond to client.admin.
> what else should i do on ceph-nodes?
>
> 2012-09-27 21:03:52,128 WARN [cloud.storage.StorageManagerImpl] (catalina-exec-24:null) Unable to establish a connection between Host[-6-Routing] and Pool[203|RBD]
> com.cloud.exception.StorageUnavailableException: Resource [StoragePool:203] is unreachable: Unable establish connection from storage head to storage pool 203 due to java.lang.NullPointerException
> at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:563)
> at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:57)
> at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2066)
> at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1027)
> at com.cloud.agent.Agent.processRequest(Agent.java:518)
> at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)
> at com.cloud.utils.nio.Task.run(Task.java:83)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1685)
> at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:1450)
> at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:215)
> at com.cloud.api.commands.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:120)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
> at com.cloud.api.ApiServer.queueCommand(ApiServer.java:543)
> at com.cloud.api.ApiServer.handleRequest(ApiServer.java:422)
> at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
> at com.cloud.api.ApiServlet.doGet(ApiServlet.java:63)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
> at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
> at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
> at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
> at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
> at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
> at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
> at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
> at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
> at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
> at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
> at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2012-09-27 21:03:52,137 WARN [cloud.storage.StorageManagerImpl] (catalina-exec-24:null) No host can access storage pool Pool[203|RBD] on cluster 1
> 2012-09-27 21:03:52,140 WARN [cloud.api.ApiDispatcher] (catalina-exec-24:null) class com.cloud.api.ServerApiException : Failed to add storage pool
>
>
>
>
>
> At 2012-07-06 23:11:47,"Wido den Hollander" <wi...@widodh.nl> wrote:
>>
>>
>> On 07/05/2012 11:34 PM, Senner, Talin wrote:
>>> Awesomeness Wido. +10. I'd be happy to do any testing on my Ubuntu
>>> 12.04 cluster + ceph .48...
>>
>> Testing is needed.
>>
>> Be aware, on Ubuntu 12.04 there is the bug with libvirt and the secrets.
>>
>> If you run into problems, check if libvirtd is linked against libroken:
>>
>> $ ldd /usr/sbin/libvirtd
>>
>> If you are running without Cephx you will need the patched version of
>> Qemu where it explicitly adds "auth_supported=none" to the Qemu args,
>> this is commit: ccb94785007d33365d49dd566e194eb0a022148d
>>
>> Wido
>>
>>>
>>> Talin
>>>
>>> On Thu, Jul 5, 2012 at 8:50 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>>
>>>>> Other than these limitations, everything works. You can create instances
>>>>> and attach RBD disks. It also supports cephx authorization, so no
>>>>> problem there!
>>>>
>>>>
>>>> I found a bug in libvirt under Ubuntu 12.04. In short, the base64
>>>> encoding/decoding inside libvirt is broken due to a third party library.
>>>>
>>>> For more information:
>>>> https://www.redhat.com/archives/libvir-list/2012-July/msg00058.html
>>>>
>>>>
>>>>>
>>>>> What do you need to run this patch?
>>>>> - A Ceph cluster
>>>>> - libvirt with RBD storage pool support (>0.9.12)
>>>>
>>>>
>>>> I recommend running 0.9.13 (just got out) since it contains RBD support. But
>>>> there is a bug if you're not running with cephx, this just got fixed:
>>>> http://libvirt.org/git/?p=libvirt.git;a=commit;h=ccb94785007d33365d49dd566e194eb0a022148d
>>>>
>>>> In a couple of weeks libvirt 0.9.14 will be released and that will contain
>>>> everything you need and will probably fix the base64/secret problem as well.
>>>>
>>>>
>>>>> - Modified libvirt-java bindings (jar is in the patch)
>>>>
>>>>
>>>> Tomorrow there will be a release of libvirt-java 0.4.8 which will contain
>>>> everything you need. No more need for a homebrew version of the libvirt Java
>>>> bindings, we can use the upstream ones!
>>>>
>>>> http://www.libvirt.org/git/?p=libvirt-java.git;a=summary
>>>>
>>>>
>>>>> - Qemu with RBD support (>0.14)
>>>>> - A extra field "user_info" in the storage pool table, see the SQL
>>>>> change in the patch
>>>>>
>>>>> You can fetch the code on my Github account [3].
>>>>
>>>>
>>>> Not true anymore, I'm now pushing to the "rbd" feature branch at the Apache
>>>> CloudStack repository.
>>>>
>>>>
>>>>>
>>>>> Warning: I'll be rebasing against the master branch regularly, so be
>>>>> aware of git pull not always working nicely.
>>>>>
>>>>> I'd like to see this code reviewed while I'm working on the latest stuff
>>>>> and getting all the patches upstream in other projects (mainly the
>>>>> libvirt Java bindings).
>>>>
>>>>
>>>> Like said, the libvirt Java bindings have gone upstream and that should be
>>>> settled by tomorrow.
>>>>
>>>> Wido
>>
>
fail to add rbd primary storage
Posted by coudstacks <cl...@163.com>.
if this step is documented.
RADOS user = client.admin
RADOS secret = key correspond to client.admin.
what else should i do on ceph-nodes?
2012-09-27 21:03:52,128 WARN [cloud.storage.StorageManagerImpl] (catalina-exec-24:null) Unable to establish a connection between Host[-6-Routing] and Pool[203|RBD]
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:203] is unreachable: Unable establish connection from storage head to storage pool 203 due to java.lang.NullPointerException
at com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.createStoragePool(LibvirtStorageAdaptor.java:563)
at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.createStoragePool(KVMStoragePoolManager.java:57)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2066)
at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1027)
at com.cloud.agent.Agent.processRequest(Agent.java:518)
at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)
at com.cloud.utils.nio.Task.run(Task.java:83)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1685)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:1450)
at com.cloud.storage.StorageManagerImpl.createPool(StorageManagerImpl.java:215)
at com.cloud.api.commands.CreateStoragePoolCmd.execute(CreateStoragePoolCmd.java:120)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:138)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:543)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:422)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:304)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:63)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:689)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:802)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2012-09-27 21:03:52,137 WARN [cloud.storage.StorageManagerImpl] (catalina-exec-24:null) No host can access storage pool Pool[203|RBD] on cluster 1
2012-09-27 21:03:52,140 WARN [cloud.api.ApiDispatcher] (catalina-exec-24:null) class com.cloud.api.ServerApiException : Failed to add storage pool
At 2012-07-06 23:11:47,"Wido den Hollander" <wi...@widodh.nl> wrote:
>
>
>On 07/05/2012 11:34 PM, Senner, Talin wrote:
>> Awesomeness Wido. +10. I'd be happy to do any testing on my Ubuntu
>> 12.04 cluster + ceph .48...
>
>Testing is needed.
>
>Be aware, on Ubuntu 12.04 there is the bug with libvirt and the secrets.
>
>If you run into problems, check if libvirtd is linked against libroken:
>
>$ ldd /usr/sbin/libvirtd
>
>If you are running without Cephx you will need the patched version of
>Qemu where it explicitly adds "auth_supported=none" to the Qemu args,
>this is commit: ccb94785007d33365d49dd566e194eb0a022148d
>
>Wido
>
>>
>> Talin
>>
>> On Thu, Jul 5, 2012 at 8:50 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>>>>
>>>> Other than these limitations, everything works. You can create instances
>>>> and attach RBD disks. It also supports cephx authorization, so no
>>>> problem there!
>>>
>>>
>>> I found a bug in libvirt under Ubuntu 12.04. In short, the base64
>>> encoding/decoding inside libvirt is broken due to a third party library.
>>>
>>> For more information:
>>> https://www.redhat.com/archives/libvir-list/2012-July/msg00058.html
>>>
>>>
>>>>
>>>> What do you need to run this patch?
>>>> - A Ceph cluster
>>>> - libvirt with RBD storage pool support (>0.9.12)
>>>
>>>
>>> I recommend running 0.9.13 (just got out) since it contains RBD support. But
>>> there is a bug if you're not running with cephx, this just got fixed:
>>> http://libvirt.org/git/?p=libvirt.git;a=commit;h=ccb94785007d33365d49dd566e194eb0a022148d
>>>
>>> In a couple of weeks libvirt 0.9.14 will be released and that will contain
>>> everything you need and will probably fix the base64/secret problem as well.
>>>
>>>
>>>> - Modified libvirt-java bindings (jar is in the patch)
>>>
>>>
>>> Tomorrow there will be a release of libvirt-java 0.4.8 which will contain
>>> everything you need. No more need for a homebrew version of the libvirt Java
>>> bindings, we can use the upstream ones!
>>>
>>> http://www.libvirt.org/git/?p=libvirt-java.git;a=summary
>>>
>>>
>>>> - Qemu with RBD support (>0.14)
>>>> - A extra field "user_info" in the storage pool table, see the SQL
>>>> change in the patch
>>>>
>>>> You can fetch the code on my Github account [3].
>>>
>>>
>>> Not true anymore, I'm now pushing to the "rbd" feature branch at the Apache
>>> CloudStack repository.
>>>
>>>
>>>>
>>>> Warning: I'll be rebasing against the master branch regularly, so be
>>>> aware of git pull not always working nicely.
>>>>
>>>> I'd like to see this code reviewed while I'm working on the latest stuff
>>>> and getting all the patches upstream in other projects (mainly the
>>>> libvirt Java bindings).
>>>
>>>
>>> Like said, the libvirt Java bindings have gone upstream and that should be
>>> settled by tomorrow.
>>>
>>> Wido
>
Re: First review of RBD support for primary storage
Posted by Wido den Hollander <wi...@widodh.nl>.
On 07/05/2012 11:34 PM, Senner, Talin wrote:
> Awesomeness Wido. +10. I'd be happy to do any testing on my Ubuntu
> 12.04 cluster + ceph .48...
Testing is needed.
Be aware, on Ubuntu 12.04 there is the bug with libvirt and the secrets.
If you run into problems, check if libvirtd is linked against libroken:
$ ldd /usr/sbin/libvirtd
If you are running without Cephx you will need the patched version of
Qemu where it explicitly adds "auth_supported=none" to the Qemu args,
this is commit: ccb94785007d33365d49dd566e194eb0a022148d
Wido
>
> Talin
>
> On Thu, Jul 5, 2012 at 8:50 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>>>
>>> Other than these limitations, everything works. You can create instances
>>> and attach RBD disks. It also supports cephx authorization, so no
>>> problem there!
>>
>>
>> I found a bug in libvirt under Ubuntu 12.04. In short, the base64
>> encoding/decoding inside libvirt is broken due to a third party library.
>>
>> For more information:
>> https://www.redhat.com/archives/libvir-list/2012-July/msg00058.html
>>
>>
>>>
>>> What do you need to run this patch?
>>> - A Ceph cluster
>>> - libvirt with RBD storage pool support (>0.9.12)
>>
>>
>> I recommend running 0.9.13 (just got out) since it contains RBD support. But
>> there is a bug if you're not running with cephx, this just got fixed:
>> http://libvirt.org/git/?p=libvirt.git;a=commit;h=ccb94785007d33365d49dd566e194eb0a022148d
>>
>> In a couple of weeks libvirt 0.9.14 will be released and that will contain
>> everything you need and will probably fix the base64/secret problem as well.
>>
>>
>>> - Modified libvirt-java bindings (jar is in the patch)
>>
>>
>> Tomorrow there will be a release of libvirt-java 0.4.8 which will contain
>> everything you need. No more need for a homebrew version of the libvirt Java
>> bindings, we can use the upstream ones!
>>
>> http://www.libvirt.org/git/?p=libvirt-java.git;a=summary
>>
>>
>>> - Qemu with RBD support (>0.14)
>>> - A extra field "user_info" in the storage pool table, see the SQL
>>> change in the patch
>>>
>>> You can fetch the code on my Github account [3].
>>
>>
>> Not true anymore, I'm now pushing to the "rbd" feature branch at the Apache
>> CloudStack repository.
>>
>>
>>>
>>> Warning: I'll be rebasing against the master branch regularly, so be
>>> aware of git pull not always working nicely.
>>>
>>> I'd like to see this code reviewed while I'm working on the latest stuff
>>> and getting all the patches upstream in other projects (mainly the
>>> libvirt Java bindings).
>>
>>
>> Like said, the libvirt Java bindings have gone upstream and that should be
>> settled by tomorrow.
>>
>> Wido
Re: First review of RBD support for primary storage
Posted by "Senner, Talin" <se...@wildcardcorp.com>.
Awesomeness Wido. +10. I'd be happy to do any testing on my Ubuntu
12.04 cluster + ceph .48...
Talin
On Thu, Jul 5, 2012 at 8:50 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>>
>> Other than these limitations, everything works. You can create instances
>> and attach RBD disks. It also supports cephx authorization, so no
>> problem there!
>
>
> I found a bug in libvirt under Ubuntu 12.04. In short, the base64
> encoding/decoding inside libvirt is broken due to a third party library.
>
> For more information:
> https://www.redhat.com/archives/libvir-list/2012-July/msg00058.html
>
>
>>
>> What do you need to run this patch?
>> - A Ceph cluster
>> - libvirt with RBD storage pool support (>0.9.12)
>
>
> I recommend running 0.9.13 (just got out) since it contains RBD support. But
> there is a bug if you're not running with cephx, this just got fixed:
> http://libvirt.org/git/?p=libvirt.git;a=commit;h=ccb94785007d33365d49dd566e194eb0a022148d
>
> In a couple of weeks libvirt 0.9.14 will be released and that will contain
> everything you need and will probably fix the base64/secret problem as well.
>
>
>> - Modified libvirt-java bindings (jar is in the patch)
>
>
> Tomorrow there will be a release of libvirt-java 0.4.8 which will contain
> everything you need. No more need for a homebrew version of the libvirt Java
> bindings, we can use the upstream ones!
>
> http://www.libvirt.org/git/?p=libvirt-java.git;a=summary
>
>
>> - Qemu with RBD support (>0.14)
>> - A extra field "user_info" in the storage pool table, see the SQL
>> change in the patch
>>
>> You can fetch the code on my Github account [3].
>
>
> Not true anymore, I'm now pushing to the "rbd" feature branch at the Apache
> CloudStack repository.
>
>
>>
>> Warning: I'll be rebasing against the master branch regularly, so be
>> aware of git pull not always working nicely.
>>
>> I'd like to see this code reviewed while I'm working on the latest stuff
>> and getting all the patches upstream in other projects (mainly the
>> libvirt Java bindings).
>
>
> Like said, the libvirt Java bindings have gone upstream and that should be
> settled by tomorrow.
>
> Wido
Re: First review of RBD support for primary storage
Posted by Wido den Hollander <wi...@widodh.nl>.
>
> Other than these limitations, everything works. You can create instances
> and attach RBD disks. It also supports cephx authorization, so no
> problem there!
I found a bug in libvirt under Ubuntu 12.04. In short, the base64
encoding/decoding inside libvirt is broken due to a third party library.
For more information:
https://www.redhat.com/archives/libvir-list/2012-July/msg00058.html
>
> What do you need to run this patch?
> - A Ceph cluster
> - libvirt with RBD storage pool support (>0.9.12)
I recommend running 0.9.13 (just got out) since it contains RBD support.
But there is a bug if you're not running with cephx, this just got
fixed:
http://libvirt.org/git/?p=libvirt.git;a=commit;h=ccb94785007d33365d49dd566e194eb0a022148d
In a couple of weeks libvirt 0.9.14 will be released and that will
contain everything you need and will probably fix the base64/secret
problem as well.
> - Modified libvirt-java bindings (jar is in the patch)
Tomorrow there will be a release of libvirt-java 0.4.8 which will
contain everything you need. No more need for a homebrew version of the
libvirt Java bindings, we can use the upstream ones!
http://www.libvirt.org/git/?p=libvirt-java.git;a=summary
> - Qemu with RBD support (>0.14)
> - A extra field "user_info" in the storage pool table, see the SQL
> change in the patch
>
> You can fetch the code on my Github account [3].
Not true anymore, I'm now pushing to the "rbd" feature branch at the
Apache CloudStack repository.
>
> Warning: I'll be rebasing against the master branch regularly, so be
> aware of git pull not always working nicely.
>
> I'd like to see this code reviewed while I'm working on the latest stuff
> and getting all the patches upstream in other projects (mainly the
> libvirt Java bindings).
Like said, the libvirt Java bindings have gone upstream and that should
be settled by tomorrow.
Wido
Re: First review of RBD support for primary storage
Posted by Wido den Hollander <wi...@widodh.nl>.
On 06/30/2012 01:12 PM, Wido den Hollander wrote:
>
>
> On 06/29/2012 07:39 PM, David Nalley wrote:
>> On Fri, Jun 29, 2012 at 11:59 AM, Wido den Hollander <wi...@widodh.nl>
>> wrote:
>>> Hi,
>>>
>>> After a couple of months worth of work I'm happy to announce that the
>>> RBD
>>> support for primary storage in CloudStack seems to be reaching a
>>> point where
>>> it's good enough to be reviewed.
>>>
>>> If you are planning to test RBD, please do read this e-mail carefully
>>> since
>>> there are still some catches.
>>>
>>> Although the code inside CloudStack doesn't seem like a lot of code,
>>> I had
>>> to modify code outside CloudStack to get RBD support working:
>>>
>>> 1. RBD storage pool support in libvirt. [0] [1]
>>> 2. Fix a couple of bugs in the libvirt-java bindings. [2]
>>>
>>> With those issues addressed I could implement RBD inside CloudStack.
>>>
>>> While doing so I ran into multiple issues inside CloudStack which
>>> delayed
>>> everything a bit.
>>>
>>> Now, the RBD support for primary storage knows limitations:
>>>
>>> - It only works with KVM
>>>
>>> - You are NOT able to snapshot RBD volumes. This is due to CloudStack
>>> wanting to backup snapshots to the secondary storage and uses 'qemu-img
>>> convert' for this. That doesn't work with RBD, but it's also very
>>> inefficient.
>>>
>>> RBD supports native snapshots inside the Ceph cluster. RBD disks also
>>> have
>>> the potential to reach very large sizes. Disks of 1TB won't be the
>>> exception. It would stress your network heavily. I'm thinking about
>>> implementing "internal snapshots", but that is step #2. For now no
>>> snapshots.
>>>
>>> - You are able create a template from a RBD volume, but creating a new
>>> instance with RBD storage from a template is still a hit-and-miss.
>>> Working
>>> on that one.
>>>
>>> Other than these limitations, everything works. You can create
>>> instances and
>>> attach RBD disks. It also supports cephx authorization, so no problem
>>> there!
>>>
>>> What do you need to run this patch?
>>> - A Ceph cluster
>>> - libvirt with RBD storage pool support (>0.9.12)
>>> - Modified libvirt-java bindings (jar is in the patch)
>>> - Qemu with RBD support (>0.14)
>>> - A extra field "user_info" in the storage pool table, see the SQL
>>> change in
>>> the patch
>>>
>>> You can fetch the code on my Github account [3].
>>>
>>> Warning: I'll be rebasing against the master branch regularly, so be
>>> aware
>>> of git pull not always working nicely.
>>>
>>> I'd like to see this code reviewed while I'm working on the latest
>>> stuff and
>>> getting all the patches upstream in other projects (mainly the
>>> libvirt Java
>>> bindings).
>>>
>>> Any suggestions or comments?
>>>
>>> Thank you!
>>>
>>> Wido
>>>
>>>
>>> [0]:
>>> http://libvirt.org/git/?p=libvirt.git;a=commit;h=74951eadef85e2d100c7dc7bd9ae1093fbda722f
>>>
>>> [1]:
>>> http://libvirt.org/git/?p=libvirt.git;a=commit;h=122fa379de44a2fd0a6d5fbcb634535d647ada17
>>>
>>> [2]: https://github.com/wido/libvirt-java/commits/cloudstack
>>> [3]: https://github.com/wido/CloudStack/commits/rbd
>>
>>
>>
>> Wido,
>>
>> I am thrilled to see Ceph support at this stage. Hopefully I'll get to
>> try this out next week.
>> Any chance you'd consider putting this in a topic branch in the ASF repo?
>
> Oh, yes, sure! It's just that I started the development while CS was
> still at Github, so I stayed there.
>
I just pushed the branch "rbd" to the ASF repo, I'll continue my work
there (and keep github in sync as well).
Wido
> I don't like rebasing in topic branches however. When we merge in RBD I
> want to rebase the topic branch and merge it in as one big patch, so
> rebasing is inevitable.
>
> Wido
>
>>
>> --David
>>
>
Re: First review of RBD support for primary storage
Posted by Wido den Hollander <wi...@widodh.nl>.
On 06/29/2012 07:39 PM, David Nalley wrote:
> On Fri, Jun 29, 2012 at 11:59 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>> Hi,
>>
>> After a couple of months worth of work I'm happy to announce that the RBD
>> support for primary storage in CloudStack seems to be reaching a point where
>> it's good enough to be reviewed.
>>
>> If you are planning to test RBD, please do read this e-mail carefully since
>> there are still some catches.
>>
>> Although the code inside CloudStack doesn't seem like a lot of code, I had
>> to modify code outside CloudStack to get RBD support working:
>>
>> 1. RBD storage pool support in libvirt. [0] [1]
>> 2. Fix a couple of bugs in the libvirt-java bindings. [2]
>>
>> With those issues addressed I could implement RBD inside CloudStack.
>>
>> While doing so I ran into multiple issues inside CloudStack which delayed
>> everything a bit.
>>
>> Now, the RBD support for primary storage knows limitations:
>>
>> - It only works with KVM
>>
>> - You are NOT able to snapshot RBD volumes. This is due to CloudStack
>> wanting to backup snapshots to the secondary storage and uses 'qemu-img
>> convert' for this. That doesn't work with RBD, but it's also very
>> inefficient.
>>
>> RBD supports native snapshots inside the Ceph cluster. RBD disks also have
>> the potential to reach very large sizes. Disks of 1TB won't be the
>> exception. It would stress your network heavily. I'm thinking about
>> implementing "internal snapshots", but that is step #2. For now no
>> snapshots.
>>
>> - You are able create a template from a RBD volume, but creating a new
>> instance with RBD storage from a template is still a hit-and-miss. Working
>> on that one.
>>
>> Other than these limitations, everything works. You can create instances and
>> attach RBD disks. It also supports cephx authorization, so no problem there!
>>
>> What do you need to run this patch?
>> - A Ceph cluster
>> - libvirt with RBD storage pool support (>0.9.12)
>> - Modified libvirt-java bindings (jar is in the patch)
>> - Qemu with RBD support (>0.14)
>> - A extra field "user_info" in the storage pool table, see the SQL change in
>> the patch
>>
>> You can fetch the code on my Github account [3].
>>
>> Warning: I'll be rebasing against the master branch regularly, so be aware
>> of git pull not always working nicely.
>>
>> I'd like to see this code reviewed while I'm working on the latest stuff and
>> getting all the patches upstream in other projects (mainly the libvirt Java
>> bindings).
>>
>> Any suggestions or comments?
>>
>> Thank you!
>>
>> Wido
>>
>>
>> [0]:
>> http://libvirt.org/git/?p=libvirt.git;a=commit;h=74951eadef85e2d100c7dc7bd9ae1093fbda722f
>> [1]:
>> http://libvirt.org/git/?p=libvirt.git;a=commit;h=122fa379de44a2fd0a6d5fbcb634535d647ada17
>> [2]: https://github.com/wido/libvirt-java/commits/cloudstack
>> [3]: https://github.com/wido/CloudStack/commits/rbd
>
>
>
> Wido,
>
> I am thrilled to see Ceph support at this stage. Hopefully I'll get to
> try this out next week.
> Any chance you'd consider putting this in a topic branch in the ASF repo?
Oh, yes, sure! It's just that I started the development while CS was
still at Github, so I stayed there.
I don't like rebasing in topic branches however. When we merge in RBD I
want to rebase the topic branch and merge it in as one big patch, so
rebasing is inevitable.
Wido
>
> --David
>
Re: First review of RBD support for primary storage
Posted by Wido den Hollander <wi...@widodh.nl>.
On 06/29/2012 07:41 PM, David Chamard wrote:
> Wido,
>
> Very nice to see. Is there any plan to support xen/xcp hypervisors in the
> future?
No, not yet. RBD is supported by Qemu, so in theory it could work with
Xen/XCP when using Qemu for full virtualization.
Also, the Qemu version needs RBD support and that is not something I can
arrange, Citrix needs to ship a Qemu version with RBD enabled.
Wido
>
> On Fri, Jun 29, 2012 at 1:39 PM, David Nalley <da...@gnsa.us> wrote:
>
>> On Fri, Jun 29, 2012 at 11:59 AM, Wido den Hollander <wi...@widodh.nl>
>> wrote:
>>> Hi,
>>>
>>> After a couple of months worth of work I'm happy to announce that the RBD
>>> support for primary storage in CloudStack seems to be reaching a point
>> where
>>> it's good enough to be reviewed.
>>>
>>> If you are planning to test RBD, please do read this e-mail carefully
>> since
>>> there are still some catches.
>>>
>>> Although the code inside CloudStack doesn't seem like a lot of code, I
>> had
>>> to modify code outside CloudStack to get RBD support working:
>>>
>>> 1. RBD storage pool support in libvirt. [0] [1]
>>> 2. Fix a couple of bugs in the libvirt-java bindings. [2]
>>>
>>> With those issues addressed I could implement RBD inside CloudStack.
>>>
>>> While doing so I ran into multiple issues inside CloudStack which delayed
>>> everything a bit.
>>>
>>> Now, the RBD support for primary storage knows limitations:
>>>
>>> - It only works with KVM
>>>
>>> - You are NOT able to snapshot RBD volumes. This is due to CloudStack
>>> wanting to backup snapshots to the secondary storage and uses 'qemu-img
>>> convert' for this. That doesn't work with RBD, but it's also very
>>> inefficient.
>>>
>>> RBD supports native snapshots inside the Ceph cluster. RBD disks also
>> have
>>> the potential to reach very large sizes. Disks of 1TB won't be the
>>> exception. It would stress your network heavily. I'm thinking about
>>> implementing "internal snapshots", but that is step #2. For now no
>>> snapshots.
>>>
>>> - You are able create a template from a RBD volume, but creating a new
>>> instance with RBD storage from a template is still a hit-and-miss.
>> Working
>>> on that one.
>>>
>>> Other than these limitations, everything works. You can create instances
>> and
>>> attach RBD disks. It also supports cephx authorization, so no problem
>> there!
>>>
>>> What do you need to run this patch?
>>> - A Ceph cluster
>>> - libvirt with RBD storage pool support (>0.9.12)
>>> - Modified libvirt-java bindings (jar is in the patch)
>>> - Qemu with RBD support (>0.14)
>>> - A extra field "user_info" in the storage pool table, see the SQL
>> change in
>>> the patch
>>>
>>> You can fetch the code on my Github account [3].
>>>
>>> Warning: I'll be rebasing against the master branch regularly, so be
>> aware
>>> of git pull not always working nicely.
>>>
>>> I'd like to see this code reviewed while I'm working on the latest stuff
>> and
>>> getting all the patches upstream in other projects (mainly the libvirt
>> Java
>>> bindings).
>>>
>>> Any suggestions or comments?
>>>
>>> Thank you!
>>>
>>> Wido
>>>
>>>
>>> [0]:
>>>
>> http://libvirt.org/git/?p=libvirt.git;a=commit;h=74951eadef85e2d100c7dc7bd9ae1093fbda722f
>>> [1]:
>>>
>> http://libvirt.org/git/?p=libvirt.git;a=commit;h=122fa379de44a2fd0a6d5fbcb634535d647ada17
>>> [2]: https://github.com/wido/libvirt-java/commits/cloudstack
>>> [3]: https://github.com/wido/CloudStack/commits/rbd
>>
>>
>>
>> Wido,
>>
>> I am thrilled to see Ceph support at this stage. Hopefully I'll get to
>> try this out next week.
>> Any chance you'd consider putting this in a topic branch in the ASF repo?
>>
>> --David
>>
>
Re: First review of RBD support for primary storage
Posted by David Chamard <da...@cloud.ca>.
Wido,
Very nice to see. Is there any plan to support xen/xcp hypervisors in the
future?
On Fri, Jun 29, 2012 at 1:39 PM, David Nalley <da...@gnsa.us> wrote:
> On Fri, Jun 29, 2012 at 11:59 AM, Wido den Hollander <wi...@widodh.nl>
> wrote:
> > Hi,
> >
> > After a couple of months worth of work I'm happy to announce that the RBD
> > support for primary storage in CloudStack seems to be reaching a point
> where
> > it's good enough to be reviewed.
> >
> > If you are planning to test RBD, please do read this e-mail carefully
> since
> > there are still some catches.
> >
> > Although the code inside CloudStack doesn't seem like a lot of code, I
> had
> > to modify code outside CloudStack to get RBD support working:
> >
> > 1. RBD storage pool support in libvirt. [0] [1]
> > 2. Fix a couple of bugs in the libvirt-java bindings. [2]
> >
> > With those issues addressed I could implement RBD inside CloudStack.
> >
> > While doing so I ran into multiple issues inside CloudStack which delayed
> > everything a bit.
> >
> > Now, the RBD support for primary storage knows limitations:
> >
> > - It only works with KVM
> >
> > - You are NOT able to snapshot RBD volumes. This is due to CloudStack
> > wanting to backup snapshots to the secondary storage and uses 'qemu-img
> > convert' for this. That doesn't work with RBD, but it's also very
> > inefficient.
> >
> > RBD supports native snapshots inside the Ceph cluster. RBD disks also
> have
> > the potential to reach very large sizes. Disks of 1TB won't be the
> > exception. It would stress your network heavily. I'm thinking about
> > implementing "internal snapshots", but that is step #2. For now no
> > snapshots.
> >
> > - You are able create a template from a RBD volume, but creating a new
> > instance with RBD storage from a template is still a hit-and-miss.
> Working
> > on that one.
> >
> > Other than these limitations, everything works. You can create instances
> and
> > attach RBD disks. It also supports cephx authorization, so no problem
> there!
> >
> > What do you need to run this patch?
> > - A Ceph cluster
> > - libvirt with RBD storage pool support (>0.9.12)
> > - Modified libvirt-java bindings (jar is in the patch)
> > - Qemu with RBD support (>0.14)
> > - A extra field "user_info" in the storage pool table, see the SQL
> change in
> > the patch
> >
> > You can fetch the code on my Github account [3].
> >
> > Warning: I'll be rebasing against the master branch regularly, so be
> aware
> > of git pull not always working nicely.
> >
> > I'd like to see this code reviewed while I'm working on the latest stuff
> and
> > getting all the patches upstream in other projects (mainly the libvirt
> Java
> > bindings).
> >
> > Any suggestions or comments?
> >
> > Thank you!
> >
> > Wido
> >
> >
> > [0]:
> >
> http://libvirt.org/git/?p=libvirt.git;a=commit;h=74951eadef85e2d100c7dc7bd9ae1093fbda722f
> > [1]:
> >
> http://libvirt.org/git/?p=libvirt.git;a=commit;h=122fa379de44a2fd0a6d5fbcb634535d647ada17
> > [2]: https://github.com/wido/libvirt-java/commits/cloudstack
> > [3]: https://github.com/wido/CloudStack/commits/rbd
>
>
>
> Wido,
>
> I am thrilled to see Ceph support at this stage. Hopefully I'll get to
> try this out next week.
> Any chance you'd consider putting this in a topic branch in the ASF repo?
>
> --David
>
Re: First review of RBD support for primary storage
Posted by David Nalley <da...@gnsa.us>.
On Fri, Jun 29, 2012 at 11:59 AM, Wido den Hollander <wi...@widodh.nl> wrote:
> Hi,
>
> After a couple of months worth of work I'm happy to announce that the RBD
> support for primary storage in CloudStack seems to be reaching a point where
> it's good enough to be reviewed.
>
> If you are planning to test RBD, please do read this e-mail carefully since
> there are still some catches.
>
> Although the code inside CloudStack doesn't seem like a lot of code, I had
> to modify code outside CloudStack to get RBD support working:
>
> 1. RBD storage pool support in libvirt. [0] [1]
> 2. Fix a couple of bugs in the libvirt-java bindings. [2]
>
> With those issues addressed I could implement RBD inside CloudStack.
>
> While doing so I ran into multiple issues inside CloudStack which delayed
> everything a bit.
>
> Now, the RBD support for primary storage knows limitations:
>
> - It only works with KVM
>
> - You are NOT able to snapshot RBD volumes. This is due to CloudStack
> wanting to backup snapshots to the secondary storage and uses 'qemu-img
> convert' for this. That doesn't work with RBD, but it's also very
> inefficient.
>
> RBD supports native snapshots inside the Ceph cluster. RBD disks also have
> the potential to reach very large sizes. Disks of 1TB won't be the
> exception. It would stress your network heavily. I'm thinking about
> implementing "internal snapshots", but that is step #2. For now no
> snapshots.
>
> - You are able create a template from a RBD volume, but creating a new
> instance with RBD storage from a template is still a hit-and-miss. Working
> on that one.
>
> Other than these limitations, everything works. You can create instances and
> attach RBD disks. It also supports cephx authorization, so no problem there!
>
> What do you need to run this patch?
> - A Ceph cluster
> - libvirt with RBD storage pool support (>0.9.12)
> - Modified libvirt-java bindings (jar is in the patch)
> - Qemu with RBD support (>0.14)
> - A extra field "user_info" in the storage pool table, see the SQL change in
> the patch
>
> You can fetch the code on my Github account [3].
>
> Warning: I'll be rebasing against the master branch regularly, so be aware
> of git pull not always working nicely.
>
> I'd like to see this code reviewed while I'm working on the latest stuff and
> getting all the patches upstream in other projects (mainly the libvirt Java
> bindings).
>
> Any suggestions or comments?
>
> Thank you!
>
> Wido
>
>
> [0]:
> http://libvirt.org/git/?p=libvirt.git;a=commit;h=74951eadef85e2d100c7dc7bd9ae1093fbda722f
> [1]:
> http://libvirt.org/git/?p=libvirt.git;a=commit;h=122fa379de44a2fd0a6d5fbcb634535d647ada17
> [2]: https://github.com/wido/libvirt-java/commits/cloudstack
> [3]: https://github.com/wido/CloudStack/commits/rbd
Wido,
I am thrilled to see Ceph support at this stage. Hopefully I'll get to
try this out next week.
Any chance you'd consider putting this in a topic branch in the ASF repo?
--David
RE: First review of RBD support for primary storage
Posted by Edison Su <Ed...@citrix.com>.
> -----Original Message-----
> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
> Sent: Thursday, July 05, 2012 3:54 PM
> To: CloudStack DeveloperList
> Subject: Re: First review of RBD support for primary storage
>
> I took a first glance at this. Really pleased about this feature. EBS-
> like
> scalable primary storage is within reach!
>
> A few comments:
> 1. I see quite a few blocks of code ( > 20 times?) that are like
> if (pool.getType() == StoragePoolType.RBD)
> I realize that there is existing code that does these kinds of
> checks
> as well. To me this can be solved simply by the "chain of
> responsibility"
> pattern: you hand over the operation to a configured chain of handlers.
> The first handler (usually) that says it can handle it, terminates the
> chain.
It's in my to-do-list, refactor storage part code, to make adding a new storage type into cloudstack much easier.
> 2. 'user_info' can actually be pushed into the 'storage_pool_details'
> table. Generally we avoid modifying existing tables if we can.
> 3. Copying a snapshot to secondary storage is desirable: to be
> consistent
> with other storage types, to be able to instantiate new volumes in
> other
> zones (when S3 support is available across the region). I'd like to
> understand the blockers here.
>
>
> On 7/2/12 5:59 AM, "Wido den Hollander" <wi...@widodh.nl> wrote:
>
> >Hi,
> >
> >On 29-06-12 17:59, Wido den Hollander wrote:
> >> Now, the RBD support for primary storage knows limitations:
> >>
> >> - It only works with KVM
> >>
> >> - You are NOT able to snapshot RBD volumes. This is due to
> CloudStack
> >> wanting to backup snapshots to the secondary storage and uses 'qemu-
> img
> >> convert' for this. That doesn't work with RBD, but it's also very
> >> inefficient.
> >>
> >> RBD supports native snapshots inside the Ceph cluster. RBD disks
> also
> >> have the potential to reach very large sizes. Disks of 1TB won't be
> the
> >> exception. It would stress your network heavily. I'm thinking about
> >> implementing "internal snapshots", but that is step #2. For now no
> >> snapshots.
> >>
> >> - You are able create a template from a RBD volume, but creating a
> new
> >> instance with RBD storage from a template is still a hit-and-miss.
> >> Working on that one.
> >>
> >
> >I just pushed a fix for creating instances from a template. That
> should
> >work now!
> >
> >Wido
Re: First review of RBD support for primary storage
Posted by Wido den Hollander <wi...@widodh.nl>.
First: Thanks for reviewing!
On 07/06/2012 12:54 AM, Chiradeep Vittal wrote:
> I took a first glance at this. Really pleased about this feature. EBS-like
> scalable primary storage is within reach!
>
> A few comments:
> 1. I see quite a few blocks of code ( > 20 times?) that are like
> if (pool.getType() == StoragePoolType.RBD)
> I realize that there is existing code that does these kinds of checks
> as well. To me this can be solved simply by the "chain of responsibility"
> pattern: you hand over the operation to a configured chain of handlers.
> The first handler (usually) that says it can handle it, terminates the
> chain.
Yes, that would indeed be better. The current code is not very flexible,
it assumes everything is a regular file or block device, which is not
the case with RBD.
In the current code I saw no other way then checking for the storage
pool type multiple times.
> 2. 'user_info' can actually be pushed into the 'storage_pool_details'
> table. Generally we avoid modifying existing tables if we can.
I get that, but user_info is something that comes from java.net.URI,
just like host, port and name. So I figure that user_info was at the
right place in the storage_pool.
> 3. Copying a snapshot to secondary storage is desirable: to be consistent
> with other storage types, to be able to instantiate new volumes in other
> zones (when S3 support is available across the region). I'd like to
> understand the blockers here.
You can't copy a snapshot out of a RBD image to another destionation,
this is not supported by qemu-img.
root@stack02:~# qemu-img convert -f raw -O qcow2 -s wido
rbd:rbd/cloudstack:mon_host=stack02.ceph.widodh.nl:auth_supported=none
/root/wido-snapshot.qcow2
qemu-img: Failed to load snapshot
root@stack02:~#
Here I'm trying to extract the snapshot "wido" out of the image
"cloudstack" and copy it to a qcow2 image.
I prefer not to use the "rbd" CLI tool since that would bring another
dependency into the picture.
This could probably be fixed inside qemu-img, but that would involve
more patching to be done.
However, there is a warning here: RBD disks can become large, very
large. In public clouds there should be a way to disable this for
administrators, otherwise users could start snapshotting 5TB disks and
copying them to the secondary storage.
That would eat CPU and network capacity.
Wido
>
>
> On 7/2/12 5:59 AM, "Wido den Hollander" <wi...@widodh.nl> wrote:
>
>> Hi,
>>
>> On 29-06-12 17:59, Wido den Hollander wrote:
>>> Now, the RBD support for primary storage knows limitations:
>>>
>>> - It only works with KVM
>>>
>>> - You are NOT able to snapshot RBD volumes. This is due to CloudStack
>>> wanting to backup snapshots to the secondary storage and uses 'qemu-img
>>> convert' for this. That doesn't work with RBD, but it's also very
>>> inefficient.
>>>
>>> RBD supports native snapshots inside the Ceph cluster. RBD disks also
>>> have the potential to reach very large sizes. Disks of 1TB won't be the
>>> exception. It would stress your network heavily. I'm thinking about
>>> implementing "internal snapshots", but that is step #2. For now no
>>> snapshots.
>>>
>>> - You are able create a template from a RBD volume, but creating a new
>>> instance with RBD storage from a template is still a hit-and-miss.
>>> Working on that one.
>>>
>>
>> I just pushed a fix for creating instances from a template. That should
>> work now!
>>
>> Wido
>
Re: First review of RBD support for primary storage
Posted by Chiradeep Vittal <Ch...@citrix.com>.
I took a first glance at this. Really pleased about this feature. EBS-like
scalable primary storage is within reach!
A few comments:
1. I see quite a few blocks of code ( > 20 times?) that are like
if (pool.getType() == StoragePoolType.RBD)
I realize that there is existing code that does these kinds of checks
as well. To me this can be solved simply by the "chain of responsibility"
pattern: you hand over the operation to a configured chain of handlers.
The first handler (usually) that says it can handle it, terminates the
chain.
2. 'user_info' can actually be pushed into the 'storage_pool_details'
table. Generally we avoid modifying existing tables if we can.
3. Copying a snapshot to secondary storage is desirable: to be consistent
with other storage types, to be able to instantiate new volumes in other
zones (when S3 support is available across the region). I'd like to
understand the blockers here.
On 7/2/12 5:59 AM, "Wido den Hollander" <wi...@widodh.nl> wrote:
>Hi,
>
>On 29-06-12 17:59, Wido den Hollander wrote:
>> Now, the RBD support for primary storage knows limitations:
>>
>> - It only works with KVM
>>
>> - You are NOT able to snapshot RBD volumes. This is due to CloudStack
>> wanting to backup snapshots to the secondary storage and uses 'qemu-img
>> convert' for this. That doesn't work with RBD, but it's also very
>> inefficient.
>>
>> RBD supports native snapshots inside the Ceph cluster. RBD disks also
>> have the potential to reach very large sizes. Disks of 1TB won't be the
>> exception. It would stress your network heavily. I'm thinking about
>> implementing "internal snapshots", but that is step #2. For now no
>> snapshots.
>>
>> - You are able create a template from a RBD volume, but creating a new
>> instance with RBD storage from a template is still a hit-and-miss.
>> Working on that one.
>>
>
>I just pushed a fix for creating instances from a template. That should
>work now!
>
>Wido
Re: First review of RBD support for primary storage
Posted by Wido den Hollander <wi...@widodh.nl>.
Hi,
On 29-06-12 17:59, Wido den Hollander wrote:
> Now, the RBD support for primary storage knows limitations:
>
> - It only works with KVM
>
> - You are NOT able to snapshot RBD volumes. This is due to CloudStack
> wanting to backup snapshots to the secondary storage and uses 'qemu-img
> convert' for this. That doesn't work with RBD, but it's also very
> inefficient.
>
> RBD supports native snapshots inside the Ceph cluster. RBD disks also
> have the potential to reach very large sizes. Disks of 1TB won't be the
> exception. It would stress your network heavily. I'm thinking about
> implementing "internal snapshots", but that is step #2. For now no
> snapshots.
>
> - You are able create a template from a RBD volume, but creating a new
> instance with RBD storage from a template is still a hit-and-miss.
> Working on that one.
>
I just pushed a fix for creating instances from a template. That should
work now!
Wido