You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Jean-Francois Nadeau <th...@gmail.com> on 2019/07/17 02:24:03 UTC
Using S3/Minio as the only secondary storage
Hello Everyone,
I was wondering if it was common or even recommended to use an S3
compatible storage system as the only secondary storage provider ?
The environment is 4.11.3.0 with KVM (Centos 7.6), and our tier1 storage
solution also provides an S3 compatible object store (apparently Minio
under the hood).
I have always used NFS to install the SSVM templates and the install script
(cloud-install-sys-tmplt) only takes a mount point. How, if possible,
would I proceed with S3 only storage ?
best,
Jean-Francois
Re: Using S3/Minio as the only secondary storage
Posted by "Riepl, Gregor (SWISS TXT)" <Gr...@swisstxt.ch>.
Hi Jean-François
> I have always used NFS to install the SSVM templates and the install
> script (cloud-install-sys-tmplt) only takes a mount point. How, if
> possible, would I proceed with S3 only storage ?
CloudStack doesn't support object storage as a backend for the
secondary storage. You'd have to use something like s3fs-fuse[1],
ObjectiveFS[2] or RioFS[3].
But I have no idea how well that will work... Take into consideration
that secondary storage will be mounted on at least the Management
Server and the SSVMs.
I think NFS is your only option for now.
What you can do, however, is mounting the S3 object store on one
machine and exporting it as an NFS share to the other hosts.
Regards,
Gregor
[1] https://github.com/s3fs-fuse/s3fs-fuse
[2] https://objectivefs.com/
[3] https://github.com/skoobe/riofs
Re: Using S3/Minio as the only secondary storage
Posted by "Riepl, Gregor (SWISS TXT)" <Gr...@swisstxt.ch>.
Hi Jean-François
> I have always used NFS to install the SSVM templates and the install
> script (cloud-install-sys-tmplt) only takes a mount point. How, if
> possible, would I proceed with S3 only storage ?
CloudStack doesn't support object storage as a backend for the
secondary storage. You'd have to use something like s3fs-fuse[1],
ObjectiveFS[2] or RioFS[3].
But I have no idea how well that will work... Take into consideration
that secondary storage will be mounted on at least the Management
Server and the SSVMs.
I think NFS is your only option for now.
What you can do, however, is mounting the S3 object store on one
machine and exporting it as an NFS share to the other hosts.
Regards,
Gregor
[1] https://github.com/s3fs-fuse/s3fs-fuse
[2] https://objectivefs.com/
[3] https://github.com/skoobe/riofs
Re: Using S3/Minio as the only secondary storage
Posted by Will Stevens <ws...@cloudops.com>.
If Syed can't fill in the blanks sufficiently for you when you talk to him
later today, let me know and I will set up a call and loop in the team on
our side who have more details and we can walk through our experience and
knowledge on the topic.
Cheers,
*Will Stevens*
Chief Technology Officer
c 514.826.0190
<https://goo.gl/NYZ8KK>
On Wed, Jul 17, 2019 at 9:30 AM Jean-Francois Nadeau <th...@gmail.com>
wrote:
> Thanks Will,
>
> I remember having the discussion with Pierre-Luc on his use of Swift for
> templates. I was curious about the differences on S3 vs Swift for SS since
> looking at the CS UI when it comes to setting up an S3 image store... the
> NFS staging is optional. And this make sense to me if your object storage
> is fast and accessible locally, why the need for staging/caching. The
> documentation could mention if it is possible to use S3 secondary and
> nothing else, starting with if SSVM templates can be uploaded to a
> bucket. I will certainly ask Syed later today :)
>
> best
>
> Jfn
>
> On Wed, Jul 17, 2019 at 6:59 AM Will Stevens <ws...@cloudops.com>
> wrote:
>
> > Hey JF,
> > We use the Swift object store as the storage backend for secondary
> > storage. I have not tried the S3 integration, but the last time I looked
> > at the code for this (admittedly, a long time ago) the Swift and s3 logic
> > was more intertwined than I liked. The CloudOps/cloud.ca team had to do
> a
> > lot of work to get the Swift integration to a reasonable working state. I
> > believe all of our changes have been upstreamed quite some time ago. I
> > don't know if anyone is doing this for the S3 implementation.
> >
> > I can't speak to the S3 implementation because I have not looked at it
> in a
> > very long time, but the Swift implementation requires a "temporary NFS
> > staging area" that essentially acts kind of like a buffer between the
> > object store and primary storage when templates and such are used by the
> > hosts.
> >
> > I think Pierre-Luc and Syed have a clearer picture of all the moving
> > pieces, but that is a quick summary of what I know without digging in.
> >
> > Hope that helps.
> >
> > Cheers,
> >
> > Will
> >
> > On Tue, Jul 16, 2019, 10:24 PM Jean-Francois Nadeau <
> > the.jfnadeau@gmail.com>
> > wrote:
> >
> > > Hello Everyone,
> > >
> > > I was wondering if it was common or even recommended to use an S3
> > > compatible storage system as the only secondary storage provider ?
> > >
> > > The environment is 4.11.3.0 with KVM (Centos 7.6), and our tier1
> storage
> > > solution also provides an S3 compatible object store (apparently Minio
> > > under the hood).
> > >
> > > I have always used NFS to install the SSVM templates and the install
> > script
> > > (cloud-install-sys-tmplt) only takes a mount point. How, if possible,
> > > would I proceed with S3 only storage ?
> > >
> > > best,
> > >
> > > Jean-Francois
> > >
> >
>
Re: Using S3/Minio as the only secondary storage
Posted by Jean-Francois Nadeau <th...@gmail.com>.
Thanks Will,
I remember having the discussion with Pierre-Luc on his use of Swift for
templates. I was curious about the differences on S3 vs Swift for SS since
looking at the CS UI when it comes to setting up an S3 image store... the
NFS staging is optional. And this make sense to me if your object storage
is fast and accessible locally, why the need for staging/caching. The
documentation could mention if it is possible to use S3 secondary and
nothing else, starting with if SSVM templates can be uploaded to a
bucket. I will certainly ask Syed later today :)
best
Jfn
On Wed, Jul 17, 2019 at 6:59 AM Will Stevens <ws...@cloudops.com> wrote:
> Hey JF,
> We use the Swift object store as the storage backend for secondary
> storage. I have not tried the S3 integration, but the last time I looked
> at the code for this (admittedly, a long time ago) the Swift and s3 logic
> was more intertwined than I liked. The CloudOps/cloud.ca team had to do a
> lot of work to get the Swift integration to a reasonable working state. I
> believe all of our changes have been upstreamed quite some time ago. I
> don't know if anyone is doing this for the S3 implementation.
>
> I can't speak to the S3 implementation because I have not looked at it in a
> very long time, but the Swift implementation requires a "temporary NFS
> staging area" that essentially acts kind of like a buffer between the
> object store and primary storage when templates and such are used by the
> hosts.
>
> I think Pierre-Luc and Syed have a clearer picture of all the moving
> pieces, but that is a quick summary of what I know without digging in.
>
> Hope that helps.
>
> Cheers,
>
> Will
>
> On Tue, Jul 16, 2019, 10:24 PM Jean-Francois Nadeau <
> the.jfnadeau@gmail.com>
> wrote:
>
> > Hello Everyone,
> >
> > I was wondering if it was common or even recommended to use an S3
> > compatible storage system as the only secondary storage provider ?
> >
> > The environment is 4.11.3.0 with KVM (Centos 7.6), and our tier1 storage
> > solution also provides an S3 compatible object store (apparently Minio
> > under the hood).
> >
> > I have always used NFS to install the SSVM templates and the install
> script
> > (cloud-install-sys-tmplt) only takes a mount point. How, if possible,
> > would I proceed with S3 only storage ?
> >
> > best,
> >
> > Jean-Francois
> >
>
Re: Using S3/Minio as the only secondary storage
Posted by Jean-Francois Nadeau <th...@gmail.com>.
Thanks Will,
I remember having the discussion with Pierre-Luc on his use of Swift for
templates. I was curious about the differences on S3 vs Swift for SS since
looking at the CS UI when it comes to setting up an S3 image store... the
NFS staging is optional. And this make sense to me if your object storage
is fast and accessible locally, why the need for staging/caching. The
documentation could mention if it is possible to use S3 secondary and
nothing else, starting with if SSVM templates can be uploaded to a
bucket. I will certainly ask Syed later today :)
best
Jfn
On Wed, Jul 17, 2019 at 6:59 AM Will Stevens <ws...@cloudops.com> wrote:
> Hey JF,
> We use the Swift object store as the storage backend for secondary
> storage. I have not tried the S3 integration, but the last time I looked
> at the code for this (admittedly, a long time ago) the Swift and s3 logic
> was more intertwined than I liked. The CloudOps/cloud.ca team had to do a
> lot of work to get the Swift integration to a reasonable working state. I
> believe all of our changes have been upstreamed quite some time ago. I
> don't know if anyone is doing this for the S3 implementation.
>
> I can't speak to the S3 implementation because I have not looked at it in a
> very long time, but the Swift implementation requires a "temporary NFS
> staging area" that essentially acts kind of like a buffer between the
> object store and primary storage when templates and such are used by the
> hosts.
>
> I think Pierre-Luc and Syed have a clearer picture of all the moving
> pieces, but that is a quick summary of what I know without digging in.
>
> Hope that helps.
>
> Cheers,
>
> Will
>
> On Tue, Jul 16, 2019, 10:24 PM Jean-Francois Nadeau <
> the.jfnadeau@gmail.com>
> wrote:
>
> > Hello Everyone,
> >
> > I was wondering if it was common or even recommended to use an S3
> > compatible storage system as the only secondary storage provider ?
> >
> > The environment is 4.11.3.0 with KVM (Centos 7.6), and our tier1 storage
> > solution also provides an S3 compatible object store (apparently Minio
> > under the hood).
> >
> > I have always used NFS to install the SSVM templates and the install
> script
> > (cloud-install-sys-tmplt) only takes a mount point. How, if possible,
> > would I proceed with S3 only storage ?
> >
> > best,
> >
> > Jean-Francois
> >
>
Re: Using S3/Minio as the only secondary storage
Posted by Will Stevens <ws...@cloudops.com>.
Hey JF,
We use the Swift object store as the storage backend for secondary
storage. I have not tried the S3 integration, but the last time I looked
at the code for this (admittedly, a long time ago) the Swift and s3 logic
was more intertwined than I liked. The CloudOps/cloud.ca team had to do a
lot of work to get the Swift integration to a reasonable working state. I
believe all of our changes have been upstreamed quite some time ago. I
don't know if anyone is doing this for the S3 implementation.
I can't speak to the S3 implementation because I have not looked at it in a
very long time, but the Swift implementation requires a "temporary NFS
staging area" that essentially acts kind of like a buffer between the
object store and primary storage when templates and such are used by the
hosts.
I think Pierre-Luc and Syed have a clearer picture of all the moving
pieces, but that is a quick summary of what I know without digging in.
Hope that helps.
Cheers,
Will
On Tue, Jul 16, 2019, 10:24 PM Jean-Francois Nadeau <th...@gmail.com>
wrote:
> Hello Everyone,
>
> I was wondering if it was common or even recommended to use an S3
> compatible storage system as the only secondary storage provider ?
>
> The environment is 4.11.3.0 with KVM (Centos 7.6), and our tier1 storage
> solution also provides an S3 compatible object store (apparently Minio
> under the hood).
>
> I have always used NFS to install the SSVM templates and the install script
> (cloud-install-sys-tmplt) only takes a mount point. How, if possible,
> would I proceed with S3 only storage ?
>
> best,
>
> Jean-Francois
>