You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Nux! <nu...@li.nux.ro> on 2016/07/19 12:38:35 UTC
State of the S3 secondary storage
Hello,
Could anyone summarise the state of S3 (and clones) for secondary storage?
I read at some point that it's the only secondary storage type supported inter-zone, so it would appeal from a DR pov, but not sure how mature it is or if there are any serious gotchas involved.
Thanks
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
Re: State of the S3 secondary storage
Posted by Nux! <nu...@li.nux.ro>.
That's a great idea Ilya, but it would indeed "discriminate" against VMware, HyperV and others.
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "ilya" <il...@gmail.com>
> To: users@cloudstack.apache.org
> Sent: Wednesday, 20 July, 2016 08:49:53
> Subject: Re: State of the S3 secondary storage
> We've been kicking around idea internally on abstracting secondary
> storage entirely and offload it to cloudstack agent.
>
> Our biggest gripe is NFS.
>
> If we can expose S3 (over HTTPs) like service and offload to cloudstack
> agent - we will remove large dependency.
>
> The only issue we see - is vmware, in which case it will have to be
> backed by NFS...
>
>
>
>
>
> On 7/19/16 2:37 PM, Nux! wrote:
>> Glusterfs ;)
Re: State of the S3 secondary storage
Posted by ilya <il...@gmail.com>.
We've been kicking around idea internally on abstracting secondary
storage entirely and offload it to cloudstack agent.
Our biggest gripe is NFS.
If we can expose S3 (over HTTPs) like service and offload to cloudstack
agent - we will remove large dependency.
The only issue we see - is vmware, in which case it will have to be
backed by NFS...
On 7/19/16 2:37 PM, Nux! wrote:
> Glusterfs ;)
>
Re: State of the S3 secondary storage
Posted by Nux! <nu...@li.nux.ro>.
Glusterfs ;)
--
Sent from my mobile device. Please excuse my brevity.
On 19 July 2016 21:24:45 BST, Will Stevens <ws...@cloudops.com> wrote:
>Oh ya. Gotta love NFS syncing to servers halfway around the world. :P
>
>*Will STEVENS*
>Lead Developer
>
>*CloudOps* *| *Cloud Solutions Experts
>420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>w cloudops.com *|* tw @CloudOps_
>
>On Tue, Jul 19, 2016 at 4:06 PM, Syed Ahmed <sa...@cloudops.com>
>wrote:
>
>> Right, but is that an artificial limitation imposed by Cloudstack of
>is
>> there something that will break if we make NFS region wide? If not,
>we can
>> look at moving the NFS to a region wide storage.
>>
>> -Syed
>>
>> On Tue, Jul 19, 2016 at 3:54 PM, Will Stevens <ws...@cloudops.com>
>> wrote:
>>
>> > Probably because ACS does not treat it as region wide, unlike the
>object
>> > storage implementations.
>> >
>> > *Will STEVENS*
>> > Lead Developer
>> >
>> > *CloudOps* *| *Cloud Solutions Experts
>> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> > w cloudops.com *|* tw @CloudOps_
>> >
>> > On Tue, Jul 19, 2016 at 2:52 PM, Syed Ahmed <sa...@cloudops.com>
>wrote:
>> >
>> > > Is there a reason why NFS cannot be region-wide?
>> > >
>> > > -Syed
>> > >
>> > > On Tue, Jul 19, 2016 at 2:08 PM, Nux! <nu...@li.nux.ro> wrote:
>> > >
>> > > > Thanks for your input guys.
>> > > >
>> > > > I think I'll stay with NFS for now after your replies, but it'd
>be
>> > pretty
>> > > > sweet the day when we'll be having region wide secondary
>storage.
>> > > >
>> > > > I'll need to think of another way to provide DR, perhaps some
>> automatic
>> > > > snapshot+download of sorts.
>> > > >
>> > > > Lucian
>> > > >
>> > > > --
>> > > > Sent from the Delta quadrant using Borg technology!
>> > > >
>> > > > Nux!
>> > > > www.nux.ro
>> > > >
>> > > > ----- Original Message -----
>> > > > > From: "Syed Ahmed" <sa...@cloudops.com>
>> > > > > To: dev@cloudstack.apache.org
>> > > > > Cc: "Nux!" <nu...@li.nux.ro>, "Cloudstack Users List" <
>> > > > users@cloudstack.apache.org>
>> > > > > Sent: Tuesday, 19 July, 2016 17:06:52
>> > > > > Subject: Re: State of the S3 secondary storage
>> > > >
>> > > > > As Will put it, the implementation for using Object store as
>> > secondary
>> > > > > storage relies on using Staging NFS. You would also loose the
>> > > > functionality
>> > > > > to do differential snapshots. All your snapshots will
>basically be
>> > full
>> > > > > volume snapshots. So, if you have a lot of snapshots being
>taken at
>> > > > > scheduled intervals, you might end up with wasting a lot of
>space
>> on
>> > > your
>> > > > > S3. Not to mention all of this has to go through the NFS
>which can
>> > (an
>> > > in
>> > > > > our experience has) fill up preventing you from creating new
>VMs
>> > > (because
>> > > > > templates cannot be downloaded). However you do benefit from
>a
>> region
>> > > > wide
>> > > > > Image store. We've worked around the NFS problem by basically
>> having
>> > > > > multiple staging NFS stores.
>> > > > >
>> > > > > I have plans to make Object stores better integrated in
>Cloudstack.
>> > The
>> > > > > holy grail would be to avoid staging NFS completely (or
>atleast in
>> > the
>> > > > > upload path). But for now we have to live with the
>limitations.
>> > > > >
>> > > > > -Syed
>> > > > >
>> > > > >
>> > > > > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <
>> > wstevens@cloudops.com>
>> > > > > wrote:
>> > > > >
>> > > > >> We are using Swift as secondary storage. It also offers
>> inter-zone
>> > > > >> secondary storage. We have had to do fixes to make it work
>> > correctly,
>> > > > but
>> > > > >> we have pushed those fixes upstream so they are available in
>4.9
>> > (and
>> > > > some
>> > > > >> fixes earlier than that).
>> > > > >>
>> > > > >> Like Wido said, the staging NFS is still needed which is a
>real
>> > > problem
>> > > > >> with these implementations. You end up copying the data
>many
>> times
>> > > and
>> > > > you
>> > > > >> will run into problems if you have lots of customers doing
>> scheduled
>> > > > >> snapshots at the same time because you can fill your NFS
>staging
>> > area
>> > > > >> unless you give it a lot of space to work with.
>> > > > >>
>> > > > >> The object storage integrations are working, but they are
>not
>> > designed
>> > > > very
>> > > > >> well at present. You have code for both S3 and Swift
>basically
>> just
>> > > > >> separated with conditionals and much of the logic is shared.
> This
>> > is
>> > > > very
>> > > > >> brittle because making changes in one integration can (and
>> > > historically
>> > > > >> has) broken the other implementation. This is a relatively
>big
>> > > problem
>> > > > >> because most people working with either Swift or S3 will
>only have
>> > the
>> > > > >> ability to test one of them, so they don't really have the
>ability
>> > to
>> > > > test
>> > > > >> if they have broken the other. It does not help that there
>is
>> Zero
>> > CI
>> > > > >> coverage run for either the Swift or S3 integrations due to
>the
>> > > > difficulty
>> > > > >> building and tearing down object storage environments on the
>fly.
>> > > > >>
>> > > > >> All in all, yes they should be working, but expect a bit of
>a
>> bumpy
>> > > > road.
>> > > > >>
>> > > > >> Hope that helps,
>> > > > >>
>> > > > >> *Will STEVENS*
>> > > > >> Lead Developer
>> > > > >>
>> > > > >> *CloudOps* *| *Cloud Solutions Experts
>> > > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> > > > >> w cloudops.com *|* tw @CloudOps_
>> > > > >>
>> > > > >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <
>> > wido@widodh.nl>
>> > > > >> wrote:
>> > > > >>
>> > > > >> >
>> > > > >> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
>> > > > >> > >
>> > > > >> > >
>> > > > >> > > Hello,
>> > > > >> > >
>> > > > >> > > Could anyone summarise the state of S3 (and clones) for
>> > secondary
>> > > > >> > storage?
>> > > > >> > > I read at some point that it's the only secondary
>storage type
>> > > > >> supported
>> > > > >> > inter-zone, so it would appeal from a DR pov, but not sure
>how
>> > > mature
>> > > > it
>> > > > >> is
>> > > > >> > or if there are any serious gotchas involved.
>> > > > >> > >
>> > > > >> >
>> > > > >> > We have it running for our largest region (Amsterdam) with
>> Ceph's
>> > > > RADOS
>> > > > >> > Gateway as backend. We had some issues, but they seem to
>be
>> fixed
>> > > > after
>> > > > >> our
>> > > > >> > latest patches.
>> > > > >> >
>> > > > >> > You still need the staging NFS though, so that doesn't
>solve
>> > > anything.
>> > > > >> >
>> > > > >> > Wido
>> > > > >> >
>> > > > >> > > Thanks
>> > > > >> > > Lucian
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > > Sent from the Delta quadrant using Borg technology!
>> > > > >> > >
>> > > > >> > > Nux!
>> > > > >> > > www.nux.ro
>> > > > >> >
>> > > >
>> > >
>> >
>>
Re: State of the S3 secondary storage
Posted by Nux! <nu...@li.nux.ro>.
Glusterfs ;)
--
Sent from my mobile device. Please excuse my brevity.
On 19 July 2016 21:24:45 BST, Will Stevens <ws...@cloudops.com> wrote:
>Oh ya. Gotta love NFS syncing to servers halfway around the world. :P
>
>*Will STEVENS*
>Lead Developer
>
>*CloudOps* *| *Cloud Solutions Experts
>420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>w cloudops.com *|* tw @CloudOps_
>
>On Tue, Jul 19, 2016 at 4:06 PM, Syed Ahmed <sa...@cloudops.com>
>wrote:
>
>> Right, but is that an artificial limitation imposed by Cloudstack of
>is
>> there something that will break if we make NFS region wide? If not,
>we can
>> look at moving the NFS to a region wide storage.
>>
>> -Syed
>>
>> On Tue, Jul 19, 2016 at 3:54 PM, Will Stevens <ws...@cloudops.com>
>> wrote:
>>
>> > Probably because ACS does not treat it as region wide, unlike the
>object
>> > storage implementations.
>> >
>> > *Will STEVENS*
>> > Lead Developer
>> >
>> > *CloudOps* *| *Cloud Solutions Experts
>> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> > w cloudops.com *|* tw @CloudOps_
>> >
>> > On Tue, Jul 19, 2016 at 2:52 PM, Syed Ahmed <sa...@cloudops.com>
>wrote:
>> >
>> > > Is there a reason why NFS cannot be region-wide?
>> > >
>> > > -Syed
>> > >
>> > > On Tue, Jul 19, 2016 at 2:08 PM, Nux! <nu...@li.nux.ro> wrote:
>> > >
>> > > > Thanks for your input guys.
>> > > >
>> > > > I think I'll stay with NFS for now after your replies, but it'd
>be
>> > pretty
>> > > > sweet the day when we'll be having region wide secondary
>storage.
>> > > >
>> > > > I'll need to think of another way to provide DR, perhaps some
>> automatic
>> > > > snapshot+download of sorts.
>> > > >
>> > > > Lucian
>> > > >
>> > > > --
>> > > > Sent from the Delta quadrant using Borg technology!
>> > > >
>> > > > Nux!
>> > > > www.nux.ro
>> > > >
>> > > > ----- Original Message -----
>> > > > > From: "Syed Ahmed" <sa...@cloudops.com>
>> > > > > To: dev@cloudstack.apache.org
>> > > > > Cc: "Nux!" <nu...@li.nux.ro>, "Cloudstack Users List" <
>> > > > users@cloudstack.apache.org>
>> > > > > Sent: Tuesday, 19 July, 2016 17:06:52
>> > > > > Subject: Re: State of the S3 secondary storage
>> > > >
>> > > > > As Will put it, the implementation for using Object store as
>> > secondary
>> > > > > storage relies on using Staging NFS. You would also loose the
>> > > > functionality
>> > > > > to do differential snapshots. All your snapshots will
>basically be
>> > full
>> > > > > volume snapshots. So, if you have a lot of snapshots being
>taken at
>> > > > > scheduled intervals, you might end up with wasting a lot of
>space
>> on
>> > > your
>> > > > > S3. Not to mention all of this has to go through the NFS
>which can
>> > (an
>> > > in
>> > > > > our experience has) fill up preventing you from creating new
>VMs
>> > > (because
>> > > > > templates cannot be downloaded). However you do benefit from
>a
>> region
>> > > > wide
>> > > > > Image store. We've worked around the NFS problem by basically
>> having
>> > > > > multiple staging NFS stores.
>> > > > >
>> > > > > I have plans to make Object stores better integrated in
>Cloudstack.
>> > The
>> > > > > holy grail would be to avoid staging NFS completely (or
>atleast in
>> > the
>> > > > > upload path). But for now we have to live with the
>limitations.
>> > > > >
>> > > > > -Syed
>> > > > >
>> > > > >
>> > > > > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <
>> > wstevens@cloudops.com>
>> > > > > wrote:
>> > > > >
>> > > > >> We are using Swift as secondary storage. It also offers
>> inter-zone
>> > > > >> secondary storage. We have had to do fixes to make it work
>> > correctly,
>> > > > but
>> > > > >> we have pushed those fixes upstream so they are available in
>4.9
>> > (and
>> > > > some
>> > > > >> fixes earlier than that).
>> > > > >>
>> > > > >> Like Wido said, the staging NFS is still needed which is a
>real
>> > > problem
>> > > > >> with these implementations. You end up copying the data
>many
>> times
>> > > and
>> > > > you
>> > > > >> will run into problems if you have lots of customers doing
>> scheduled
>> > > > >> snapshots at the same time because you can fill your NFS
>staging
>> > area
>> > > > >> unless you give it a lot of space to work with.
>> > > > >>
>> > > > >> The object storage integrations are working, but they are
>not
>> > designed
>> > > > very
>> > > > >> well at present. You have code for both S3 and Swift
>basically
>> just
>> > > > >> separated with conditionals and much of the logic is shared.
> This
>> > is
>> > > > very
>> > > > >> brittle because making changes in one integration can (and
>> > > historically
>> > > > >> has) broken the other implementation. This is a relatively
>big
>> > > problem
>> > > > >> because most people working with either Swift or S3 will
>only have
>> > the
>> > > > >> ability to test one of them, so they don't really have the
>ability
>> > to
>> > > > test
>> > > > >> if they have broken the other. It does not help that there
>is
>> Zero
>> > CI
>> > > > >> coverage run for either the Swift or S3 integrations due to
>the
>> > > > difficulty
>> > > > >> building and tearing down object storage environments on the
>fly.
>> > > > >>
>> > > > >> All in all, yes they should be working, but expect a bit of
>a
>> bumpy
>> > > > road.
>> > > > >>
>> > > > >> Hope that helps,
>> > > > >>
>> > > > >> *Will STEVENS*
>> > > > >> Lead Developer
>> > > > >>
>> > > > >> *CloudOps* *| *Cloud Solutions Experts
>> > > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> > > > >> w cloudops.com *|* tw @CloudOps_
>> > > > >>
>> > > > >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <
>> > wido@widodh.nl>
>> > > > >> wrote:
>> > > > >>
>> > > > >> >
>> > > > >> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
>> > > > >> > >
>> > > > >> > >
>> > > > >> > > Hello,
>> > > > >> > >
>> > > > >> > > Could anyone summarise the state of S3 (and clones) for
>> > secondary
>> > > > >> > storage?
>> > > > >> > > I read at some point that it's the only secondary
>storage type
>> > > > >> supported
>> > > > >> > inter-zone, so it would appeal from a DR pov, but not sure
>how
>> > > mature
>> > > > it
>> > > > >> is
>> > > > >> > or if there are any serious gotchas involved.
>> > > > >> > >
>> > > > >> >
>> > > > >> > We have it running for our largest region (Amsterdam) with
>> Ceph's
>> > > > RADOS
>> > > > >> > Gateway as backend. We had some issues, but they seem to
>be
>> fixed
>> > > > after
>> > > > >> our
>> > > > >> > latest patches.
>> > > > >> >
>> > > > >> > You still need the staging NFS though, so that doesn't
>solve
>> > > anything.
>> > > > >> >
>> > > > >> > Wido
>> > > > >> >
>> > > > >> > > Thanks
>> > > > >> > > Lucian
>> > > > >> > >
>> > > > >> > > --
>> > > > >> > > Sent from the Delta quadrant using Borg technology!
>> > > > >> > >
>> > > > >> > > Nux!
>> > > > >> > > www.nux.ro
>> > > > >> >
>> > > >
>> > >
>> >
>>
Re: State of the S3 secondary storage
Posted by Will Stevens <ws...@cloudops.com>.
Oh ya. Gotta love NFS syncing to servers halfway around the world. :P
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Tue, Jul 19, 2016 at 4:06 PM, Syed Ahmed <sa...@cloudops.com> wrote:
> Right, but is that an artificial limitation imposed by Cloudstack of is
> there something that will break if we make NFS region wide? If not, we can
> look at moving the NFS to a region wide storage.
>
> -Syed
>
> On Tue, Jul 19, 2016 at 3:54 PM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > Probably because ACS does not treat it as region wide, unlike the object
> > storage implementations.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Tue, Jul 19, 2016 at 2:52 PM, Syed Ahmed <sa...@cloudops.com> wrote:
> >
> > > Is there a reason why NFS cannot be region-wide?
> > >
> > > -Syed
> > >
> > > On Tue, Jul 19, 2016 at 2:08 PM, Nux! <nu...@li.nux.ro> wrote:
> > >
> > > > Thanks for your input guys.
> > > >
> > > > I think I'll stay with NFS for now after your replies, but it'd be
> > pretty
> > > > sweet the day when we'll be having region wide secondary storage.
> > > >
> > > > I'll need to think of another way to provide DR, perhaps some
> automatic
> > > > snapshot+download of sorts.
> > > >
> > > > Lucian
> > > >
> > > > --
> > > > Sent from the Delta quadrant using Borg technology!
> > > >
> > > > Nux!
> > > > www.nux.ro
> > > >
> > > > ----- Original Message -----
> > > > > From: "Syed Ahmed" <sa...@cloudops.com>
> > > > > To: dev@cloudstack.apache.org
> > > > > Cc: "Nux!" <nu...@li.nux.ro>, "Cloudstack Users List" <
> > > > users@cloudstack.apache.org>
> > > > > Sent: Tuesday, 19 July, 2016 17:06:52
> > > > > Subject: Re: State of the S3 secondary storage
> > > >
> > > > > As Will put it, the implementation for using Object store as
> > secondary
> > > > > storage relies on using Staging NFS. You would also loose the
> > > > functionality
> > > > > to do differential snapshots. All your snapshots will basically be
> > full
> > > > > volume snapshots. So, if you have a lot of snapshots being taken at
> > > > > scheduled intervals, you might end up with wasting a lot of space
> on
> > > your
> > > > > S3. Not to mention all of this has to go through the NFS which can
> > (an
> > > in
> > > > > our experience has) fill up preventing you from creating new VMs
> > > (because
> > > > > templates cannot be downloaded). However you do benefit from a
> region
> > > > wide
> > > > > Image store. We've worked around the NFS problem by basically
> having
> > > > > multiple staging NFS stores.
> > > > >
> > > > > I have plans to make Object stores better integrated in Cloudstack.
> > The
> > > > > holy grail would be to avoid staging NFS completely (or atleast in
> > the
> > > > > upload path). But for now we have to live with the limitations.
> > > > >
> > > > > -Syed
> > > > >
> > > > >
> > > > > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <
> > wstevens@cloudops.com>
> > > > > wrote:
> > > > >
> > > > >> We are using Swift as secondary storage. It also offers
> inter-zone
> > > > >> secondary storage. We have had to do fixes to make it work
> > correctly,
> > > > but
> > > > >> we have pushed those fixes upstream so they are available in 4.9
> > (and
> > > > some
> > > > >> fixes earlier than that).
> > > > >>
> > > > >> Like Wido said, the staging NFS is still needed which is a real
> > > problem
> > > > >> with these implementations. You end up copying the data many
> times
> > > and
> > > > you
> > > > >> will run into problems if you have lots of customers doing
> scheduled
> > > > >> snapshots at the same time because you can fill your NFS staging
> > area
> > > > >> unless you give it a lot of space to work with.
> > > > >>
> > > > >> The object storage integrations are working, but they are not
> > designed
> > > > very
> > > > >> well at present. You have code for both S3 and Swift basically
> just
> > > > >> separated with conditionals and much of the logic is shared. This
> > is
> > > > very
> > > > >> brittle because making changes in one integration can (and
> > > historically
> > > > >> has) broken the other implementation. This is a relatively big
> > > problem
> > > > >> because most people working with either Swift or S3 will only have
> > the
> > > > >> ability to test one of them, so they don't really have the ability
> > to
> > > > test
> > > > >> if they have broken the other. It does not help that there is
> Zero
> > CI
> > > > >> coverage run for either the Swift or S3 integrations due to the
> > > > difficulty
> > > > >> building and tearing down object storage environments on the fly.
> > > > >>
> > > > >> All in all, yes they should be working, but expect a bit of a
> bumpy
> > > > road.
> > > > >>
> > > > >> Hope that helps,
> > > > >>
> > > > >> *Will STEVENS*
> > > > >> Lead Developer
> > > > >>
> > > > >> *CloudOps* *| *Cloud Solutions Experts
> > > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > >> w cloudops.com *|* tw @CloudOps_
> > > > >>
> > > > >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <
> > wido@widodh.nl>
> > > > >> wrote:
> > > > >>
> > > > >> >
> > > > >> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
> > > > >> > >
> > > > >> > >
> > > > >> > > Hello,
> > > > >> > >
> > > > >> > > Could anyone summarise the state of S3 (and clones) for
> > secondary
> > > > >> > storage?
> > > > >> > > I read at some point that it's the only secondary storage type
> > > > >> supported
> > > > >> > inter-zone, so it would appeal from a DR pov, but not sure how
> > > mature
> > > > it
> > > > >> is
> > > > >> > or if there are any serious gotchas involved.
> > > > >> > >
> > > > >> >
> > > > >> > We have it running for our largest region (Amsterdam) with
> Ceph's
> > > > RADOS
> > > > >> > Gateway as backend. We had some issues, but they seem to be
> fixed
> > > > after
> > > > >> our
> > > > >> > latest patches.
> > > > >> >
> > > > >> > You still need the staging NFS though, so that doesn't solve
> > > anything.
> > > > >> >
> > > > >> > Wido
> > > > >> >
> > > > >> > > Thanks
> > > > >> > > Lucian
> > > > >> > >
> > > > >> > > --
> > > > >> > > Sent from the Delta quadrant using Borg technology!
> > > > >> > >
> > > > >> > > Nux!
> > > > >> > > www.nux.ro
> > > > >> >
> > > >
> > >
> >
>
Re: State of the S3 secondary storage
Posted by Will Stevens <ws...@cloudops.com>.
Oh ya. Gotta love NFS syncing to servers halfway around the world. :P
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Tue, Jul 19, 2016 at 4:06 PM, Syed Ahmed <sa...@cloudops.com> wrote:
> Right, but is that an artificial limitation imposed by Cloudstack of is
> there something that will break if we make NFS region wide? If not, we can
> look at moving the NFS to a region wide storage.
>
> -Syed
>
> On Tue, Jul 19, 2016 at 3:54 PM, Will Stevens <ws...@cloudops.com>
> wrote:
>
> > Probably because ACS does not treat it as region wide, unlike the object
> > storage implementations.
> >
> > *Will STEVENS*
> > Lead Developer
> >
> > *CloudOps* *| *Cloud Solutions Experts
> > 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > w cloudops.com *|* tw @CloudOps_
> >
> > On Tue, Jul 19, 2016 at 2:52 PM, Syed Ahmed <sa...@cloudops.com> wrote:
> >
> > > Is there a reason why NFS cannot be region-wide?
> > >
> > > -Syed
> > >
> > > On Tue, Jul 19, 2016 at 2:08 PM, Nux! <nu...@li.nux.ro> wrote:
> > >
> > > > Thanks for your input guys.
> > > >
> > > > I think I'll stay with NFS for now after your replies, but it'd be
> > pretty
> > > > sweet the day when we'll be having region wide secondary storage.
> > > >
> > > > I'll need to think of another way to provide DR, perhaps some
> automatic
> > > > snapshot+download of sorts.
> > > >
> > > > Lucian
> > > >
> > > > --
> > > > Sent from the Delta quadrant using Borg technology!
> > > >
> > > > Nux!
> > > > www.nux.ro
> > > >
> > > > ----- Original Message -----
> > > > > From: "Syed Ahmed" <sa...@cloudops.com>
> > > > > To: dev@cloudstack.apache.org
> > > > > Cc: "Nux!" <nu...@li.nux.ro>, "Cloudstack Users List" <
> > > > users@cloudstack.apache.org>
> > > > > Sent: Tuesday, 19 July, 2016 17:06:52
> > > > > Subject: Re: State of the S3 secondary storage
> > > >
> > > > > As Will put it, the implementation for using Object store as
> > secondary
> > > > > storage relies on using Staging NFS. You would also loose the
> > > > functionality
> > > > > to do differential snapshots. All your snapshots will basically be
> > full
> > > > > volume snapshots. So, if you have a lot of snapshots being taken at
> > > > > scheduled intervals, you might end up with wasting a lot of space
> on
> > > your
> > > > > S3. Not to mention all of this has to go through the NFS which can
> > (an
> > > in
> > > > > our experience has) fill up preventing you from creating new VMs
> > > (because
> > > > > templates cannot be downloaded). However you do benefit from a
> region
> > > > wide
> > > > > Image store. We've worked around the NFS problem by basically
> having
> > > > > multiple staging NFS stores.
> > > > >
> > > > > I have plans to make Object stores better integrated in Cloudstack.
> > The
> > > > > holy grail would be to avoid staging NFS completely (or atleast in
> > the
> > > > > upload path). But for now we have to live with the limitations.
> > > > >
> > > > > -Syed
> > > > >
> > > > >
> > > > > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <
> > wstevens@cloudops.com>
> > > > > wrote:
> > > > >
> > > > >> We are using Swift as secondary storage. It also offers
> inter-zone
> > > > >> secondary storage. We have had to do fixes to make it work
> > correctly,
> > > > but
> > > > >> we have pushed those fixes upstream so they are available in 4.9
> > (and
> > > > some
> > > > >> fixes earlier than that).
> > > > >>
> > > > >> Like Wido said, the staging NFS is still needed which is a real
> > > problem
> > > > >> with these implementations. You end up copying the data many
> times
> > > and
> > > > you
> > > > >> will run into problems if you have lots of customers doing
> scheduled
> > > > >> snapshots at the same time because you can fill your NFS staging
> > area
> > > > >> unless you give it a lot of space to work with.
> > > > >>
> > > > >> The object storage integrations are working, but they are not
> > designed
> > > > very
> > > > >> well at present. You have code for both S3 and Swift basically
> just
> > > > >> separated with conditionals and much of the logic is shared. This
> > is
> > > > very
> > > > >> brittle because making changes in one integration can (and
> > > historically
> > > > >> has) broken the other implementation. This is a relatively big
> > > problem
> > > > >> because most people working with either Swift or S3 will only have
> > the
> > > > >> ability to test one of them, so they don't really have the ability
> > to
> > > > test
> > > > >> if they have broken the other. It does not help that there is
> Zero
> > CI
> > > > >> coverage run for either the Swift or S3 integrations due to the
> > > > difficulty
> > > > >> building and tearing down object storage environments on the fly.
> > > > >>
> > > > >> All in all, yes they should be working, but expect a bit of a
> bumpy
> > > > road.
> > > > >>
> > > > >> Hope that helps,
> > > > >>
> > > > >> *Will STEVENS*
> > > > >> Lead Developer
> > > > >>
> > > > >> *CloudOps* *| *Cloud Solutions Experts
> > > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > > >> w cloudops.com *|* tw @CloudOps_
> > > > >>
> > > > >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <
> > wido@widodh.nl>
> > > > >> wrote:
> > > > >>
> > > > >> >
> > > > >> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
> > > > >> > >
> > > > >> > >
> > > > >> > > Hello,
> > > > >> > >
> > > > >> > > Could anyone summarise the state of S3 (and clones) for
> > secondary
> > > > >> > storage?
> > > > >> > > I read at some point that it's the only secondary storage type
> > > > >> supported
> > > > >> > inter-zone, so it would appeal from a DR pov, but not sure how
> > > mature
> > > > it
> > > > >> is
> > > > >> > or if there are any serious gotchas involved.
> > > > >> > >
> > > > >> >
> > > > >> > We have it running for our largest region (Amsterdam) with
> Ceph's
> > > > RADOS
> > > > >> > Gateway as backend. We had some issues, but they seem to be
> fixed
> > > > after
> > > > >> our
> > > > >> > latest patches.
> > > > >> >
> > > > >> > You still need the staging NFS though, so that doesn't solve
> > > anything.
> > > > >> >
> > > > >> > Wido
> > > > >> >
> > > > >> > > Thanks
> > > > >> > > Lucian
> > > > >> > >
> > > > >> > > --
> > > > >> > > Sent from the Delta quadrant using Borg technology!
> > > > >> > >
> > > > >> > > Nux!
> > > > >> > > www.nux.ro
> > > > >> >
> > > >
> > >
> >
>
Re: State of the S3 secondary storage
Posted by Syed Ahmed <sa...@cloudops.com>.
Right, but is that an artificial limitation imposed by Cloudstack of is
there something that will break if we make NFS region wide? If not, we can
look at moving the NFS to a region wide storage.
-Syed
On Tue, Jul 19, 2016 at 3:54 PM, Will Stevens <ws...@cloudops.com> wrote:
> Probably because ACS does not treat it as region wide, unlike the object
> storage implementations.
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Tue, Jul 19, 2016 at 2:52 PM, Syed Ahmed <sa...@cloudops.com> wrote:
>
> > Is there a reason why NFS cannot be region-wide?
> >
> > -Syed
> >
> > On Tue, Jul 19, 2016 at 2:08 PM, Nux! <nu...@li.nux.ro> wrote:
> >
> > > Thanks for your input guys.
> > >
> > > I think I'll stay with NFS for now after your replies, but it'd be
> pretty
> > > sweet the day when we'll be having region wide secondary storage.
> > >
> > > I'll need to think of another way to provide DR, perhaps some automatic
> > > snapshot+download of sorts.
> > >
> > > Lucian
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> > >
> > > ----- Original Message -----
> > > > From: "Syed Ahmed" <sa...@cloudops.com>
> > > > To: dev@cloudstack.apache.org
> > > > Cc: "Nux!" <nu...@li.nux.ro>, "Cloudstack Users List" <
> > > users@cloudstack.apache.org>
> > > > Sent: Tuesday, 19 July, 2016 17:06:52
> > > > Subject: Re: State of the S3 secondary storage
> > >
> > > > As Will put it, the implementation for using Object store as
> secondary
> > > > storage relies on using Staging NFS. You would also loose the
> > > functionality
> > > > to do differential snapshots. All your snapshots will basically be
> full
> > > > volume snapshots. So, if you have a lot of snapshots being taken at
> > > > scheduled intervals, you might end up with wasting a lot of space on
> > your
> > > > S3. Not to mention all of this has to go through the NFS which can
> (an
> > in
> > > > our experience has) fill up preventing you from creating new VMs
> > (because
> > > > templates cannot be downloaded). However you do benefit from a region
> > > wide
> > > > Image store. We've worked around the NFS problem by basically having
> > > > multiple staging NFS stores.
> > > >
> > > > I have plans to make Object stores better integrated in Cloudstack.
> The
> > > > holy grail would be to avoid staging NFS completely (or atleast in
> the
> > > > upload path). But for now we have to live with the limitations.
> > > >
> > > > -Syed
> > > >
> > > >
> > > > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <
> wstevens@cloudops.com>
> > > > wrote:
> > > >
> > > >> We are using Swift as secondary storage. It also offers inter-zone
> > > >> secondary storage. We have had to do fixes to make it work
> correctly,
> > > but
> > > >> we have pushed those fixes upstream so they are available in 4.9
> (and
> > > some
> > > >> fixes earlier than that).
> > > >>
> > > >> Like Wido said, the staging NFS is still needed which is a real
> > problem
> > > >> with these implementations. You end up copying the data many times
> > and
> > > you
> > > >> will run into problems if you have lots of customers doing scheduled
> > > >> snapshots at the same time because you can fill your NFS staging
> area
> > > >> unless you give it a lot of space to work with.
> > > >>
> > > >> The object storage integrations are working, but they are not
> designed
> > > very
> > > >> well at present. You have code for both S3 and Swift basically just
> > > >> separated with conditionals and much of the logic is shared. This
> is
> > > very
> > > >> brittle because making changes in one integration can (and
> > historically
> > > >> has) broken the other implementation. This is a relatively big
> > problem
> > > >> because most people working with either Swift or S3 will only have
> the
> > > >> ability to test one of them, so they don't really have the ability
> to
> > > test
> > > >> if they have broken the other. It does not help that there is Zero
> CI
> > > >> coverage run for either the Swift or S3 integrations due to the
> > > difficulty
> > > >> building and tearing down object storage environments on the fly.
> > > >>
> > > >> All in all, yes they should be working, but expect a bit of a bumpy
> > > road.
> > > >>
> > > >> Hope that helps,
> > > >>
> > > >> *Will STEVENS*
> > > >> Lead Developer
> > > >>
> > > >> *CloudOps* *| *Cloud Solutions Experts
> > > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > > >> w cloudops.com *|* tw @CloudOps_
> > > >>
> > > >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <
> wido@widodh.nl>
> > > >> wrote:
> > > >>
> > > >> >
> > > >> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
> > > >> > >
> > > >> > >
> > > >> > > Hello,
> > > >> > >
> > > >> > > Could anyone summarise the state of S3 (and clones) for
> secondary
> > > >> > storage?
> > > >> > > I read at some point that it's the only secondary storage type
> > > >> supported
> > > >> > inter-zone, so it would appeal from a DR pov, but not sure how
> > mature
> > > it
> > > >> is
> > > >> > or if there are any serious gotchas involved.
> > > >> > >
> > > >> >
> > > >> > We have it running for our largest region (Amsterdam) with Ceph's
> > > RADOS
> > > >> > Gateway as backend. We had some issues, but they seem to be fixed
> > > after
> > > >> our
> > > >> > latest patches.
> > > >> >
> > > >> > You still need the staging NFS though, so that doesn't solve
> > anything.
> > > >> >
> > > >> > Wido
> > > >> >
> > > >> > > Thanks
> > > >> > > Lucian
> > > >> > >
> > > >> > > --
> > > >> > > Sent from the Delta quadrant using Borg technology!
> > > >> > >
> > > >> > > Nux!
> > > >> > > www.nux.ro
> > > >> >
> > >
> >
>
Re: State of the S3 secondary storage
Posted by Will Stevens <ws...@cloudops.com>.
Probably because ACS does not treat it as region wide, unlike the object
storage implementations.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Tue, Jul 19, 2016 at 2:52 PM, Syed Ahmed <sa...@cloudops.com> wrote:
> Is there a reason why NFS cannot be region-wide?
>
> -Syed
>
> On Tue, Jul 19, 2016 at 2:08 PM, Nux! <nu...@li.nux.ro> wrote:
>
> > Thanks for your input guys.
> >
> > I think I'll stay with NFS for now after your replies, but it'd be pretty
> > sweet the day when we'll be having region wide secondary storage.
> >
> > I'll need to think of another way to provide DR, perhaps some automatic
> > snapshot+download of sorts.
> >
> > Lucian
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > ----- Original Message -----
> > > From: "Syed Ahmed" <sa...@cloudops.com>
> > > To: dev@cloudstack.apache.org
> > > Cc: "Nux!" <nu...@li.nux.ro>, "Cloudstack Users List" <
> > users@cloudstack.apache.org>
> > > Sent: Tuesday, 19 July, 2016 17:06:52
> > > Subject: Re: State of the S3 secondary storage
> >
> > > As Will put it, the implementation for using Object store as secondary
> > > storage relies on using Staging NFS. You would also loose the
> > functionality
> > > to do differential snapshots. All your snapshots will basically be full
> > > volume snapshots. So, if you have a lot of snapshots being taken at
> > > scheduled intervals, you might end up with wasting a lot of space on
> your
> > > S3. Not to mention all of this has to go through the NFS which can (an
> in
> > > our experience has) fill up preventing you from creating new VMs
> (because
> > > templates cannot be downloaded). However you do benefit from a region
> > wide
> > > Image store. We've worked around the NFS problem by basically having
> > > multiple staging NFS stores.
> > >
> > > I have plans to make Object stores better integrated in Cloudstack. The
> > > holy grail would be to avoid staging NFS completely (or atleast in the
> > > upload path). But for now we have to live with the limitations.
> > >
> > > -Syed
> > >
> > >
> > > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > >> We are using Swift as secondary storage. It also offers inter-zone
> > >> secondary storage. We have had to do fixes to make it work correctly,
> > but
> > >> we have pushed those fixes upstream so they are available in 4.9 (and
> > some
> > >> fixes earlier than that).
> > >>
> > >> Like Wido said, the staging NFS is still needed which is a real
> problem
> > >> with these implementations. You end up copying the data many times
> and
> > you
> > >> will run into problems if you have lots of customers doing scheduled
> > >> snapshots at the same time because you can fill your NFS staging area
> > >> unless you give it a lot of space to work with.
> > >>
> > >> The object storage integrations are working, but they are not designed
> > very
> > >> well at present. You have code for both S3 and Swift basically just
> > >> separated with conditionals and much of the logic is shared. This is
> > very
> > >> brittle because making changes in one integration can (and
> historically
> > >> has) broken the other implementation. This is a relatively big
> problem
> > >> because most people working with either Swift or S3 will only have the
> > >> ability to test one of them, so they don't really have the ability to
> > test
> > >> if they have broken the other. It does not help that there is Zero CI
> > >> coverage run for either the Swift or S3 integrations due to the
> > difficulty
> > >> building and tearing down object storage environments on the fly.
> > >>
> > >> All in all, yes they should be working, but expect a bit of a bumpy
> > road.
> > >>
> > >> Hope that helps,
> > >>
> > >> *Will STEVENS*
> > >> Lead Developer
> > >>
> > >> *CloudOps* *| *Cloud Solutions Experts
> > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > >> w cloudops.com *|* tw @CloudOps_
> > >>
> > >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <wi...@widodh.nl>
> > >> wrote:
> > >>
> > >> >
> > >> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
> > >> > >
> > >> > >
> > >> > > Hello,
> > >> > >
> > >> > > Could anyone summarise the state of S3 (and clones) for secondary
> > >> > storage?
> > >> > > I read at some point that it's the only secondary storage type
> > >> supported
> > >> > inter-zone, so it would appeal from a DR pov, but not sure how
> mature
> > it
> > >> is
> > >> > or if there are any serious gotchas involved.
> > >> > >
> > >> >
> > >> > We have it running for our largest region (Amsterdam) with Ceph's
> > RADOS
> > >> > Gateway as backend. We had some issues, but they seem to be fixed
> > after
> > >> our
> > >> > latest patches.
> > >> >
> > >> > You still need the staging NFS though, so that doesn't solve
> anything.
> > >> >
> > >> > Wido
> > >> >
> > >> > > Thanks
> > >> > > Lucian
> > >> > >
> > >> > > --
> > >> > > Sent from the Delta quadrant using Borg technology!
> > >> > >
> > >> > > Nux!
> > >> > > www.nux.ro
> > >> >
> >
>
Re: State of the S3 secondary storage
Posted by Will Stevens <ws...@cloudops.com>.
Probably because ACS does not treat it as region wide, unlike the object
storage implementations.
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Tue, Jul 19, 2016 at 2:52 PM, Syed Ahmed <sa...@cloudops.com> wrote:
> Is there a reason why NFS cannot be region-wide?
>
> -Syed
>
> On Tue, Jul 19, 2016 at 2:08 PM, Nux! <nu...@li.nux.ro> wrote:
>
> > Thanks for your input guys.
> >
> > I think I'll stay with NFS for now after your replies, but it'd be pretty
> > sweet the day when we'll be having region wide secondary storage.
> >
> > I'll need to think of another way to provide DR, perhaps some automatic
> > snapshot+download of sorts.
> >
> > Lucian
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
> >
> > ----- Original Message -----
> > > From: "Syed Ahmed" <sa...@cloudops.com>
> > > To: dev@cloudstack.apache.org
> > > Cc: "Nux!" <nu...@li.nux.ro>, "Cloudstack Users List" <
> > users@cloudstack.apache.org>
> > > Sent: Tuesday, 19 July, 2016 17:06:52
> > > Subject: Re: State of the S3 secondary storage
> >
> > > As Will put it, the implementation for using Object store as secondary
> > > storage relies on using Staging NFS. You would also loose the
> > functionality
> > > to do differential snapshots. All your snapshots will basically be full
> > > volume snapshots. So, if you have a lot of snapshots being taken at
> > > scheduled intervals, you might end up with wasting a lot of space on
> your
> > > S3. Not to mention all of this has to go through the NFS which can (an
> in
> > > our experience has) fill up preventing you from creating new VMs
> (because
> > > templates cannot be downloaded). However you do benefit from a region
> > wide
> > > Image store. We've worked around the NFS problem by basically having
> > > multiple staging NFS stores.
> > >
> > > I have plans to make Object stores better integrated in Cloudstack. The
> > > holy grail would be to avoid staging NFS completely (or atleast in the
> > > upload path). But for now we have to live with the limitations.
> > >
> > > -Syed
> > >
> > >
> > > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <ws...@cloudops.com>
> > > wrote:
> > >
> > >> We are using Swift as secondary storage. It also offers inter-zone
> > >> secondary storage. We have had to do fixes to make it work correctly,
> > but
> > >> we have pushed those fixes upstream so they are available in 4.9 (and
> > some
> > >> fixes earlier than that).
> > >>
> > >> Like Wido said, the staging NFS is still needed which is a real
> problem
> > >> with these implementations. You end up copying the data many times
> and
> > you
> > >> will run into problems if you have lots of customers doing scheduled
> > >> snapshots at the same time because you can fill your NFS staging area
> > >> unless you give it a lot of space to work with.
> > >>
> > >> The object storage integrations are working, but they are not designed
> > very
> > >> well at present. You have code for both S3 and Swift basically just
> > >> separated with conditionals and much of the logic is shared. This is
> > very
> > >> brittle because making changes in one integration can (and
> historically
> > >> has) broken the other implementation. This is a relatively big
> problem
> > >> because most people working with either Swift or S3 will only have the
> > >> ability to test one of them, so they don't really have the ability to
> > test
> > >> if they have broken the other. It does not help that there is Zero CI
> > >> coverage run for either the Swift or S3 integrations due to the
> > difficulty
> > >> building and tearing down object storage environments on the fly.
> > >>
> > >> All in all, yes they should be working, but expect a bit of a bumpy
> > road.
> > >>
> > >> Hope that helps,
> > >>
> > >> *Will STEVENS*
> > >> Lead Developer
> > >>
> > >> *CloudOps* *| *Cloud Solutions Experts
> > >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> > >> w cloudops.com *|* tw @CloudOps_
> > >>
> > >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <wi...@widodh.nl>
> > >> wrote:
> > >>
> > >> >
> > >> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
> > >> > >
> > >> > >
> > >> > > Hello,
> > >> > >
> > >> > > Could anyone summarise the state of S3 (and clones) for secondary
> > >> > storage?
> > >> > > I read at some point that it's the only secondary storage type
> > >> supported
> > >> > inter-zone, so it would appeal from a DR pov, but not sure how
> mature
> > it
> > >> is
> > >> > or if there are any serious gotchas involved.
> > >> > >
> > >> >
> > >> > We have it running for our largest region (Amsterdam) with Ceph's
> > RADOS
> > >> > Gateway as backend. We had some issues, but they seem to be fixed
> > after
> > >> our
> > >> > latest patches.
> > >> >
> > >> > You still need the staging NFS though, so that doesn't solve
> anything.
> > >> >
> > >> > Wido
> > >> >
> > >> > > Thanks
> > >> > > Lucian
> > >> > >
> > >> > > --
> > >> > > Sent from the Delta quadrant using Borg technology!
> > >> > >
> > >> > > Nux!
> > >> > > www.nux.ro
> > >> >
> >
>
Re: State of the S3 secondary storage
Posted by Syed Ahmed <sa...@cloudops.com>.
Is there a reason why NFS cannot be region-wide?
-Syed
On Tue, Jul 19, 2016 at 2:08 PM, Nux! <nu...@li.nux.ro> wrote:
> Thanks for your input guys.
>
> I think I'll stay with NFS for now after your replies, but it'd be pretty
> sweet the day when we'll be having region wide secondary storage.
>
> I'll need to think of another way to provide DR, perhaps some automatic
> snapshot+download of sorts.
>
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
>
> ----- Original Message -----
> > From: "Syed Ahmed" <sa...@cloudops.com>
> > To: dev@cloudstack.apache.org
> > Cc: "Nux!" <nu...@li.nux.ro>, "Cloudstack Users List" <
> users@cloudstack.apache.org>
> > Sent: Tuesday, 19 July, 2016 17:06:52
> > Subject: Re: State of the S3 secondary storage
>
> > As Will put it, the implementation for using Object store as secondary
> > storage relies on using Staging NFS. You would also loose the
> functionality
> > to do differential snapshots. All your snapshots will basically be full
> > volume snapshots. So, if you have a lot of snapshots being taken at
> > scheduled intervals, you might end up with wasting a lot of space on your
> > S3. Not to mention all of this has to go through the NFS which can (an in
> > our experience has) fill up preventing you from creating new VMs (because
> > templates cannot be downloaded). However you do benefit from a region
> wide
> > Image store. We've worked around the NFS problem by basically having
> > multiple staging NFS stores.
> >
> > I have plans to make Object stores better integrated in Cloudstack. The
> > holy grail would be to avoid staging NFS completely (or atleast in the
> > upload path). But for now we have to live with the limitations.
> >
> > -Syed
> >
> >
> > On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <ws...@cloudops.com>
> > wrote:
> >
> >> We are using Swift as secondary storage. It also offers inter-zone
> >> secondary storage. We have had to do fixes to make it work correctly,
> but
> >> we have pushed those fixes upstream so they are available in 4.9 (and
> some
> >> fixes earlier than that).
> >>
> >> Like Wido said, the staging NFS is still needed which is a real problem
> >> with these implementations. You end up copying the data many times and
> you
> >> will run into problems if you have lots of customers doing scheduled
> >> snapshots at the same time because you can fill your NFS staging area
> >> unless you give it a lot of space to work with.
> >>
> >> The object storage integrations are working, but they are not designed
> very
> >> well at present. You have code for both S3 and Swift basically just
> >> separated with conditionals and much of the logic is shared. This is
> very
> >> brittle because making changes in one integration can (and historically
> >> has) broken the other implementation. This is a relatively big problem
> >> because most people working with either Swift or S3 will only have the
> >> ability to test one of them, so they don't really have the ability to
> test
> >> if they have broken the other. It does not help that there is Zero CI
> >> coverage run for either the Swift or S3 integrations due to the
> difficulty
> >> building and tearing down object storage environments on the fly.
> >>
> >> All in all, yes they should be working, but expect a bit of a bumpy
> road.
> >>
> >> Hope that helps,
> >>
> >> *Will STEVENS*
> >> Lead Developer
> >>
> >> *CloudOps* *| *Cloud Solutions Experts
> >> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> >> w cloudops.com *|* tw @CloudOps_
> >>
> >> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <wi...@widodh.nl>
> >> wrote:
> >>
> >> >
> >> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
> >> > >
> >> > >
> >> > > Hello,
> >> > >
> >> > > Could anyone summarise the state of S3 (and clones) for secondary
> >> > storage?
> >> > > I read at some point that it's the only secondary storage type
> >> supported
> >> > inter-zone, so it would appeal from a DR pov, but not sure how mature
> it
> >> is
> >> > or if there are any serious gotchas involved.
> >> > >
> >> >
> >> > We have it running for our largest region (Amsterdam) with Ceph's
> RADOS
> >> > Gateway as backend. We had some issues, but they seem to be fixed
> after
> >> our
> >> > latest patches.
> >> >
> >> > You still need the staging NFS though, so that doesn't solve anything.
> >> >
> >> > Wido
> >> >
> >> > > Thanks
> >> > > Lucian
> >> > >
> >> > > --
> >> > > Sent from the Delta quadrant using Borg technology!
> >> > >
> >> > > Nux!
> >> > > www.nux.ro
> >> >
>
Re: State of the S3 secondary storage
Posted by Nux! <nu...@li.nux.ro>.
Thanks for your input guys.
I think I'll stay with NFS for now after your replies, but it'd be pretty sweet the day when we'll be having region wide secondary storage.
I'll need to think of another way to provide DR, perhaps some automatic snapshot+download of sorts.
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Syed Ahmed" <sa...@cloudops.com>
> To: dev@cloudstack.apache.org
> Cc: "Nux!" <nu...@li.nux.ro>, "Cloudstack Users List" <us...@cloudstack.apache.org>
> Sent: Tuesday, 19 July, 2016 17:06:52
> Subject: Re: State of the S3 secondary storage
> As Will put it, the implementation for using Object store as secondary
> storage relies on using Staging NFS. You would also loose the functionality
> to do differential snapshots. All your snapshots will basically be full
> volume snapshots. So, if you have a lot of snapshots being taken at
> scheduled intervals, you might end up with wasting a lot of space on your
> S3. Not to mention all of this has to go through the NFS which can (an in
> our experience has) fill up preventing you from creating new VMs (because
> templates cannot be downloaded). However you do benefit from a region wide
> Image store. We've worked around the NFS problem by basically having
> multiple staging NFS stores.
>
> I have plans to make Object stores better integrated in Cloudstack. The
> holy grail would be to avoid staging NFS completely (or atleast in the
> upload path). But for now we have to live with the limitations.
>
> -Syed
>
>
> On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
>> We are using Swift as secondary storage. It also offers inter-zone
>> secondary storage. We have had to do fixes to make it work correctly, but
>> we have pushed those fixes upstream so they are available in 4.9 (and some
>> fixes earlier than that).
>>
>> Like Wido said, the staging NFS is still needed which is a real problem
>> with these implementations. You end up copying the data many times and you
>> will run into problems if you have lots of customers doing scheduled
>> snapshots at the same time because you can fill your NFS staging area
>> unless you give it a lot of space to work with.
>>
>> The object storage integrations are working, but they are not designed very
>> well at present. You have code for both S3 and Swift basically just
>> separated with conditionals and much of the logic is shared. This is very
>> brittle because making changes in one integration can (and historically
>> has) broken the other implementation. This is a relatively big problem
>> because most people working with either Swift or S3 will only have the
>> ability to test one of them, so they don't really have the ability to test
>> if they have broken the other. It does not help that there is Zero CI
>> coverage run for either the Swift or S3 integrations due to the difficulty
>> building and tearing down object storage environments on the fly.
>>
>> All in all, yes they should be working, but expect a bit of a bumpy road.
>>
>> Hope that helps,
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> *CloudOps* *| *Cloud Solutions Experts
>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> w cloudops.com *|* tw @CloudOps_
>>
>> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <wi...@widodh.nl>
>> wrote:
>>
>> >
>> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
>> > >
>> > >
>> > > Hello,
>> > >
>> > > Could anyone summarise the state of S3 (and clones) for secondary
>> > storage?
>> > > I read at some point that it's the only secondary storage type
>> supported
>> > inter-zone, so it would appeal from a DR pov, but not sure how mature it
>> is
>> > or if there are any serious gotchas involved.
>> > >
>> >
>> > We have it running for our largest region (Amsterdam) with Ceph's RADOS
>> > Gateway as backend. We had some issues, but they seem to be fixed after
>> our
>> > latest patches.
>> >
>> > You still need the staging NFS though, so that doesn't solve anything.
>> >
>> > Wido
>> >
>> > > Thanks
>> > > Lucian
>> > >
>> > > --
>> > > Sent from the Delta quadrant using Borg technology!
>> > >
>> > > Nux!
>> > > www.nux.ro
>> >
Re: State of the S3 secondary storage
Posted by Nux! <nu...@li.nux.ro>.
Thanks for your input guys.
I think I'll stay with NFS for now after your replies, but it'd be pretty sweet the day when we'll be having region wide secondary storage.
I'll need to think of another way to provide DR, perhaps some automatic snapshot+download of sorts.
Lucian
--
Sent from the Delta quadrant using Borg technology!
Nux!
www.nux.ro
----- Original Message -----
> From: "Syed Ahmed" <sa...@cloudops.com>
> To: dev@cloudstack.apache.org
> Cc: "Nux!" <nu...@li.nux.ro>, "Cloudstack Users List" <us...@cloudstack.apache.org>
> Sent: Tuesday, 19 July, 2016 17:06:52
> Subject: Re: State of the S3 secondary storage
> As Will put it, the implementation for using Object store as secondary
> storage relies on using Staging NFS. You would also loose the functionality
> to do differential snapshots. All your snapshots will basically be full
> volume snapshots. So, if you have a lot of snapshots being taken at
> scheduled intervals, you might end up with wasting a lot of space on your
> S3. Not to mention all of this has to go through the NFS which can (an in
> our experience has) fill up preventing you from creating new VMs (because
> templates cannot be downloaded). However you do benefit from a region wide
> Image store. We've worked around the NFS problem by basically having
> multiple staging NFS stores.
>
> I have plans to make Object stores better integrated in Cloudstack. The
> holy grail would be to avoid staging NFS completely (or atleast in the
> upload path). But for now we have to live with the limitations.
>
> -Syed
>
>
> On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <ws...@cloudops.com>
> wrote:
>
>> We are using Swift as secondary storage. It also offers inter-zone
>> secondary storage. We have had to do fixes to make it work correctly, but
>> we have pushed those fixes upstream so they are available in 4.9 (and some
>> fixes earlier than that).
>>
>> Like Wido said, the staging NFS is still needed which is a real problem
>> with these implementations. You end up copying the data many times and you
>> will run into problems if you have lots of customers doing scheduled
>> snapshots at the same time because you can fill your NFS staging area
>> unless you give it a lot of space to work with.
>>
>> The object storage integrations are working, but they are not designed very
>> well at present. You have code for both S3 and Swift basically just
>> separated with conditionals and much of the logic is shared. This is very
>> brittle because making changes in one integration can (and historically
>> has) broken the other implementation. This is a relatively big problem
>> because most people working with either Swift or S3 will only have the
>> ability to test one of them, so they don't really have the ability to test
>> if they have broken the other. It does not help that there is Zero CI
>> coverage run for either the Swift or S3 integrations due to the difficulty
>> building and tearing down object storage environments on the fly.
>>
>> All in all, yes they should be working, but expect a bit of a bumpy road.
>>
>> Hope that helps,
>>
>> *Will STEVENS*
>> Lead Developer
>>
>> *CloudOps* *| *Cloud Solutions Experts
>> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
>> w cloudops.com *|* tw @CloudOps_
>>
>> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <wi...@widodh.nl>
>> wrote:
>>
>> >
>> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
>> > >
>> > >
>> > > Hello,
>> > >
>> > > Could anyone summarise the state of S3 (and clones) for secondary
>> > storage?
>> > > I read at some point that it's the only secondary storage type
>> supported
>> > inter-zone, so it would appeal from a DR pov, but not sure how mature it
>> is
>> > or if there are any serious gotchas involved.
>> > >
>> >
>> > We have it running for our largest region (Amsterdam) with Ceph's RADOS
>> > Gateway as backend. We had some issues, but they seem to be fixed after
>> our
>> > latest patches.
>> >
>> > You still need the staging NFS though, so that doesn't solve anything.
>> >
>> > Wido
>> >
>> > > Thanks
>> > > Lucian
>> > >
>> > > --
>> > > Sent from the Delta quadrant using Borg technology!
>> > >
>> > > Nux!
>> > > www.nux.ro
>> >
Re: State of the S3 secondary storage
Posted by Syed Ahmed <sa...@cloudops.com>.
As Will put it, the implementation for using Object store as secondary
storage relies on using Staging NFS. You would also loose the functionality
to do differential snapshots. All your snapshots will basically be full
volume snapshots. So, if you have a lot of snapshots being taken at
scheduled intervals, you might end up with wasting a lot of space on your
S3. Not to mention all of this has to go through the NFS which can (an in
our experience has) fill up preventing you from creating new VMs (because
templates cannot be downloaded). However you do benefit from a region wide
Image store. We've worked around the NFS problem by basically having
multiple staging NFS stores.
I have plans to make Object stores better integrated in Cloudstack. The
holy grail would be to avoid staging NFS completely (or atleast in the
upload path). But for now we have to live with the limitations.
-Syed
On Tue, Jul 19, 2016 at 11:41 AM, Will Stevens <ws...@cloudops.com>
wrote:
> We are using Swift as secondary storage. It also offers inter-zone
> secondary storage. We have had to do fixes to make it work correctly, but
> we have pushed those fixes upstream so they are available in 4.9 (and some
> fixes earlier than that).
>
> Like Wido said, the staging NFS is still needed which is a real problem
> with these implementations. You end up copying the data many times and you
> will run into problems if you have lots of customers doing scheduled
> snapshots at the same time because you can fill your NFS staging area
> unless you give it a lot of space to work with.
>
> The object storage integrations are working, but they are not designed very
> well at present. You have code for both S3 and Swift basically just
> separated with conditionals and much of the logic is shared. This is very
> brittle because making changes in one integration can (and historically
> has) broken the other implementation. This is a relatively big problem
> because most people working with either Swift or S3 will only have the
> ability to test one of them, so they don't really have the ability to test
> if they have broken the other. It does not help that there is Zero CI
> coverage run for either the Swift or S3 integrations due to the difficulty
> building and tearing down object storage environments on the fly.
>
> All in all, yes they should be working, but expect a bit of a bumpy road.
>
> Hope that helps,
>
> *Will STEVENS*
> Lead Developer
>
> *CloudOps* *| *Cloud Solutions Experts
> 420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
> w cloudops.com *|* tw @CloudOps_
>
> On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <wi...@widodh.nl>
> wrote:
>
> >
> > > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
> > >
> > >
> > > Hello,
> > >
> > > Could anyone summarise the state of S3 (and clones) for secondary
> > storage?
> > > I read at some point that it's the only secondary storage type
> supported
> > inter-zone, so it would appeal from a DR pov, but not sure how mature it
> is
> > or if there are any serious gotchas involved.
> > >
> >
> > We have it running for our largest region (Amsterdam) with Ceph's RADOS
> > Gateway as backend. We had some issues, but they seem to be fixed after
> our
> > latest patches.
> >
> > You still need the staging NFS though, so that doesn't solve anything.
> >
> > Wido
> >
> > > Thanks
> > > Lucian
> > >
> > > --
> > > Sent from the Delta quadrant using Borg technology!
> > >
> > > Nux!
> > > www.nux.ro
> >
>
Re: State of the S3 secondary storage
Posted by Will Stevens <ws...@cloudops.com>.
We are using Swift as secondary storage. It also offers inter-zone
secondary storage. We have had to do fixes to make it work correctly, but
we have pushed those fixes upstream so they are available in 4.9 (and some
fixes earlier than that).
Like Wido said, the staging NFS is still needed which is a real problem
with these implementations. You end up copying the data many times and you
will run into problems if you have lots of customers doing scheduled
snapshots at the same time because you can fill your NFS staging area
unless you give it a lot of space to work with.
The object storage integrations are working, but they are not designed very
well at present. You have code for both S3 and Swift basically just
separated with conditionals and much of the logic is shared. This is very
brittle because making changes in one integration can (and historically
has) broken the other implementation. This is a relatively big problem
because most people working with either Swift or S3 will only have the
ability to test one of them, so they don't really have the ability to test
if they have broken the other. It does not help that there is Zero CI
coverage run for either the Swift or S3 integrations due to the difficulty
building and tearing down object storage environments on the fly.
All in all, yes they should be working, but expect a bit of a bumpy road.
Hope that helps,
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>
> > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
> >
> >
> > Hello,
> >
> > Could anyone summarise the state of S3 (and clones) for secondary
> storage?
> > I read at some point that it's the only secondary storage type supported
> inter-zone, so it would appeal from a DR pov, but not sure how mature it is
> or if there are any serious gotchas involved.
> >
>
> We have it running for our largest region (Amsterdam) with Ceph's RADOS
> Gateway as backend. We had some issues, but they seem to be fixed after our
> latest patches.
>
> You still need the staging NFS though, so that doesn't solve anything.
>
> Wido
>
> > Thanks
> > Lucian
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
>
Re: State of the S3 secondary storage
Posted by Will Stevens <ws...@cloudops.com>.
We are using Swift as secondary storage. It also offers inter-zone
secondary storage. We have had to do fixes to make it work correctly, but
we have pushed those fixes upstream so they are available in 4.9 (and some
fixes earlier than that).
Like Wido said, the staging NFS is still needed which is a real problem
with these implementations. You end up copying the data many times and you
will run into problems if you have lots of customers doing scheduled
snapshots at the same time because you can fill your NFS staging area
unless you give it a lot of space to work with.
The object storage integrations are working, but they are not designed very
well at present. You have code for both S3 and Swift basically just
separated with conditionals and much of the logic is shared. This is very
brittle because making changes in one integration can (and historically
has) broken the other implementation. This is a relatively big problem
because most people working with either Swift or S3 will only have the
ability to test one of them, so they don't really have the ability to test
if they have broken the other. It does not help that there is Zero CI
coverage run for either the Swift or S3 integrations due to the difficulty
building and tearing down object storage environments on the fly.
All in all, yes they should be working, but expect a bit of a bumpy road.
Hope that helps,
*Will STEVENS*
Lead Developer
*CloudOps* *| *Cloud Solutions Experts
420 rue Guy *|* Montreal *|* Quebec *|* H3J 1S6
w cloudops.com *|* tw @CloudOps_
On Tue, Jul 19, 2016 at 11:26 AM, Wido den Hollander <wi...@widodh.nl> wrote:
>
> > Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
> >
> >
> > Hello,
> >
> > Could anyone summarise the state of S3 (and clones) for secondary
> storage?
> > I read at some point that it's the only secondary storage type supported
> inter-zone, so it would appeal from a DR pov, but not sure how mature it is
> or if there are any serious gotchas involved.
> >
>
> We have it running for our largest region (Amsterdam) with Ceph's RADOS
> Gateway as backend. We had some issues, but they seem to be fixed after our
> latest patches.
>
> You still need the staging NFS though, so that doesn't solve anything.
>
> Wido
>
> > Thanks
> > Lucian
> >
> > --
> > Sent from the Delta quadrant using Borg technology!
> >
> > Nux!
> > www.nux.ro
>
Re: State of the S3 secondary storage
Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
>
>
> Hello,
>
> Could anyone summarise the state of S3 (and clones) for secondary storage?
> I read at some point that it's the only secondary storage type supported inter-zone, so it would appeal from a DR pov, but not sure how mature it is or if there are any serious gotchas involved.
>
We have it running for our largest region (Amsterdam) with Ceph's RADOS Gateway as backend. We had some issues, but they seem to be fixed after our latest patches.
You still need the staging NFS though, so that doesn't solve anything.
Wido
> Thanks
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro
Re: State of the S3 secondary storage
Posted by Wido den Hollander <wi...@widodh.nl>.
> Op 19 juli 2016 om 14:38 schreef Nux! <nu...@li.nux.ro>:
>
>
> Hello,
>
> Could anyone summarise the state of S3 (and clones) for secondary storage?
> I read at some point that it's the only secondary storage type supported inter-zone, so it would appeal from a DR pov, but not sure how mature it is or if there are any serious gotchas involved.
>
We have it running for our largest region (Amsterdam) with Ceph's RADOS Gateway as backend. We had some issues, but they seem to be fixed after our latest patches.
You still need the staging NFS though, so that doesn't solve anything.
Wido
> Thanks
> Lucian
>
> --
> Sent from the Delta quadrant using Borg technology!
>
> Nux!
> www.nux.ro