You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@cloudstack.apache.org by Ivan Kudryavtsev <ku...@bw-sw.com> on 2017/08/31 05:33:37 UTC
A solution for snapshots stuck in Allocated/BackingUp states
Hello, community.
Today we (again) met snapshots stuck in exotic states and I decided to
create small guide how to fix it. AFAIK, somebody tried to manage KB in
GitHub about useful tools, cases, etc. It might be that small article can
be included. I wrote it because short googling didn't give positive results
about the problem. Also, all comments are welcome.
https://bitworks.software/blog/en/2017-08-31-cloudstack-removing-snapshots-stuck-in-allocated-backingup-states.html
Also, the same for Russian-speaking people:
https://bitworks.software/blog/2017-08-31-cloudstack-removing-snapshots-stuck-in-allocated-backingup-states.html
--
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>
Re: A solution for snapshots stuck in Allocated/BackingUp states
Posted by Andrija Panic <an...@gmail.com>.
May I suggest another permanent fix (sounds like a joke, but I'm actually
serious) - move away from Primary Storage whose behavior is to copy
snapshoted data to Secondary Storage - to the Primary Storage that doesn't
copy content to Secondary Storage -i.e. SolidFire :)
Imagine having 500GB disk customer, who configured hourly snapshots with
CEPH/NFS.... this is the reason we at first disabled hourly snapshots in
GUI :)
We also sometimes hit this problem, all CEPH and NFS volumes (ACS 4.5, ACS
4.8) and this is due to sometimes also concurrent snaps on same volume or
VM, this is more of ACS logic problem, not underlying infra problem - i.e.
volume owner configured both daily and weekly/monthly snapshot to start at
identical time (same HH:MM), so one of them is stuck in Allocated state etc
(and BTW this was on SolidFire, where snapshots are instant, stays on
SolidFire, and takes i.e. 3 seconds to make proper snap to BackedUp state -
even here we had issues, agani pure ACS/user logic root cause)
Best
Andrija
On 31 August 2017 at 10:15, Ivan Kudryavtsev <ku...@bw-sw.com>
wrote:
> Hi, Dag.
>
> You're really right about that, but I mean that If the cloud is running and
> you don't have developer capacity to investigate, the long term solution is
> that works for you right now. It's really workaround and I don't pretend
> arguing with that.
>
> 2017-08-31 15:08 GMT+07:00 Dag Sonstebo <Da...@shapeblue.com>:
>
> > Hi Ivan,
> >
> > Thanks for the guide, good workaround for someone hitting the same issue.
> > I must respectfully disagree with your sentiment though. As you have
> > mentioned in your article issues like this tend to have a root cause –
> > network outage, hypervisor issues etc. – and these issues vary from
> > environment to environment and should be investigated on a case by case
> > basis. Your suggested long term solution is just a workaround – error
> > trapping and subsequent housekeeping should be improved in the CloudStack
> > code, so the best thing to do is investigate, find root cause and log a
> > Jira ticket such that it can be improved going forward.
> >
> > Regards,
> > Dag Sonstebo
> > Cloud Architect
> > ShapeBlue
> >
> > On 31/08/2017, 06:33, "Ivan Kudryavtsev" <ku...@bw-sw.com>
> wrote:
> >
> > Hello, community.
> >
> > Today we (again) met snapshots stuck in exotic states and I decided
> to
> > create small guide how to fix it. AFAIK, somebody tried to manage KB
> in
> > GitHub about useful tools, cases, etc. It might be that small article
> > can
> > be included. I wrote it because short googling didn't give positive
> > results
> > about the problem. Also, all comments are welcome.
> >
> > https://bitworks.software/blog/en/2017-08-31-cloudstack-
> > removing-snapshots-stuck-in-allocated-backingup-states.html
> >
> > Also, the same for Russian-speaking people:
> >
> > https://bitworks.software/blog/2017-08-31-cloudstack-
> > removing-snapshots-stuck-in-allocated-backingup-states.html
> >
> > --
> > With best regards, Ivan Kudryavtsev
> > Bitworks Software, Ltd.
> > Cell: +7-923-414-1515
> > WWW: http://bitworks.software/ <http://bw-sw.com/>
> >
> >
> >
> > Dag.Sonstebo@shapeblue.com
> > www.shapeblue.com
> > 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> > @shapeblue
> >
> >
> >
> >
>
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>
--
Andrija Panić
Re: A solution for snapshots stuck in Allocated/BackingUp states
Posted by Ivan Kudryavtsev <ku...@bw-sw.com>.
Hi, Dag.
You're really right about that, but I mean that If the cloud is running and
you don't have developer capacity to investigate, the long term solution is
that works for you right now. It's really workaround and I don't pretend
arguing with that.
2017-08-31 15:08 GMT+07:00 Dag Sonstebo <Da...@shapeblue.com>:
> Hi Ivan,
>
> Thanks for the guide, good workaround for someone hitting the same issue.
> I must respectfully disagree with your sentiment though. As you have
> mentioned in your article issues like this tend to have a root cause –
> network outage, hypervisor issues etc. – and these issues vary from
> environment to environment and should be investigated on a case by case
> basis. Your suggested long term solution is just a workaround – error
> trapping and subsequent housekeeping should be improved in the CloudStack
> code, so the best thing to do is investigate, find root cause and log a
> Jira ticket such that it can be improved going forward.
>
> Regards,
> Dag Sonstebo
> Cloud Architect
> ShapeBlue
>
> On 31/08/2017, 06:33, "Ivan Kudryavtsev" <ku...@bw-sw.com> wrote:
>
> Hello, community.
>
> Today we (again) met snapshots stuck in exotic states and I decided to
> create small guide how to fix it. AFAIK, somebody tried to manage KB in
> GitHub about useful tools, cases, etc. It might be that small article
> can
> be included. I wrote it because short googling didn't give positive
> results
> about the problem. Also, all comments are welcome.
>
> https://bitworks.software/blog/en/2017-08-31-cloudstack-
> removing-snapshots-stuck-in-allocated-backingup-states.html
>
> Also, the same for Russian-speaking people:
>
> https://bitworks.software/blog/2017-08-31-cloudstack-
> removing-snapshots-stuck-in-allocated-backingup-states.html
>
> --
> With best regards, Ivan Kudryavtsev
> Bitworks Software, Ltd.
> Cell: +7-923-414-1515
> WWW: http://bitworks.software/ <http://bw-sw.com/>
>
>
>
> Dag.Sonstebo@shapeblue.com
> www.shapeblue.com
> 53 Chandos Place, Covent Garden, London WC2N 4HSUK
> @shapeblue
>
>
>
>
--
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>
Re: A solution for snapshots stuck in Allocated/BackingUp states
Posted by Dag Sonstebo <Da...@shapeblue.com>.
Hi Ivan,
Thanks for the guide, good workaround for someone hitting the same issue. I must respectfully disagree with your sentiment though. As you have mentioned in your article issues like this tend to have a root cause –
network outage, hypervisor issues etc. – and these issues vary from environment to environment and should be investigated on a case by case basis. Your suggested long term solution is just a workaround – error trapping and subsequent housekeeping should be improved in the CloudStack code, so the best thing to do is investigate, find root cause and log a Jira ticket such that it can be improved going forward.
Regards,
Dag Sonstebo
Cloud Architect
ShapeBlue
On 31/08/2017, 06:33, "Ivan Kudryavtsev" <ku...@bw-sw.com> wrote:
Hello, community.
Today we (again) met snapshots stuck in exotic states and I decided to
create small guide how to fix it. AFAIK, somebody tried to manage KB in
GitHub about useful tools, cases, etc. It might be that small article can
be included. I wrote it because short googling didn't give positive results
about the problem. Also, all comments are welcome.
https://bitworks.software/blog/en/2017-08-31-cloudstack-removing-snapshots-stuck-in-allocated-backingup-states.html
Also, the same for Russian-speaking people:
https://bitworks.software/blog/2017-08-31-cloudstack-removing-snapshots-stuck-in-allocated-backingup-states.html
--
With best regards, Ivan Kudryavtsev
Bitworks Software, Ltd.
Cell: +7-923-414-1515
WWW: http://bitworks.software/ <http://bw-sw.com/>
Dag.Sonstebo@shapeblue.com
www.shapeblue.com
53 Chandos Place, Covent Garden, London WC2N 4HSUK
@shapeblue