You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Christian Lorenz <Ch...@webtrekk.com> on 2018/08/14 13:10:42 UTC
JBOD disk failure
Hi,
given a cluster with RF=3 and CL=LOCAL_ONE and application is deleting data, what happens if the nodes are setup with JBOD and one disk fails? Do I get consistent results while the broken drive is replaced and a nodetool repair is running on the node with the replaced drive?
Kind regards,
Christian
Re: JBOD disk failure
Posted by daemeon reiydelle <da...@gmail.com>.
you have to explain what you mean by "JBOD". All in one large vdisk?
Separate drives?
At the end of the day, if a device fails in a way that the data housed on
that device (or array) is no longer available, that HDFS storage is marked
down. HDFS now needs to create a 3rd replicant. Various timers control how
long HDFS waits to see if the device comes back on line. But assume
immediately for convenience. Remember that a write is to a (random) copy of
the data, and that datanode then replicates to the next node, and so forth.
The in-process-of-being-created 3rd copy will also get those delete
"updates". Have you read up on how "deleting" a record works?
<======>
Be the reason someone smiles today.
Or the reason they need a drink.
Whichever works.
*Daemeon C.M. Reiydelle*
*email: daemeonr@gmail.com <da...@gmail.com>*
*San Francisco 1.415.501.0198/London 44 020 8144 9872/Skype
daemeon.c.m.reiydelle*
On Tue, Aug 14, 2018 at 6:10 AM Christian Lorenz <
Christian.Lorenz@webtrekk.com> wrote:
> Hi,
>
>
>
> given a cluster with RF=3 and CL=LOCAL_ONE and application is deleting
> data, what happens if the nodes are setup with JBOD and one disk fails? Do
> I get consistent results while the broken drive is replaced and a nodetool
> repair is running on the node with the replaced drive?
>
>
>
> Kind regards,
>
> Christian
>
Re: JBOD disk failure - just say no
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
We recently helped a team deal with some JBOD issues, they can be quite
painful, and the experience depends a bit on the C* version in use. We
wrote a blog post about it (published today):
http://thelastpickle.com/blog/2018/08/22/the-fine-print-when-using-multiple-data-directories.html
Hope this helps.
Jon
On Mon, Aug 20, 2018 at 5:49 PM James Briggs <ja...@yahoo.com.invalid>
wrote:
> Cassandra JBOD has a bunch of issues, so I don't recommend it for
> production:
>
> 1) disks fill up with load (data) unevenly, meaning you can run out on a
> disk while some are half-full
> 2) one bad disk can take out the whole node
> 3) instead of a small failure probability on an LVM/RAID volume, with JBOD
> you end up near 100% chance of failure after 3 years or so.
> 4) generally you will not have enough warning of a looming failure with
> JBOD compared to LVM/RAID. (Some
> companies take a week or two to replace a failed disk.)
>
> JBOD is easy to setup, but hard to manage.
>
> Thanks, James.
>
>
>
> ------------------------------
> *From:* kurt greaves <ku...@instaclustr.com>
> *To:* User <us...@cassandra.apache.org>
> *Sent:* Friday, August 17, 2018 5:42 AM
> *Subject:* Re: JBOD disk failure
>
> As far as I'm aware, yes. I recall hearing someone mention tying system
> tables to a particular disk but at the moment that doesn't exist.
>
> On Fri., 17 Aug. 2018, 01:04 Eric Evans, <jo...@gmail.com>
> wrote:
>
> On Wed, Aug 15, 2018 at 3:23 AM kurt greaves <ku...@instaclustr.com> wrote:
> > Yep. It might require a full node replace depending on what data is lost
> from the system tables. In some cases you might be able to recover from
> partially lost system info, but it's not a sure thing.
>
> Ugh, does it really just boil down to what part of `system` happens to
> be on the disk in question? In my mind, that makes the only sane
> operational procedure for a failed disk to be: "replace the entire
> node". IOW, I don't think we can realistically claim you can survive
> a failed a JBOD device if it relies on happenstance.
>
> > On Wed., 15 Aug. 2018, 17:55 Christian Lorenz, <
> Christian.Lorenz@webtrekk.com > wrote:
> >>
> >> Thank you for the answers. We are using the current version 3.11.3 So
> this one includes CASSANDRA-6696.
> >>
> >> So if I get this right, losing system tables will need a full node
> rebuild. Otherwise repair will get the node consistent again.
> >
> > [ ... ]
>
> --
> Eric Evans
> john.eric.evans@gmail.com
>
> ------------------------------ ------------------------------ ---------
> To unsubscribe, e-mail: user-unsubscribe@cassandra. apache.org
> <us...@cassandra.apache.org>
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
>
>
--
Jon Haddad
http://www.rustyrazorblade.com
twitter: rustyrazorblade
Re: JBOD disk failure - just say no
Posted by James Briggs <ja...@yahoo.com.INVALID>.
Cassandra JBOD has a bunch of issues, so I don't recommend it for production:
1) disks fill up with load (data) unevenly, meaning you can run out on a disk while some are half-full2) one bad disk can take out the whole node3) instead of a small failure probability on an LVM/RAID volume, with JBOD you end up near 100% chance of failure after 3 years or so.4) generally you will not have enough warning of a looming failure with JBOD compared to LVM/RAID. (Somecompanies take a week or two to replace a failed disk.)
JBOD is easy to setup, but hard to manage. Thanks, James.
From: kurt greaves <ku...@instaclustr.com>
To: User <us...@cassandra.apache.org>
Sent: Friday, August 17, 2018 5:42 AM
Subject: Re: JBOD disk failure
As far as I'm aware, yes. I recall hearing someone mention tying system tables to a particular disk but at the moment that doesn't exist.
On Fri., 17 Aug. 2018, 01:04 Eric Evans, <jo...@gmail.com> wrote:
On Wed, Aug 15, 2018 at 3:23 AM kurt greaves <ku...@instaclustr.com> wrote:
> Yep. It might require a full node replace depending on what data is lost from the system tables. In some cases you might be able to recover from partially lost system info, but it's not a sure thing.
Ugh, does it really just boil down to what part of `system` happens to
be on the disk in question? In my mind, that makes the only sane
operational procedure for a failed disk to be: "replace the entire
node". IOW, I don't think we can realistically claim you can survive
a failed a JBOD device if it relies on happenstance.
> On Wed., 15 Aug. 2018, 17:55 Christian Lorenz, <Christian.Lorenz@webtrekk.com > wrote:
>>
>> Thank you for the answers. We are using the current version 3.11.3 So this one includes CASSANDRA-6696.
>>
>> So if I get this right, losing system tables will need a full node rebuild. Otherwise repair will get the node consistent again.
>
> [ ... ]
--
Eric Evans
john.eric.evans@gmail.com
------------------------------ ------------------------------ ---------
To unsubscribe, e-mail: user-unsubscribe@cassandra. apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: JBOD disk failure
Posted by kurt greaves <ku...@instaclustr.com>.
As far as I'm aware, yes. I recall hearing someone mention tying system
tables to a particular disk but at the moment that doesn't exist.
On Fri., 17 Aug. 2018, 01:04 Eric Evans, <jo...@gmail.com> wrote:
> On Wed, Aug 15, 2018 at 3:23 AM kurt greaves <ku...@instaclustr.com> wrote:
> > Yep. It might require a full node replace depending on what data is lost
> from the system tables. In some cases you might be able to recover from
> partially lost system info, but it's not a sure thing.
>
> Ugh, does it really just boil down to what part of `system` happens to
> be on the disk in question? In my mind, that makes the only sane
> operational procedure for a failed disk to be: "replace the entire
> node". IOW, I don't think we can realistically claim you can survive
> a failed a JBOD device if it relies on happenstance.
>
> > On Wed., 15 Aug. 2018, 17:55 Christian Lorenz, <
> Christian.Lorenz@webtrekk.com> wrote:
> >>
> >> Thank you for the answers. We are using the current version 3.11.3 So
> this one includes CASSANDRA-6696.
> >>
> >> So if I get this right, losing system tables will need a full node
> rebuild. Otherwise repair will get the node consistent again.
> >
> > [ ... ]
>
> --
> Eric Evans
> john.eric.evans@gmail.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
Re: JBOD disk failure
Posted by Eric Evans <jo...@gmail.com>.
On Wed, Aug 15, 2018 at 3:23 AM kurt greaves <ku...@instaclustr.com> wrote:
> Yep. It might require a full node replace depending on what data is lost from the system tables. In some cases you might be able to recover from partially lost system info, but it's not a sure thing.
Ugh, does it really just boil down to what part of `system` happens to
be on the disk in question? In my mind, that makes the only sane
operational procedure for a failed disk to be: "replace the entire
node". IOW, I don't think we can realistically claim you can survive
a failed a JBOD device if it relies on happenstance.
> On Wed., 15 Aug. 2018, 17:55 Christian Lorenz, <Ch...@webtrekk.com> wrote:
>>
>> Thank you for the answers. We are using the current version 3.11.3 So this one includes CASSANDRA-6696.
>>
>> So if I get this right, losing system tables will need a full node rebuild. Otherwise repair will get the node consistent again.
>
> [ ... ]
--
Eric Evans
john.eric.evans@gmail.com
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: JBOD disk failure
Posted by kurt greaves <ku...@instaclustr.com>.
Yep. It might require a full node replace depending on what data is lost
from the system tables. In some cases you might be able to recover from
partially lost system info, but it's not a sure thing.
On Wed., 15 Aug. 2018, 17:55 Christian Lorenz, <
Christian.Lorenz@webtrekk.com> wrote:
> Thank you for the answers. We are using the current version 3.11.3 So this
> one includes CASSANDRA-6696.
>
> So if I get this right, losing system tables will need a full node
> rebuild. Otherwise repair will get the node consistent again.
>
>
>
> Regards,
>
> Christian
>
>
>
>
>
> *Von: *kurt greaves <ku...@instaclustr.com>
> *Antworten an: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Datum: *Mittwoch, 15. August 2018 um 04:53
> *An: *User <us...@cassandra.apache.org>
> *Betreff: *Re: JBOD disk failure
>
>
>
> If that disk had important data in the system tables however you might
> have some trouble and need to replace the entire instance anyway.
>
>
>
> On 15 August 2018 at 12:20, Jeff Jirsa <jj...@gmail.com> wrote:
>
> Depends on version
>
>
>
> For versions without the fix from Cassandra-6696, the only safe option on
> single disk failure is to stop and replace the whole instance - this is
> important because in older versions of Cassandra, you could have data in
> one sstable, a tombstone shadowing it in another disk, and it could be very
> far behind gc_grace_seconds. On disk failure in this scenario, if the disk
> holding the tombstone is lost, repair will propagate the
> (deleted/resurrected) data to the other replicas, which probably isn’t what
> you want to happen.
>
>
>
> With 6696, you should be safe to replace the disk and run repair - 6696
> will keep data for a given token range all on the same disks, so the
> resurrection problem is solved.
>
>
>
>
>
> --
>
> Jeff Jirsa
>
>
>
>
> On Aug 14, 2018, at 6:10 AM, Christian Lorenz <
> Christian.Lorenz@webtrekk.com> wrote:
>
> Hi,
>
>
>
> given a cluster with RF=3 and CL=LOCAL_ONE and application is deleting
> data, what happens if the nodes are setup with JBOD and one disk fails? Do
> I get consistent results while the broken drive is replaced and a nodetool
> repair is running on the node with the replaced drive?
>
>
>
> Kind regards,
>
> Christian
>
>
>
Re: JBOD disk failure
Posted by Christian Lorenz <Ch...@webtrekk.com>.
Thank you for the answers. We are using the current version 3.11.3 So this one includes CASSANDRA-6696.
So if I get this right, losing system tables will need a full node rebuild. Otherwise repair will get the node consistent again.
Regards,
Christian
Von: kurt greaves <ku...@instaclustr.com>
Antworten an: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Datum: Mittwoch, 15. August 2018 um 04:53
An: User <us...@cassandra.apache.org>
Betreff: Re: JBOD disk failure
If that disk had important data in the system tables however you might have some trouble and need to replace the entire instance anyway.
On 15 August 2018 at 12:20, Jeff Jirsa <jj...@gmail.com>> wrote:
Depends on version
For versions without the fix from Cassandra-6696, the only safe option on single disk failure is to stop and replace the whole instance - this is important because in older versions of Cassandra, you could have data in one sstable, a tombstone shadowing it in another disk, and it could be very far behind gc_grace_seconds. On disk failure in this scenario, if the disk holding the tombstone is lost, repair will propagate the (deleted/resurrected) data to the other replicas, which probably isn’t what you want to happen.
With 6696, you should be safe to replace the disk and run repair - 6696 will keep data for a given token range all on the same disks, so the resurrection problem is solved.
--
Jeff Jirsa
On Aug 14, 2018, at 6:10 AM, Christian Lorenz <Ch...@webtrekk.com>> wrote:
Hi,
given a cluster with RF=3 and CL=LOCAL_ONE and application is deleting data, what happens if the nodes are setup with JBOD and one disk fails? Do I get consistent results while the broken drive is replaced and a nodetool repair is running on the node with the replaced drive?
Kind regards,
Christian
Re: JBOD disk failure
Posted by kurt greaves <ku...@instaclustr.com>.
If that disk had important data in the system tables however you might have
some trouble and need to replace the entire instance anyway.
On 15 August 2018 at 12:20, Jeff Jirsa <jj...@gmail.com> wrote:
> Depends on version
>
> For versions without the fix from Cassandra-6696, the only safe option on
> single disk failure is to stop and replace the whole instance - this is
> important because in older versions of Cassandra, you could have data in
> one sstable, a tombstone shadowing it in another disk, and it could be very
> far behind gc_grace_seconds. On disk failure in this scenario, if the disk
> holding the tombstone is lost, repair will propagate the
> (deleted/resurrected) data to the other replicas, which probably isn’t what
> you want to happen.
>
> With 6696, you should be safe to replace the disk and run repair - 6696
> will keep data for a given token range all on the same disks, so the
> resurrection problem is solved.
>
>
> --
> Jeff Jirsa
>
>
> On Aug 14, 2018, at 6:10 AM, Christian Lorenz <
> Christian.Lorenz@webtrekk.com> wrote:
>
> Hi,
>
>
>
> given a cluster with RF=3 and CL=LOCAL_ONE and application is deleting
> data, what happens if the nodes are setup with JBOD and one disk fails? Do
> I get consistent results while the broken drive is replaced and a nodetool
> repair is running on the node with the replaced drive?
>
>
>
> Kind regards,
>
> Christian
>
>
Re: JBOD disk failure
Posted by Jeff Jirsa <jj...@gmail.com>.
Depends on version
For versions without the fix from Cassandra-6696, the only safe option on single disk failure is to stop and replace the whole instance - this is important because in older versions of Cassandra, you could have data in one sstable, a tombstone shadowing it in another disk, and it could be very far behind gc_grace_seconds. On disk failure in this scenario, if the disk holding the tombstone is lost, repair will propagate the (deleted/resurrected) data to the other replicas, which probably isn’t what you want to happen.
With 6696, you should be safe to replace the disk and run repair - 6696 will keep data for a given token range all on the same disks, so the resurrection problem is solved.
--
Jeff Jirsa
> On Aug 14, 2018, at 6:10 AM, Christian Lorenz <Ch...@webtrekk.com> wrote:
>
> Hi,
>
> given a cluster with RF=3 and CL=LOCAL_ONE and application is deleting data, what happens if the nodes are setup with JBOD and one disk fails? Do I get consistent results while the broken drive is replaced and a nodetool repair is running on the node with the replaced drive?
>
> Kind regards,
> Christian