You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by lars hofhansl <la...@apache.org> on 2013/03/14 02:12:41 UTC
Replication hosed after simple cluster restart
We just ran into an interesting scenario. We restarted a cluster that was setup as a replication source.
The stop went cleanly.
Upon restart *all* regionservers aborted within a few seconds with variations of these errors:
http://pastebin.com/3iQVuBqS
This is scary!
-- Lars
Re: Replication hosed after simple cluster restart
Posted by Himanshu Vashishtha <hv...@cs.ualberta.ca>.
Err.. The jira is https://issues.apache.org/jira/browse/HBASE-7122
On Wed, Mar 13, 2013 at 6:51 PM, Himanshu Vashishtha
<hv...@cs.ualberta.ca> wrote:
> The log message you are seeing have been there for a long time I
> remember (it is a debug level message).
> I had a patch long back
> https://issues.apache.org/jira/browse/HBASE-7937, which became stale.
>
> Stack: it is not the fault of the multi command, it is the way the
> code is using it is wrong. There is a race b/w reading and moving the
> znodes. Basically, what should be done is in case a regionserver fails
> to move the znodes, it should return an empty list, and NOT what it
> has read earlier. This is because other regionsever might have moved
> the znodes.
>
> On Wed, Mar 13, 2013 at 6:45 PM, lars hofhansl <la...@apache.org> wrote:
>> Hey no problem. It's cool that we found it in a test env. It's probably
>> quite hard to reproduce.
>> This is in 0.94.5 but this feature is off by default.
>>
>> What's the general thought here, should I kill the current 0.94.6 rc for
>> this?
>> My gut says: Yes.
>>
>>
>> I'm also a bit worried about these:
>> 2013-03-14 01:42:42,271 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
>> log for replication
>> shared-dnds1-12-sfm.ops.sfdc.net%2C60020%2C1363220608780.1363220609572 at 0
>> 2013-03-14 01:42:42,358 WARN
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got:
>> java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
>> 2013-03-14 01:42:42,358 WARN
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Waited
>> too long for this file, considering dumping
>> 2013-03-14 01:42:42,358 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable
>> to open a reader, sleeping 1000 times 10
>>
>> This happens after bouncing the cluster a 2nd time and these messages repeat
>> every 10s (for hours now). This is a separate problem I think.
>>
>> -- Lars
>>
>> ________________________________
>> From: Himanshu Vashishtha <hv...@cs.ualberta.ca>
>>
>> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>> Cc: Ted Yu <yu...@gmail.com>
>> Sent: Wednesday, March 13, 2013 6:38 PM
>>
>> Subject: Re: Replication hosed after simple cluster restart
>>
>> This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
>> might not be able to move later on, resulting in bogus znodes.
>> I'll fix this asap. Weird it didn't happen in my testing earlier.
>> Sorry about this.
>>
>> On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <la...@apache.org> wrote:
>>> Sorry 0.94.6RC1
>>> (I complain about folks not reporting the version all the time, and then I
>>> do it too)
>>>
>>>
>>>
>>> ________________________________
>>> From: Ted Yu <yu...@gmail.com>
>>> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>>> Sent: Wednesday, March 13, 2013 6:17 PM
>>> Subject: Re: Replication hosed after simple cluster restart
>>>
>>>
>>> Did this happen on 0.94.5 ?
>>>
>>> Thanks
>>>
>>>
>>> On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <la...@apache.org> wrote:
>>>
>>> We just ran into an interesting scenario. We restarted a cluster that was
>>> setup as a replication source.
>>>>The stop went cleanly.
>>>>
>>>>Upon restart *all* regionservers aborted within a few seconds with
>>>> variations of these errors:
>>>>http://pastebin.com/3iQVuBqS
>>>>
>>>>This is scary!
>>>>
>>>>-- Lars
>>
>>
Re: Replication hosed after simple cluster restart
Posted by Himanshu Vashishtha <hv...@cs.ualberta.ca>.
The log message you are seeing have been there for a long time I
remember (it is a debug level message).
I had a patch long back
https://issues.apache.org/jira/browse/HBASE-7937, which became stale.
Stack: it is not the fault of the multi command, it is the way the
code is using it is wrong. There is a race b/w reading and moving the
znodes. Basically, what should be done is in case a regionserver fails
to move the znodes, it should return an empty list, and NOT what it
has read earlier. This is because other regionsever might have moved
the znodes.
On Wed, Mar 13, 2013 at 6:45 PM, lars hofhansl <la...@apache.org> wrote:
> Hey no problem. It's cool that we found it in a test env. It's probably
> quite hard to reproduce.
> This is in 0.94.5 but this feature is off by default.
>
> What's the general thought here, should I kill the current 0.94.6 rc for
> this?
> My gut says: Yes.
>
>
> I'm also a bit worried about these:
> 2013-03-14 01:42:42,271 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
> log for replication
> shared-dnds1-12-sfm.ops.sfdc.net%2C60020%2C1363220608780.1363220609572 at 0
> 2013-03-14 01:42:42,358 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
> at
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
> 2013-03-14 01:42:42,358 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Waited
> too long for this file, considering dumping
> 2013-03-14 01:42:42,358 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable
> to open a reader, sleeping 1000 times 10
>
> This happens after bouncing the cluster a 2nd time and these messages repeat
> every 10s (for hours now). This is a separate problem I think.
>
> -- Lars
>
> ________________________________
> From: Himanshu Vashishtha <hv...@cs.ualberta.ca>
>
> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> Cc: Ted Yu <yu...@gmail.com>
> Sent: Wednesday, March 13, 2013 6:38 PM
>
> Subject: Re: Replication hosed after simple cluster restart
>
> This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
> might not be able to move later on, resulting in bogus znodes.
> I'll fix this asap. Weird it didn't happen in my testing earlier.
> Sorry about this.
>
> On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <la...@apache.org> wrote:
>> Sorry 0.94.6RC1
>> (I complain about folks not reporting the version all the time, and then I
>> do it too)
>>
>>
>>
>> ________________________________
>> From: Ted Yu <yu...@gmail.com>
>> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>> Sent: Wednesday, March 13, 2013 6:17 PM
>> Subject: Re: Replication hosed after simple cluster restart
>>
>>
>> Did this happen on 0.94.5 ?
>>
>> Thanks
>>
>>
>> On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <la...@apache.org> wrote:
>>
>> We just ran into an interesting scenario. We restarted a cluster that was
>> setup as a replication source.
>>>The stop went cleanly.
>>>
>>>Upon restart *all* regionservers aborted within a few seconds with
>>> variations of these errors:
>>>http://pastebin.com/3iQVuBqS
>>>
>>>This is scary!
>>>
>>>-- Lars
>
>
Re: Replication hosed after simple cluster restart
Posted by Himanshu Vashishtha <hv...@cs.ualberta.ca>.
I'll +1 in favor of sinking it for the reason as if someone wants to
use and run into this issue, it will stop the regionserver.
Lars: in case you need any help for the rc2 candidate, please pull me
in. I'd love to do so.
Thanks,
Himanshu
On Wed, Mar 13, 2013 at 8:06 PM, Ted Yu <yu...@gmail.com> wrote:
> This was the JIRA that introduced copyQueuesFromRSUsingMulti():
> HBASE-2611 Handle RS that fails while processing the failure of another one
> (Himanshu Vashishtha)
>
> It went into 0.94.5
> And the feature is off by default:
>
> <name>hbase.zookeeper.useMulti</name>
> <value>false</value>
>
> The fact that Lars first reported the following problem meant that no other
> user tried this feature.
>
> Hence I think 0.94.6 RC1 doesn't need to be sunk.
>
> Cheers
>
> On Wed, Mar 13, 2013 at 6:45 PM, lars hofhansl <la...@apache.org> wrote:
>>
>> Hey no problem. It's cool that we found it in a test env. It's probably
>> quite hard to reproduce.
>> This is in 0.94.5 but this feature is off by default.
>>
>> What's the general thought here, should I kill the current 0.94.6 rc for
>> this?
>> My gut says: Yes.
>>
>>
>> I'm also a bit worried about these:
>> 2013-03-14 01:42:42,271 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
>> log for replication
>> shared-dnds1-12-sfm.ops.sfdc.net%2C60020%2C1363220608780.1363220609572 at 0
>> 2013-03-14 01:42:42,358 WARN
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got:
>> java.io.EOFException
>> at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
>> at
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
>> 2013-03-14 01:42:42,358 WARN
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Waited
>> too long for this file, considering dumping
>> 2013-03-14 01:42:42,358 DEBUG
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable
>> to open a reader, sleeping 1000 times 10
>>
>> This happens after bouncing the cluster a 2nd time and these messages
>> repeat every 10s (for hours now). This is a separate problem I think.
>>
>> -- Lars
>>
>> ________________________________
>> From: Himanshu Vashishtha <hv...@cs.ualberta.ca>
>>
>> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>> Cc: Ted Yu <yu...@gmail.com>
>> Sent: Wednesday, March 13, 2013 6:38 PM
>>
>> Subject: Re: Replication hosed after simple cluster restart
>>
>> This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
>> might not be able to move later on, resulting in bogus znodes.
>> I'll fix this asap. Weird it didn't happen in my testing earlier.
>> Sorry about this.
>>
>>
>> On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <la...@apache.org> wrote:
>> > Sorry 0.94.6RC1
>> > (I complain about folks not reporting the version all the time, and then
>> > I do it too)
>> >
>> >
>> >
>> > ________________________________
>> > From: Ted Yu <yu...@gmail.com>
>> > To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
>> > Sent: Wednesday, March 13, 2013 6:17 PM
>> > Subject: Re: Replication hosed after simple cluster restart
>> >
>> >
>> > Did this happen on 0.94.5 ?
>> >
>> > Thanks
>> >
>> >
>> > On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <la...@apache.org> wrote:
>> >
>> > We just ran into an interesting scenario. We restarted a cluster that
>> > was setup as a replication source.
>> >>The stop went cleanly.
>> >>
>> >>Upon restart *all* regionservers aborted within a few seconds with
>> >> variations of these errors:
>> >>http://pastebin.com/3iQVuBqS
>> >>
>> >>This is scary!
>> >>
>> >>-- Lars
>>
>>
>
Re: Replication hosed after simple cluster restart
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I agree that we should try to add a test for this. A chance Lars you
found it, but will be even better if the test suite can detect such
things.
Does any-one have opened the JIRA for the test part?
JM
2013/3/14 lars hofhansl <la...@apache.org>:
> I have proposed some minor changes (including adding the jitter) on HBASE-8099.
> Turns out there already is a wait-time to give the cluster a chance to shutdown. It defaults to 2s, which was not enough in our case.
>
> Let's do a test (if we think that can be done) in a different jira.
>
>
> -- Lars
> ________________________________
> From: Himanshu Vashishtha <hv...@cs.ualberta.ca>
> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Wednesday, March 13, 2013 8:59 PM
> Subject: Re: Replication hosed after simple cluster restart
>
> On Wed, Mar 13, 2013 at 8:48 PM, lars hofhansl <la...@apache.org> wrote:
>> Yeah, lemme sink the RC... We do have a fix.
>>
>>
>> Consider it sunk.
>>
>> In the end there are some more issues to discuss anyway.
>> - Can we avoid RSs taking over queues during a clean shutdown/restart? Without multi we can actually loose data to replicate this way (one RS is shut down, another takes over and is itself shut down) - unless I misunderstand.
>
> I agree. because even if they do move, they are not using locality as
> the regionserver which eventually takes it over will remotely read the
> log files. One way I can think of is do a scan on the available
> regionservers in the /hbase/rs znodes and then decide whether it
> should start the failover processing.
>
>>
>> - Should we stagger the attempts to move the queues for example with a random wait between 0 and 10s, so that not all RSs try at the same time?
>> - A test for this scenario? (That's probably tricky)
>
> How about adding a jitter (random sleep (0-10 sec]) in the run method
> of the NodeFailoverWorker before it actually starts the failover
> processing? I will try to come up with a test case.
>
>>
>>
>> -- Lars
>>
>>
>>
>> ________________________________
>> From: Andrew Purtell <ap...@apache.org>
>> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
>> Sent: Wednesday, March 13, 2013 8:22 PM
>> Subject: Re: Replication hosed after simple cluster restart
>>
>> If Himanshu (?) can fix it quickly we should try to get it in here IMHO.
>>
>> On Wednesday, March 13, 2013, Ted Yu wrote:
>>
>>> This was the JIRA that introduced copyQueuesFromRSUsingMulti():
>>> HBASE-2611 Handle RS that fails while processing the failure of another one
>>> (Himanshu Vashishtha)
>>>
>>> It went into 0.94.5
>>> And the feature is off by default:
>>>
>>> <name>hbase.zookeeper.useMulti</name>
>>> <value>false</value>
>>>
>>> The fact that Lars first reported the following problem meant that no other
>>> user tried this feature.
>>>
>>> Hence I think 0.94.6 RC1 doesn't need to be sunk.
>>>
>>> Cheers
>>>
>>> On Wed, Mar 13, 2013 at 6:45 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>>> wrote:
>>>
>>> > Hey no problem. It's cool that we found it in a test env. It's probably
>>> > quite hard to reproduce.
>>> > This is in 0.94.5 but this feature is off by default.
>>> >
>>> > What's the general thought here, should I kill the current 0.94.6 rc for
>>> > this?
>>> > My gut says: Yes.
>>> >
>>> >
>>> > I'm also a bit worried about these:
>>> > 2013-03-14 01:42:42,271 DEBUG
>>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>> Opening
>>> > log for replication shared-dnds1-12-sfm.ops.sfdc.net
>>> %2C60020%2C1363220608780.1363220609572
>>> > at 0
>>> > 2013-03-14 01:42:42,358 WARN
>>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1
>>> Got:
>>> > java.io.EOFException
>>> > at java.io.DataInputStream.readFully(DataInputStream.java:180)
>>> > at java.io.DataInputStream.readFully(DataInputStream.java:152)
>>> > at
>>> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
>>> > at
>>> >
>>> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
>>> > at
>>> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
>>> > at
>>> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
>>> > at
>>> > org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
>>> > at
>>> >
>>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
>>> > 2013-03-14 01:42:42,358 WARN
>>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>> Waited
>>> > too long for this file, considering dumping
>>> > 2013-03-14 01:42:42,358 DEBUG
>>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>>> Unable
>>> > to open a reader, sleeping 1000 times 10
>>> >
>>> > This happens after bouncing the cluster a 2nd time and these messages
>>> > repeat every 10s (for hours now). This is a separate problem I think.
>>> >
>>> > -- Lars
>>> >
>>> > ------------------------------
>>> > *From:* Himanshu Vashishtha <hvashish@cs.ualberta.ca <javascript:;>>
>>> >
>>> > *To:* dev@hbase.apache.org <javascript:;>; lars hofhansl <
>>> larsh@apache.org <javascript:;>>
>>> > *Cc:* Ted Yu <yuzhihong@gmail.com <javascript:;>>
>>> > *Sent:* Wednesday, March 13, 2013 6:38 PM
>>> >
>>> > *Subject:* Re: Replication hosed after simple cluster restart
>>> >
>>> > This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
>>> > might not be able to move later on, resulting in bogus znodes.
>>> > I'll fix this asap. Weird it didn't happen in my testing earlier.
>>> > Sorry about this.
>>> >
>>> >
>>> > On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>>> wrote:
>>> > > Sorry 0.94.6RC1
>>> > > (I complain about folks not reporting the version all the time, and
>>> then
>>> > I do it too)
>>> > >
>>> > >
>>> > >
>>> > > ________________________________
>>> > > From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
>>> > > To: dev@hbase.apache.org <javascript:;>; lars hofhansl <
>>> larsh@apache.org <javascript:;>>
>>> > > Sent: Wednesday, March 13, 2013 6:17 PM
>>> > > Subject: Re: Replication hosed after simple cluster restart
>>> > >
>>> > >
>>> > > Did this happen on 0.94.5 ?
>>> > >
>>> > > Thanks
>>> > >
>>> > >
>>> > > On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>>> wrote:
>>> > >
>>> > > We just ran into an interesting scenario. We restarted a cluster that
>>> > was setup as a replication source.
>>> > >>The stop went cleanly.
>>> > >>
>>> > >>Upon restart *all* regionservers aborted within a few seconds with
>>> > variations of these errors:
>>> > >>http://pastebin.com/3iQVuBqS
>>> > >>
>>> > >>This is scary!
>>> > >>
>>> > >>-- Lars
>>> >
>>> >
>>> >
>>>
>>
>>
>> --
>> Best regards,
>>
>> - Andy
>>
>> Problems worthy of attack prove their worth by hitting back. - Piet Hein
>> (via Tom White)
Re: Replication hosed after simple cluster restart
Posted by lars hofhansl <la...@apache.org>.
I have proposed some minor changes (including adding the jitter) on HBASE-8099.
Turns out there already is a wait-time to give the cluster a chance to shutdown. It defaults to 2s, which was not enough in our case.
Let's do a test (if we think that can be done) in a different jira.
-- Lars
________________________________
From: Himanshu Vashishtha <hv...@cs.ualberta.ca>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Wednesday, March 13, 2013 8:59 PM
Subject: Re: Replication hosed after simple cluster restart
On Wed, Mar 13, 2013 at 8:48 PM, lars hofhansl <la...@apache.org> wrote:
> Yeah, lemme sink the RC... We do have a fix.
>
>
> Consider it sunk.
>
> In the end there are some more issues to discuss anyway.
> - Can we avoid RSs taking over queues during a clean shutdown/restart? Without multi we can actually loose data to replicate this way (one RS is shut down, another takes over and is itself shut down) - unless I misunderstand.
I agree. because even if they do move, they are not using locality as
the regionserver which eventually takes it over will remotely read the
log files. One way I can think of is do a scan on the available
regionservers in the /hbase/rs znodes and then decide whether it
should start the failover processing.
>
> - Should we stagger the attempts to move the queues for example with a random wait between 0 and 10s, so that not all RSs try at the same time?
> - A test for this scenario? (That's probably tricky)
How about adding a jitter (random sleep (0-10 sec]) in the run method
of the NodeFailoverWorker before it actually starts the failover
processing? I will try to come up with a test case.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Andrew Purtell <ap...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Sent: Wednesday, March 13, 2013 8:22 PM
> Subject: Re: Replication hosed after simple cluster restart
>
> If Himanshu (?) can fix it quickly we should try to get it in here IMHO.
>
> On Wednesday, March 13, 2013, Ted Yu wrote:
>
>> This was the JIRA that introduced copyQueuesFromRSUsingMulti():
>> HBASE-2611 Handle RS that fails while processing the failure of another one
>> (Himanshu Vashishtha)
>>
>> It went into 0.94.5
>> And the feature is off by default:
>>
>> <name>hbase.zookeeper.useMulti</name>
>> <value>false</value>
>>
>> The fact that Lars first reported the following problem meant that no other
>> user tried this feature.
>>
>> Hence I think 0.94.6 RC1 doesn't need to be sunk.
>>
>> Cheers
>>
>> On Wed, Mar 13, 2013 at 6:45 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>>
>> > Hey no problem. It's cool that we found it in a test env. It's probably
>> > quite hard to reproduce.
>> > This is in 0.94.5 but this feature is off by default.
>> >
>> > What's the general thought here, should I kill the current 0.94.6 rc for
>> > this?
>> > My gut says: Yes.
>> >
>> >
>> > I'm also a bit worried about these:
>> > 2013-03-14 01:42:42,271 DEBUG
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Opening
>> > log for replication shared-dnds1-12-sfm.ops.sfdc.net
>> %2C60020%2C1363220608780.1363220609572
>> > at 0
>> > 2013-03-14 01:42:42,358 WARN
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1
>> Got:
>> > java.io.EOFException
>> > at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> > at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> > at
>> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
>> > at
>> >
>> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
>> > at
>> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
>> > at
>> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
>> > at
>> > org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
>> > at
>> >
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
>> > at
>> >
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
>> > at
>> >
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
>> > 2013-03-14 01:42:42,358 WARN
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Waited
>> > too long for this file, considering dumping
>> > 2013-03-14 01:42:42,358 DEBUG
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Unable
>> > to open a reader, sleeping 1000 times 10
>> >
>> > This happens after bouncing the cluster a 2nd time and these messages
>> > repeat every 10s (for hours now). This is a separate problem I think.
>> >
>> > -- Lars
>> >
>> > ------------------------------
>> > *From:* Himanshu Vashishtha <hvashish@cs.ualberta.ca <javascript:;>>
>> >
>> > *To:* dev@hbase.apache.org <javascript:;>; lars hofhansl <
>> larsh@apache.org <javascript:;>>
>> > *Cc:* Ted Yu <yuzhihong@gmail.com <javascript:;>>
>> > *Sent:* Wednesday, March 13, 2013 6:38 PM
>> >
>> > *Subject:* Re: Replication hosed after simple cluster restart
>> >
>> > This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
>> > might not be able to move later on, resulting in bogus znodes.
>> > I'll fix this asap. Weird it didn't happen in my testing earlier.
>> > Sorry about this.
>> >
>> >
>> > On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>> > > Sorry 0.94.6RC1
>> > > (I complain about folks not reporting the version all the time, and
>> then
>> > I do it too)
>> > >
>> > >
>> > >
>> > > ________________________________
>> > > From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
>> > > To: dev@hbase.apache.org <javascript:;>; lars hofhansl <
>> larsh@apache.org <javascript:;>>
>> > > Sent: Wednesday, March 13, 2013 6:17 PM
>> > > Subject: Re: Replication hosed after simple cluster restart
>> > >
>> > >
>> > > Did this happen on 0.94.5 ?
>> > >
>> > > Thanks
>> > >
>> > >
>> > > On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>> > >
>> > > We just ran into an interesting scenario. We restarted a cluster that
>> > was setup as a replication source.
>> > >>The stop went cleanly.
>> > >>
>> > >>Upon restart *all* regionservers aborted within a few seconds with
>> > variations of these errors:
>> > >>http://pastebin.com/3iQVuBqS
>> > >>
>> > >>This is scary!
>> > >>
>> > >>-- Lars
>> >
>> >
>> >
>>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: Replication hosed after simple cluster restart
Posted by Himanshu Vashishtha <hv...@cs.ualberta.ca>.
On Wed, Mar 13, 2013 at 8:48 PM, lars hofhansl <la...@apache.org> wrote:
> Yeah, lemme sink the RC... We do have a fix.
>
>
> Consider it sunk.
>
> In the end there are some more issues to discuss anyway.
> - Can we avoid RSs taking over queues during a clean shutdown/restart? Without multi we can actually loose data to replicate this way (one RS is shut down, another takes over and is itself shut down) - unless I misunderstand.
I agree. because even if they do move, they are not using locality as
the regionserver which eventually takes it over will remotely read the
log files. One way I can think of is do a scan on the available
regionservers in the /hbase/rs znodes and then decide whether it
should start the failover processing.
>
> - Should we stagger the attempts to move the queues for example with a random wait between 0 and 10s, so that not all RSs try at the same time?
> - A test for this scenario? (That's probably tricky)
How about adding a jitter (random sleep (0-10 sec]) in the run method
of the NodeFailoverWorker before it actually starts the failover
processing? I will try to come up with a test case.
>
>
> -- Lars
>
>
>
> ________________________________
> From: Andrew Purtell <ap...@apache.org>
> To: "dev@hbase.apache.org" <de...@hbase.apache.org>
> Sent: Wednesday, March 13, 2013 8:22 PM
> Subject: Re: Replication hosed after simple cluster restart
>
> If Himanshu (?) can fix it quickly we should try to get it in here IMHO.
>
> On Wednesday, March 13, 2013, Ted Yu wrote:
>
>> This was the JIRA that introduced copyQueuesFromRSUsingMulti():
>> HBASE-2611 Handle RS that fails while processing the failure of another one
>> (Himanshu Vashishtha)
>>
>> It went into 0.94.5
>> And the feature is off by default:
>>
>> <name>hbase.zookeeper.useMulti</name>
>> <value>false</value>
>>
>> The fact that Lars first reported the following problem meant that no other
>> user tried this feature.
>>
>> Hence I think 0.94.6 RC1 doesn't need to be sunk.
>>
>> Cheers
>>
>> On Wed, Mar 13, 2013 at 6:45 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>>
>> > Hey no problem. It's cool that we found it in a test env. It's probably
>> > quite hard to reproduce.
>> > This is in 0.94.5 but this feature is off by default.
>> >
>> > What's the general thought here, should I kill the current 0.94.6 rc for
>> > this?
>> > My gut says: Yes.
>> >
>> >
>> > I'm also a bit worried about these:
>> > 2013-03-14 01:42:42,271 DEBUG
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Opening
>> > log for replication shared-dnds1-12-sfm.ops.sfdc.net
>> %2C60020%2C1363220608780.1363220609572
>> > at 0
>> > 2013-03-14 01:42:42,358 WARN
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1
>> Got:
>> > java.io.EOFException
>> > at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> > at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> > at
>> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
>> > at
>> >
>> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
>> > at
>> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
>> > at
>> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
>> > at
>> > org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
>> > at
>> >
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
>> > at
>> >
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
>> > at
>> >
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
>> > 2013-03-14 01:42:42,358 WARN
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Waited
>> > too long for this file, considering dumping
>> > 2013-03-14 01:42:42,358 DEBUG
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Unable
>> > to open a reader, sleeping 1000 times 10
>> >
>> > This happens after bouncing the cluster a 2nd time and these messages
>> > repeat every 10s (for hours now). This is a separate problem I think.
>> >
>> > -- Lars
>> >
>> > ------------------------------
>> > *From:* Himanshu Vashishtha <hvashish@cs.ualberta.ca <javascript:;>>
>> >
>> > *To:* dev@hbase.apache.org <javascript:;>; lars hofhansl <
>> larsh@apache.org <javascript:;>>
>> > *Cc:* Ted Yu <yuzhihong@gmail.com <javascript:;>>
>> > *Sent:* Wednesday, March 13, 2013 6:38 PM
>> >
>> > *Subject:* Re: Replication hosed after simple cluster restart
>> >
>> > This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
>> > might not be able to move later on, resulting in bogus znodes.
>> > I'll fix this asap. Weird it didn't happen in my testing earlier.
>> > Sorry about this.
>> >
>> >
>> > On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>> > > Sorry 0.94.6RC1
>> > > (I complain about folks not reporting the version all the time, and
>> then
>> > I do it too)
>> > >
>> > >
>> > >
>> > > ________________________________
>> > > From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
>> > > To: dev@hbase.apache.org <javascript:;>; lars hofhansl <
>> larsh@apache.org <javascript:;>>
>> > > Sent: Wednesday, March 13, 2013 6:17 PM
>> > > Subject: Re: Replication hosed after simple cluster restart
>> > >
>> > >
>> > > Did this happen on 0.94.5 ?
>> > >
>> > > Thanks
>> > >
>> > >
>> > > On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>> > >
>> > > We just ran into an interesting scenario. We restarted a cluster that
>> > was setup as a replication source.
>> > >>The stop went cleanly.
>> > >>
>> > >>Upon restart *all* regionservers aborted within a few seconds with
>> > variations of these errors:
>> > >>http://pastebin.com/3iQVuBqS
>> > >>
>> > >>This is scary!
>> > >>
>> > >>-- Lars
>> >
>> >
>> >
>>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: Replication hosed after simple cluster restart
Posted by lars hofhansl <la...@apache.org>.
Yeah, lemme sink the RC... We do have a fix.
Consider it sunk.
In the end there are some more issues to discuss anyway.
- Can we avoid RSs taking over queues during a clean shutdown/restart? Without multi we can actually loose data to replicate this way (one RS is shut down, another takes over and is itself shut down) - unless I misunderstand.
- Should we stagger the attempts to move the queues for example with a random wait between 0 and 10s, so that not all RSs try at the same time?
- A test for this scenario? (That's probably tricky)
-- Lars
________________________________
From: Andrew Purtell <ap...@apache.org>
To: "dev@hbase.apache.org" <de...@hbase.apache.org>
Sent: Wednesday, March 13, 2013 8:22 PM
Subject: Re: Replication hosed after simple cluster restart
If Himanshu (?) can fix it quickly we should try to get it in here IMHO.
On Wednesday, March 13, 2013, Ted Yu wrote:
> This was the JIRA that introduced copyQueuesFromRSUsingMulti():
> HBASE-2611 Handle RS that fails while processing the failure of another one
> (Himanshu Vashishtha)
>
> It went into 0.94.5
> And the feature is off by default:
>
> <name>hbase.zookeeper.useMulti</name>
> <value>false</value>
>
> The fact that Lars first reported the following problem meant that no other
> user tried this feature.
>
> Hence I think 0.94.6 RC1 doesn't need to be sunk.
>
> Cheers
>
> On Wed, Mar 13, 2013 at 6:45 PM, lars hofhansl <larsh@apache.org<javascript:;>>
> wrote:
>
> > Hey no problem. It's cool that we found it in a test env. It's probably
> > quite hard to reproduce.
> > This is in 0.94.5 but this feature is off by default.
> >
> > What's the general thought here, should I kill the current 0.94.6 rc for
> > this?
> > My gut says: Yes.
> >
> >
> > I'm also a bit worried about these:
> > 2013-03-14 01:42:42,271 DEBUG
> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Opening
> > log for replication shared-dnds1-12-sfm.ops.sfdc.net
> %2C60020%2C1363220608780.1363220609572
> > at 0
> > 2013-03-14 01:42:42,358 WARN
> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1
> Got:
> > java.io.EOFException
> > at java.io.DataInputStream.readFully(DataInputStream.java:180)
> > at java.io.DataInputStream.readFully(DataInputStream.java:152)
> > at
> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
> > at
> >
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
> > at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
> > at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
> > at
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
> > at
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
> > at
> > org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
> > at
> >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
> > at
> >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
> > at
> >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
> > 2013-03-14 01:42:42,358 WARN
> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Waited
> > too long for this file, considering dumping
> > 2013-03-14 01:42:42,358 DEBUG
> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Unable
> > to open a reader, sleeping 1000 times 10
> >
> > This happens after bouncing the cluster a 2nd time and these messages
> > repeat every 10s (for hours now). This is a separate problem I think.
> >
> > -- Lars
> >
> > ------------------------------
> > *From:* Himanshu Vashishtha <hvashish@cs.ualberta.ca <javascript:;>>
> >
> > *To:* dev@hbase.apache.org <javascript:;>; lars hofhansl <
> larsh@apache.org <javascript:;>>
> > *Cc:* Ted Yu <yuzhihong@gmail.com <javascript:;>>
> > *Sent:* Wednesday, March 13, 2013 6:38 PM
> >
> > *Subject:* Re: Replication hosed after simple cluster restart
> >
> > This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
> > might not be able to move later on, resulting in bogus znodes.
> > I'll fix this asap. Weird it didn't happen in my testing earlier.
> > Sorry about this.
> >
> >
> > On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <larsh@apache.org<javascript:;>>
> wrote:
> > > Sorry 0.94.6RC1
> > > (I complain about folks not reporting the version all the time, and
> then
> > I do it too)
> > >
> > >
> > >
> > > ________________________________
> > > From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
> > > To: dev@hbase.apache.org <javascript:;>; lars hofhansl <
> larsh@apache.org <javascript:;>>
> > > Sent: Wednesday, March 13, 2013 6:17 PM
> > > Subject: Re: Replication hosed after simple cluster restart
> > >
> > >
> > > Did this happen on 0.94.5 ?
> > >
> > > Thanks
> > >
> > >
> > > On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <larsh@apache.org<javascript:;>>
> wrote:
> > >
> > > We just ran into an interesting scenario. We restarted a cluster that
> > was setup as a replication source.
> > >>The stop went cleanly.
> > >>
> > >>Upon restart *all* regionservers aborted within a few seconds with
> > variations of these errors:
> > >>http://pastebin.com/3iQVuBqS
> > >>
> > >>This is scary!
> > >>
> > >>-- Lars
> >
> >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Replication hosed after simple cluster restart
Posted by Himanshu Vashishtha <hv...@cs.ualberta.ca>.
Yes, a patch is there on https://issues.apache.org/jira/browse/HBASE-8099.
On Wed, Mar 13, 2013 at 8:22 PM, Andrew Purtell <ap...@apache.org> wrote:
> If Himanshu (?) can fix it quickly we should try to get it in here IMHO.
>
> On Wednesday, March 13, 2013, Ted Yu wrote:
>
>> This was the JIRA that introduced copyQueuesFromRSUsingMulti():
>> HBASE-2611 Handle RS that fails while processing the failure of another one
>> (Himanshu Vashishtha)
>>
>> It went into 0.94.5
>> And the feature is off by default:
>>
>> <name>hbase.zookeeper.useMulti</name>
>> <value>false</value>
>>
>> The fact that Lars first reported the following problem meant that no other
>> user tried this feature.
>>
>> Hence I think 0.94.6 RC1 doesn't need to be sunk.
>>
>> Cheers
>>
>> On Wed, Mar 13, 2013 at 6:45 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>>
>> > Hey no problem. It's cool that we found it in a test env. It's probably
>> > quite hard to reproduce.
>> > This is in 0.94.5 but this feature is off by default.
>> >
>> > What's the general thought here, should I kill the current 0.94.6 rc for
>> > this?
>> > My gut says: Yes.
>> >
>> >
>> > I'm also a bit worried about these:
>> > 2013-03-14 01:42:42,271 DEBUG
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Opening
>> > log for replication shared-dnds1-12-sfm.ops.sfdc.net
>> %2C60020%2C1363220608780.1363220609572
>> > at 0
>> > 2013-03-14 01:42:42,358 WARN
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1
>> Got:
>> > java.io.EOFException
>> > at java.io.DataInputStream.readFully(DataInputStream.java:180)
>> > at java.io.DataInputStream.readFully(DataInputStream.java:152)
>> > at
>> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
>> > at
>> >
>> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
>> > at
>> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
>> > at
>> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
>> > at
>> >
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
>> > at
>> > org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
>> > at
>> >
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
>> > at
>> >
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
>> > at
>> >
>> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
>> > 2013-03-14 01:42:42,358 WARN
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Waited
>> > too long for this file, considering dumping
>> > 2013-03-14 01:42:42,358 DEBUG
>> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
>> Unable
>> > to open a reader, sleeping 1000 times 10
>> >
>> > This happens after bouncing the cluster a 2nd time and these messages
>> > repeat every 10s (for hours now). This is a separate problem I think.
>> >
>> > -- Lars
>> >
>> > ------------------------------
>> > *From:* Himanshu Vashishtha <hvashish@cs.ualberta.ca <javascript:;>>
>> >
>> > *To:* dev@hbase.apache.org <javascript:;>; lars hofhansl <
>> larsh@apache.org <javascript:;>>
>> > *Cc:* Ted Yu <yuzhihong@gmail.com <javascript:;>>
>> > *Sent:* Wednesday, March 13, 2013 6:38 PM
>> >
>> > *Subject:* Re: Replication hosed after simple cluster restart
>> >
>> > This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
>> > might not be able to move later on, resulting in bogus znodes.
>> > I'll fix this asap. Weird it didn't happen in my testing earlier.
>> > Sorry about this.
>> >
>> >
>> > On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>> > > Sorry 0.94.6RC1
>> > > (I complain about folks not reporting the version all the time, and
>> then
>> > I do it too)
>> > >
>> > >
>> > >
>> > > ________________________________
>> > > From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
>> > > To: dev@hbase.apache.org <javascript:;>; lars hofhansl <
>> larsh@apache.org <javascript:;>>
>> > > Sent: Wednesday, March 13, 2013 6:17 PM
>> > > Subject: Re: Replication hosed after simple cluster restart
>> > >
>> > >
>> > > Did this happen on 0.94.5 ?
>> > >
>> > > Thanks
>> > >
>> > >
>> > > On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <larsh@apache.org<javascript:;>>
>> wrote:
>> > >
>> > > We just ran into an interesting scenario. We restarted a cluster that
>> > was setup as a replication source.
>> > >>The stop went cleanly.
>> > >>
>> > >>Upon restart *all* regionservers aborted within a few seconds with
>> > variations of these errors:
>> > >>http://pastebin.com/3iQVuBqS
>> > >>
>> > >>This is scary!
>> > >>
>> > >>-- Lars
>> >
>> >
>> >
>>
>
>
> --
> Best regards,
>
> - Andy
>
> Problems worthy of attack prove their worth by hitting back. - Piet Hein
> (via Tom White)
Re: Replication hosed after simple cluster restart
Posted by Andrew Purtell <ap...@apache.org>.
If Himanshu (?) can fix it quickly we should try to get it in here IMHO.
On Wednesday, March 13, 2013, Ted Yu wrote:
> This was the JIRA that introduced copyQueuesFromRSUsingMulti():
> HBASE-2611 Handle RS that fails while processing the failure of another one
> (Himanshu Vashishtha)
>
> It went into 0.94.5
> And the feature is off by default:
>
> <name>hbase.zookeeper.useMulti</name>
> <value>false</value>
>
> The fact that Lars first reported the following problem meant that no other
> user tried this feature.
>
> Hence I think 0.94.6 RC1 doesn't need to be sunk.
>
> Cheers
>
> On Wed, Mar 13, 2013 at 6:45 PM, lars hofhansl <larsh@apache.org<javascript:;>>
> wrote:
>
> > Hey no problem. It's cool that we found it in a test env. It's probably
> > quite hard to reproduce.
> > This is in 0.94.5 but this feature is off by default.
> >
> > What's the general thought here, should I kill the current 0.94.6 rc for
> > this?
> > My gut says: Yes.
> >
> >
> > I'm also a bit worried about these:
> > 2013-03-14 01:42:42,271 DEBUG
> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Opening
> > log for replication shared-dnds1-12-sfm.ops.sfdc.net
> %2C60020%2C1363220608780.1363220609572
> > at 0
> > 2013-03-14 01:42:42,358 WARN
> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1
> Got:
> > java.io.EOFException
> > at java.io.DataInputStream.readFully(DataInputStream.java:180)
> > at java.io.DataInputStream.readFully(DataInputStream.java:152)
> > at
> > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
> > at
> >
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
> > at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
> > at
> > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
> > at
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
> > at
> >
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
> > at
> > org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
> > at
> >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
> > at
> >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
> > at
> >
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
> > 2013-03-14 01:42:42,358 WARN
> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Waited
> > too long for this file, considering dumping
> > 2013-03-14 01:42:42,358 DEBUG
> > org.apache.hadoop.hbase.replication.regionserver.ReplicationSource:
> Unable
> > to open a reader, sleeping 1000 times 10
> >
> > This happens after bouncing the cluster a 2nd time and these messages
> > repeat every 10s (for hours now). This is a separate problem I think.
> >
> > -- Lars
> >
> > ------------------------------
> > *From:* Himanshu Vashishtha <hvashish@cs.ualberta.ca <javascript:;>>
> >
> > *To:* dev@hbase.apache.org <javascript:;>; lars hofhansl <
> larsh@apache.org <javascript:;>>
> > *Cc:* Ted Yu <yuzhihong@gmail.com <javascript:;>>
> > *Sent:* Wednesday, March 13, 2013 6:38 PM
> >
> > *Subject:* Re: Replication hosed after simple cluster restart
> >
> > This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
> > might not be able to move later on, resulting in bogus znodes.
> > I'll fix this asap. Weird it didn't happen in my testing earlier.
> > Sorry about this.
> >
> >
> > On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <larsh@apache.org<javascript:;>>
> wrote:
> > > Sorry 0.94.6RC1
> > > (I complain about folks not reporting the version all the time, and
> then
> > I do it too)
> > >
> > >
> > >
> > > ________________________________
> > > From: Ted Yu <yuzhihong@gmail.com <javascript:;>>
> > > To: dev@hbase.apache.org <javascript:;>; lars hofhansl <
> larsh@apache.org <javascript:;>>
> > > Sent: Wednesday, March 13, 2013 6:17 PM
> > > Subject: Re: Replication hosed after simple cluster restart
> > >
> > >
> > > Did this happen on 0.94.5 ?
> > >
> > > Thanks
> > >
> > >
> > > On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <larsh@apache.org<javascript:;>>
> wrote:
> > >
> > > We just ran into an interesting scenario. We restarted a cluster that
> > was setup as a replication source.
> > >>The stop went cleanly.
> > >>
> > >>Upon restart *all* regionservers aborted within a few seconds with
> > variations of these errors:
> > >>http://pastebin.com/3iQVuBqS
> > >>
> > >>This is scary!
> > >>
> > >>-- Lars
> >
> >
> >
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet Hein
(via Tom White)
Re: Replication hosed after simple cluster restart
Posted by Ted Yu <yu...@gmail.com>.
This was the JIRA that introduced copyQueuesFromRSUsingMulti():
HBASE-2611 Handle RS that fails while processing the failure of another one
(Himanshu Vashishtha)
It went into 0.94.5
And the feature is off by default:
<name>hbase.zookeeper.useMulti</name>
<value>false</value>
The fact that Lars first reported the following problem meant that no other
user tried this feature.
Hence I think 0.94.6 RC1 doesn't need to be sunk.
Cheers
On Wed, Mar 13, 2013 at 6:45 PM, lars hofhansl <la...@apache.org> wrote:
> Hey no problem. It's cool that we found it in a test env. It's probably
> quite hard to reproduce.
> This is in 0.94.5 but this feature is off by default.
>
> What's the general thought here, should I kill the current 0.94.6 rc for
> this?
> My gut says: Yes.
>
>
> I'm also a bit worried about these:
> 2013-03-14 01:42:42,271 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening
> log for replication shared-dnds1-12-sfm.ops.sfdc.net%2C60020%2C1363220608780.1363220609572
> at 0
> 2013-03-14 01:42:42,358 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got:
> java.io.EOFException
> at java.io.DataInputStream.readFully(DataInputStream.java:180)
> at java.io.DataInputStream.readFully(DataInputStream.java:152)
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
> at
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
> 2013-03-14 01:42:42,358 WARN
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Waited
> too long for this file, considering dumping
> 2013-03-14 01:42:42,358 DEBUG
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable
> to open a reader, sleeping 1000 times 10
>
> This happens after bouncing the cluster a 2nd time and these messages
> repeat every 10s (for hours now). This is a separate problem I think.
>
> -- Lars
>
> ------------------------------
> *From:* Himanshu Vashishtha <hv...@cs.ualberta.ca>
>
> *To:* dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> *Cc:* Ted Yu <yu...@gmail.com>
> *Sent:* Wednesday, March 13, 2013 6:38 PM
>
> *Subject:* Re: Replication hosed after simple cluster restart
>
> This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
> might not be able to move later on, resulting in bogus znodes.
> I'll fix this asap. Weird it didn't happen in my testing earlier.
> Sorry about this.
>
>
> On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <la...@apache.org> wrote:
> > Sorry 0.94.6RC1
> > (I complain about folks not reporting the version all the time, and then
> I do it too)
> >
> >
> >
> > ________________________________
> > From: Ted Yu <yu...@gmail.com>
> > To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> > Sent: Wednesday, March 13, 2013 6:17 PM
> > Subject: Re: Replication hosed after simple cluster restart
> >
> >
> > Did this happen on 0.94.5 ?
> >
> > Thanks
> >
> >
> > On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <la...@apache.org> wrote:
> >
> > We just ran into an interesting scenario. We restarted a cluster that
> was setup as a replication source.
> >>The stop went cleanly.
> >>
> >>Upon restart *all* regionservers aborted within a few seconds with
> variations of these errors:
> >>http://pastebin.com/3iQVuBqS
> >>
> >>This is scary!
> >>
> >>-- Lars
>
>
>
Re: Replication hosed after simple cluster restart
Posted by lars hofhansl <la...@apache.org>.
Hey no problem. It's cool that we found it in a test env. It's probably quite hard to reproduce.
This is in 0.94.5 but this feature is off by default.
What's the general thought here, should I kill the current 0.94.6 rc for this?
My gut says: Yes.
I'm also a bit worried about these:
2013-03-14 01:42:42,271 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Opening log for replication shared-dnds1-12-sfm.ops.sfdc.net%2C60020%2C1363220608780.1363220609572 at 0
2013-03-14 01:42:42,358 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: 1 Got:
java.io.EOFException
at java.io.DataInputStream.readFully(DataInputStream.java:180)
at java.io.DataInputStream.readFully(DataInputStream.java:152)
at org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
at org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
at org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
at org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
at org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:728)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationHLogReaderManager.openReader(ReplicationHLogReaderManager.java:67)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.openReader(ReplicationSource.java:507)
at org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.run(ReplicationSource.java:313)
2013-03-14 01:42:42,358 WARN org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Waited too long for this file, considering dumping
2013-03-14 01:42:42,358 DEBUG org.apache.hadoop.hbase.replication.regionserver.ReplicationSource: Unable to open a reader, sleeping 1000 times 10
This happens after bouncing the cluster a 2nd time and these messages repeat every 10s (for hours now). This is a separate problem I think.
-- Lars
________________________________
From: Himanshu Vashishtha <hv...@cs.ualberta.ca>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Cc: Ted Yu <yu...@gmail.com>
Sent: Wednesday, March 13, 2013 6:38 PM
Subject: Re: Replication hosed after simple cluster restart
This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
might not be able to move later on, resulting in bogus znodes.
I'll fix this asap. Weird it didn't happen in my testing earlier.
Sorry about this.
On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <la...@apache.org> wrote:
> Sorry 0.94.6RC1
> (I complain about folks not reporting the version all the time, and then I do it too)
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Wednesday, March 13, 2013 6:17 PM
> Subject: Re: Replication hosed after simple cluster restart
>
>
> Did this happen on 0.94.5 ?
>
> Thanks
>
>
> On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <la...@apache.org> wrote:
>
> We just ran into an interesting scenario. We restarted a cluster that was setup as a replication source.
>>The stop went cleanly.
>>
>>Upon restart *all* regionservers aborted within a few seconds with variations of these errors:
>>http://pastebin.com/3iQVuBqS
>>
>>This is scary!
>>
>>-- Lars
Re: Replication hosed after simple cluster restart
Posted by Himanshu Vashishtha <hv...@cs.ualberta.ca>.
This is bad. Yes, copyQueuesFromRSUsingMulti returns a list which it
might not be able to move later on, resulting in bogus znodes.
I'll fix this asap. Weird it didn't happen in my testing earlier.
Sorry about this.
On Wed, Mar 13, 2013 at 6:27 PM, lars hofhansl <la...@apache.org> wrote:
> Sorry 0.94.6RC1
> (I complain about folks not reporting the version all the time, and then I do it too)
>
>
>
> ________________________________
> From: Ted Yu <yu...@gmail.com>
> To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
> Sent: Wednesday, March 13, 2013 6:17 PM
> Subject: Re: Replication hosed after simple cluster restart
>
>
> Did this happen on 0.94.5 ?
>
> Thanks
>
>
> On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <la...@apache.org> wrote:
>
> We just ran into an interesting scenario. We restarted a cluster that was setup as a replication source.
>>The stop went cleanly.
>>
>>Upon restart *all* regionservers aborted within a few seconds with variations of these errors:
>>http://pastebin.com/3iQVuBqS
>>
>>This is scary!
>>
>>-- Lars
Re: Replication hosed after simple cluster restart
Posted by lars hofhansl <la...@apache.org>.
Sorry 0.94.6RC1
(I complain about folks not reporting the version all the time, and then I do it too)
________________________________
From: Ted Yu <yu...@gmail.com>
To: dev@hbase.apache.org; lars hofhansl <la...@apache.org>
Sent: Wednesday, March 13, 2013 6:17 PM
Subject: Re: Replication hosed after simple cluster restart
Did this happen on 0.94.5 ?
Thanks
On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <la...@apache.org> wrote:
We just ran into an interesting scenario. We restarted a cluster that was setup as a replication source.
>The stop went cleanly.
>
>Upon restart *all* regionservers aborted within a few seconds with variations of these errors:
>http://pastebin.com/3iQVuBqS
>
>This is scary!
>
>-- Lars
Re: Replication hosed after simple cluster restart
Posted by Ted Yu <yu...@gmail.com>.
Did this happen on 0.94.5 ?
Thanks
On Wed, Mar 13, 2013 at 6:12 PM, lars hofhansl <la...@apache.org> wrote:
> We just ran into an interesting scenario. We restarted a cluster that was
> setup as a replication source.
> The stop went cleanly.
>
> Upon restart *all* regionservers aborted within a few seconds with
> variations of these errors:
> http://pastebin.com/3iQVuBqS
>
> This is scary!
>
> -- Lars
Re: Replication hosed after simple cluster restart
Posted by lars hofhansl <la...@apache.org>.
No, this all RSs trying to copy the failed RS' queues over. Most of them will fail, but still report all the queues that they *tried* to move. For each of those a ReplicationSource is setup, which then fails when attempting to update the replication status (because their ZK nodes were never created) and then causing the RS to abort (because the RS is setup as abortable for the zkHelper used here - which actually might be yet another problem).
-- Lars
________________________________
From: Stack <st...@duboce.net>
To: HBase Dev List <de...@hbase.apache.org>
Sent: Wednesday, March 13, 2013 6:43 PM
Subject: Re: Replication hosed after simple cluster restart
Not sure I follow. Is this our making use of multi against a zk ensemble
that doesn't support it?
On Mar 13, 2013 6:22 PM, "lars hofhansl" <la...@apache.org> wrote:
> I suppose the problem could be in
> zkHelper.copyQueuesFromRSUsingMulti(rsZnode) as called from
> ReplicationSourceManager.NodeFailoverWorker.run().
> copyQueuesFromRSUsingMulti will return the queues it read even when the
> multi operation failed (because another RS managed to execute it first).
>
> -- Lars
>
>
>
> ________________________________
> From: lars hofhansl <la...@apache.org>
> To: hbase-dev <de...@hbase.apache.org>
> Sent: Wednesday, March 13, 2013 6:12 PM
> Subject: Replication hosed after simple cluster restart
>
> We just ran into an interesting scenario. We restarted a cluster that was
> setup as a replication source.
> The stop went cleanly.
>
> Upon restart *all* regionservers aborted within a few seconds with
> variations of these errors:
> http://pastebin.com/3iQVuBqS
>
> This is scary!
>
> -- Lars
Re: Replication hosed after simple cluster restart
Posted by Stack <st...@duboce.net>.
Not sure I follow. Is this our making use of multi against a zk ensemble
that doesn't support it?
On Mar 13, 2013 6:22 PM, "lars hofhansl" <la...@apache.org> wrote:
> I suppose the problem could be in
> zkHelper.copyQueuesFromRSUsingMulti(rsZnode) as called from
> ReplicationSourceManager.NodeFailoverWorker.run().
> copyQueuesFromRSUsingMulti will return the queues it read even when the
> multi operation failed (because another RS managed to execute it first).
>
> -- Lars
>
>
>
> ________________________________
> From: lars hofhansl <la...@apache.org>
> To: hbase-dev <de...@hbase.apache.org>
> Sent: Wednesday, March 13, 2013 6:12 PM
> Subject: Replication hosed after simple cluster restart
>
> We just ran into an interesting scenario. We restarted a cluster that was
> setup as a replication source.
> The stop went cleanly.
>
> Upon restart *all* regionservers aborted within a few seconds with
> variations of these errors:
> http://pastebin.com/3iQVuBqS
>
> This is scary!
>
> -- Lars
Re: Replication hosed after simple cluster restart
Posted by lars hofhansl <la...@apache.org>.
I suppose the problem could be in zkHelper.copyQueuesFromRSUsingMulti(rsZnode) as called from ReplicationSourceManager.NodeFailoverWorker.run().
copyQueuesFromRSUsingMulti will return the queues it read even when the multi operation failed (because another RS managed to execute it first).
-- Lars
________________________________
From: lars hofhansl <la...@apache.org>
To: hbase-dev <de...@hbase.apache.org>
Sent: Wednesday, March 13, 2013 6:12 PM
Subject: Replication hosed after simple cluster restart
We just ran into an interesting scenario. We restarted a cluster that was setup as a replication source.
The stop went cleanly.
Upon restart *all* regionservers aborted within a few seconds with variations of these errors:
http://pastebin.com/3iQVuBqS
This is scary!
-- Lars