You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by tsuna <ts...@gmail.com> on 2016/01/27 07:02:35 UTC

All RegionServers stuck on BadVersion from ZK after cluster restart

Hi,
after a planned power outage one of our HBase clusters isn’t coming back up
healthy.  The master shows the 16 region servers but zero regions.  All the
RegionServers are experiencing the same problem, which is that they’re
getting a BadVersion error from ZooKeeper.  This was with HBase 1.1.2 and I
just upgraded all the nodes to 1.1.3 to see if this would make a
difference, but it didn’t.

2016-01-27 05:54:02,402 WARN  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
coordination.ZkSplitLogWorkerCoordination: BADVERSION failed to assert
ownership for /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
%2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
%252C9104%252C1452811286456.default.1453728374800
org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode =
BadVersion for /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
%2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
%252C9104%252C1452811286456.default.1453728374800
at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:429)
at
org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.attemptToOwnTask(ZkSplitLogWorkerCoordination.java:370)
at
org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$1.progress(ZkSplitLogWorkerCoordination.java:304)
at
org.apache.hadoop.hbase.util.FSHDFSUtils.checkIfCancelled(FSHDFSUtils.java:329)
at
org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:244)
at
org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:162)
at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:761)
at
org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
at
org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:235)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
at
org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-01-27 05:54:02,404 WARN  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
coordination.ZkSplitLogWorkerCoordination: Failed to heartbeat the
task/hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
%2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
%252C9104%252C1452811286456.default.1453728374800

I’m attaching the full log of the RS this was extracted from, which I just
restarted on 1.1.3, in case that’s of any help.

I’ve never seen this before and after a bit of digging, I’m not really
going anywhere.  Any ideas / suggestions?

-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by tsuna <ts...@gmail.com>.
Resending to user@hadoop.apache.org now that I’m subscribed to that list.

On Thu, Jan 28, 2016 at 10:50 AM, tsuna <ts...@gmail.com> wrote:

> Just to close the loop on this ordeal…
>
> I started by clearing /hbase/splitWAL in ZK and restarting all the RS and
> the HM.  This didn’t change anything.
>
> On Wed, Jan 27, 2016 at 8:42 AM, tsuna <ts...@gmail.com> wrote:
> > 16/01/27 16:33:39 INFO namenode.FSNamesystem: Recovering [Lease.
> > Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
> > src=/hbase/WALs/r12s1.sjc.aristanetworks.com
> ,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com
> %2C9104%2C1452811288618.default.1453728791276
> > 16/01/27 16:33:39 WARN BlockStateChange: BLOCK*
> > BlockInfoUnderConstruction.initLeaseRecovery: No blocks found, lease
> > removed.
>
> I ran dfsadmin fsck -move to make sure that all the files that had lost
> blocks were moved to /lost+found, and this obviously didn’t help HBase,
> because as I stated earlier, only one WAL had lost a block, and 94% of the
> blocks lost affected the HFile of one of the regions.
>
> Yet, somehow, the error above appeared for every single one of the region
> servers, and I ended up having to move more WAL files manually to
> /lost+found:
>
> foo@r12s3:~/hadoop-2.7.1$ ./bin/hdfs dfs -ls /lost+found
> Found 15 items
> drwxr--r--   - foo supergroup          0 2016-01-28 05:56 /lost+found/hbase
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:33 /lost+found/
> r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:29 /lost+found/
> r12s10.sjc.aristanetworks.com%2C9104%2C1452811286704.default.1453728581434
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:31 /lost+found/
> r12s11.sjc.aristanetworks.com%2C9104%2C1452811286222.default.1453728710303
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
> r12s13.sjc.aristanetworks.com%2C9104%2C1452811287287.default.1453728621698
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
> r12s14.sjc.aristanetworks.com%2C9104%2C1452811286288.default.1453728336644
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
> r12s15.sjc.aristanetworks.com%2C9104%2C1453158959800.default.1453728342559
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
> r12s16.sjc.aristanetworks.com%2C9104%2C1452811286456.default.1453728374800
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
> r12s2.sjc.aristanetworks.com%2C9104%2C1452811286448.default.1453728137282
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
> r12s3.sjc.aristanetworks.com%2C9104%2C1452811286093.default.1453728393926
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:35 /lost+found/
> r12s4.sjc.aristanetworks.com%2C9104%2C1452811289547.default.1453728949397
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
> r12s5.sjc.aristanetworks.com%2C9104%2C1452811125084.default.1453728624262
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
> r12s6.sjc.aristanetworks.com%2C9104%2C1452811286154.default.1453728483550
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
> r12s7.sjc.aristanetworks.com%2C9104%2C1452811287528.default.1453728528180
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
> r12s8.sjc.aristanetworks.com%2C9104%2C1452811287196.default.1453728125912
>
> After doing this and restarting the HMaster, everything came back up
> fine.  I don’t know if doing this caused any additional data loss – this is
> a dev cluster so data loss isn’t a big deal, but if I was to run into this
> issue in production, I would certainly be very nervous about this whole
> situation.
>
> This might turn more into an HDFS question at this point, so I’m Cc’ing
> hdfs-user@ just in case anybody has anything to say there.
>
> We’re going to upgrade to Hadoop 2.7.2 soon, just in case.
>

-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by tsuna <ts...@gmail.com>.
Resending to user@hadoop.apache.org now that I’m subscribed to that list.

On Thu, Jan 28, 2016 at 10:50 AM, tsuna <ts...@gmail.com> wrote:

> Just to close the loop on this ordeal…
>
> I started by clearing /hbase/splitWAL in ZK and restarting all the RS and
> the HM.  This didn’t change anything.
>
> On Wed, Jan 27, 2016 at 8:42 AM, tsuna <ts...@gmail.com> wrote:
> > 16/01/27 16:33:39 INFO namenode.FSNamesystem: Recovering [Lease.
> > Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
> > src=/hbase/WALs/r12s1.sjc.aristanetworks.com
> ,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com
> %2C9104%2C1452811288618.default.1453728791276
> > 16/01/27 16:33:39 WARN BlockStateChange: BLOCK*
> > BlockInfoUnderConstruction.initLeaseRecovery: No blocks found, lease
> > removed.
>
> I ran dfsadmin fsck -move to make sure that all the files that had lost
> blocks were moved to /lost+found, and this obviously didn’t help HBase,
> because as I stated earlier, only one WAL had lost a block, and 94% of the
> blocks lost affected the HFile of one of the regions.
>
> Yet, somehow, the error above appeared for every single one of the region
> servers, and I ended up having to move more WAL files manually to
> /lost+found:
>
> foo@r12s3:~/hadoop-2.7.1$ ./bin/hdfs dfs -ls /lost+found
> Found 15 items
> drwxr--r--   - foo supergroup          0 2016-01-28 05:56 /lost+found/hbase
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:33 /lost+found/
> r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:29 /lost+found/
> r12s10.sjc.aristanetworks.com%2C9104%2C1452811286704.default.1453728581434
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:31 /lost+found/
> r12s11.sjc.aristanetworks.com%2C9104%2C1452811286222.default.1453728710303
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
> r12s13.sjc.aristanetworks.com%2C9104%2C1452811287287.default.1453728621698
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
> r12s14.sjc.aristanetworks.com%2C9104%2C1452811286288.default.1453728336644
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
> r12s15.sjc.aristanetworks.com%2C9104%2C1453158959800.default.1453728342559
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
> r12s16.sjc.aristanetworks.com%2C9104%2C1452811286456.default.1453728374800
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
> r12s2.sjc.aristanetworks.com%2C9104%2C1452811286448.default.1453728137282
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
> r12s3.sjc.aristanetworks.com%2C9104%2C1452811286093.default.1453728393926
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:35 /lost+found/
> r12s4.sjc.aristanetworks.com%2C9104%2C1452811289547.default.1453728949397
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
> r12s5.sjc.aristanetworks.com%2C9104%2C1452811125084.default.1453728624262
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
> r12s6.sjc.aristanetworks.com%2C9104%2C1452811286154.default.1453728483550
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
> r12s7.sjc.aristanetworks.com%2C9104%2C1452811287528.default.1453728528180
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
> r12s8.sjc.aristanetworks.com%2C9104%2C1452811287196.default.1453728125912
>
> After doing this and restarting the HMaster, everything came back up
> fine.  I don’t know if doing this caused any additional data loss – this is
> a dev cluster so data loss isn’t a big deal, but if I was to run into this
> issue in production, I would certainly be very nervous about this whole
> situation.
>
> This might turn more into an HDFS question at this point, so I’m Cc’ing
> hdfs-user@ just in case anybody has anything to say there.
>
> We’re going to upgrade to Hadoop 2.7.2 soon, just in case.
>

-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by tsuna <ts...@gmail.com>.
Resending to user@hadoop.apache.org now that I’m subscribed to that list.

On Thu, Jan 28, 2016 at 10:50 AM, tsuna <ts...@gmail.com> wrote:

> Just to close the loop on this ordeal…
>
> I started by clearing /hbase/splitWAL in ZK and restarting all the RS and
> the HM.  This didn’t change anything.
>
> On Wed, Jan 27, 2016 at 8:42 AM, tsuna <ts...@gmail.com> wrote:
> > 16/01/27 16:33:39 INFO namenode.FSNamesystem: Recovering [Lease.
> > Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
> > src=/hbase/WALs/r12s1.sjc.aristanetworks.com
> ,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com
> %2C9104%2C1452811288618.default.1453728791276
> > 16/01/27 16:33:39 WARN BlockStateChange: BLOCK*
> > BlockInfoUnderConstruction.initLeaseRecovery: No blocks found, lease
> > removed.
>
> I ran dfsadmin fsck -move to make sure that all the files that had lost
> blocks were moved to /lost+found, and this obviously didn’t help HBase,
> because as I stated earlier, only one WAL had lost a block, and 94% of the
> blocks lost affected the HFile of one of the regions.
>
> Yet, somehow, the error above appeared for every single one of the region
> servers, and I ended up having to move more WAL files manually to
> /lost+found:
>
> foo@r12s3:~/hadoop-2.7.1$ ./bin/hdfs dfs -ls /lost+found
> Found 15 items
> drwxr--r--   - foo supergroup          0 2016-01-28 05:56 /lost+found/hbase
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:33 /lost+found/
> r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:29 /lost+found/
> r12s10.sjc.aristanetworks.com%2C9104%2C1452811286704.default.1453728581434
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:31 /lost+found/
> r12s11.sjc.aristanetworks.com%2C9104%2C1452811286222.default.1453728710303
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
> r12s13.sjc.aristanetworks.com%2C9104%2C1452811287287.default.1453728621698
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
> r12s14.sjc.aristanetworks.com%2C9104%2C1452811286288.default.1453728336644
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
> r12s15.sjc.aristanetworks.com%2C9104%2C1453158959800.default.1453728342559
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
> r12s16.sjc.aristanetworks.com%2C9104%2C1452811286456.default.1453728374800
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
> r12s2.sjc.aristanetworks.com%2C9104%2C1452811286448.default.1453728137282
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
> r12s3.sjc.aristanetworks.com%2C9104%2C1452811286093.default.1453728393926
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:35 /lost+found/
> r12s4.sjc.aristanetworks.com%2C9104%2C1452811289547.default.1453728949397
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
> r12s5.sjc.aristanetworks.com%2C9104%2C1452811125084.default.1453728624262
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
> r12s6.sjc.aristanetworks.com%2C9104%2C1452811286154.default.1453728483550
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
> r12s7.sjc.aristanetworks.com%2C9104%2C1452811287528.default.1453728528180
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
> r12s8.sjc.aristanetworks.com%2C9104%2C1452811287196.default.1453728125912
>
> After doing this and restarting the HMaster, everything came back up
> fine.  I don’t know if doing this caused any additional data loss – this is
> a dev cluster so data loss isn’t a big deal, but if I was to run into this
> issue in production, I would certainly be very nervous about this whole
> situation.
>
> This might turn more into an HDFS question at this point, so I’m Cc’ing
> hdfs-user@ just in case anybody has anything to say there.
>
> We’re going to upgrade to Hadoop 2.7.2 soon, just in case.
>

-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by tsuna <ts...@gmail.com>.
Resending to user@hadoop.apache.org now that I’m subscribed to that list.

On Thu, Jan 28, 2016 at 10:50 AM, tsuna <ts...@gmail.com> wrote:

> Just to close the loop on this ordeal…
>
> I started by clearing /hbase/splitWAL in ZK and restarting all the RS and
> the HM.  This didn’t change anything.
>
> On Wed, Jan 27, 2016 at 8:42 AM, tsuna <ts...@gmail.com> wrote:
> > 16/01/27 16:33:39 INFO namenode.FSNamesystem: Recovering [Lease.
> > Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
> > src=/hbase/WALs/r12s1.sjc.aristanetworks.com
> ,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com
> %2C9104%2C1452811288618.default.1453728791276
> > 16/01/27 16:33:39 WARN BlockStateChange: BLOCK*
> > BlockInfoUnderConstruction.initLeaseRecovery: No blocks found, lease
> > removed.
>
> I ran dfsadmin fsck -move to make sure that all the files that had lost
> blocks were moved to /lost+found, and this obviously didn’t help HBase,
> because as I stated earlier, only one WAL had lost a block, and 94% of the
> blocks lost affected the HFile of one of the regions.
>
> Yet, somehow, the error above appeared for every single one of the region
> servers, and I ended up having to move more WAL files manually to
> /lost+found:
>
> foo@r12s3:~/hadoop-2.7.1$ ./bin/hdfs dfs -ls /lost+found
> Found 15 items
> drwxr--r--   - foo supergroup          0 2016-01-28 05:56 /lost+found/hbase
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:33 /lost+found/
> r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:29 /lost+found/
> r12s10.sjc.aristanetworks.com%2C9104%2C1452811286704.default.1453728581434
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:31 /lost+found/
> r12s11.sjc.aristanetworks.com%2C9104%2C1452811286222.default.1453728710303
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
> r12s13.sjc.aristanetworks.com%2C9104%2C1452811287287.default.1453728621698
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
> r12s14.sjc.aristanetworks.com%2C9104%2C1452811286288.default.1453728336644
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
> r12s15.sjc.aristanetworks.com%2C9104%2C1453158959800.default.1453728342559
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
> r12s16.sjc.aristanetworks.com%2C9104%2C1452811286456.default.1453728374800
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
> r12s2.sjc.aristanetworks.com%2C9104%2C1452811286448.default.1453728137282
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
> r12s3.sjc.aristanetworks.com%2C9104%2C1452811286093.default.1453728393926
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:35 /lost+found/
> r12s4.sjc.aristanetworks.com%2C9104%2C1452811289547.default.1453728949397
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
> r12s5.sjc.aristanetworks.com%2C9104%2C1452811125084.default.1453728624262
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
> r12s6.sjc.aristanetworks.com%2C9104%2C1452811286154.default.1453728483550
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
> r12s7.sjc.aristanetworks.com%2C9104%2C1452811287528.default.1453728528180
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
> r12s8.sjc.aristanetworks.com%2C9104%2C1452811287196.default.1453728125912
>
> After doing this and restarting the HMaster, everything came back up
> fine.  I don’t know if doing this caused any additional data loss – this is
> a dev cluster so data loss isn’t a big deal, but if I was to run into this
> issue in production, I would certainly be very nervous about this whole
> situation.
>
> This might turn more into an HDFS question at this point, so I’m Cc’ing
> hdfs-user@ just in case anybody has anything to say there.
>
> We’re going to upgrade to Hadoop 2.7.2 soon, just in case.
>

-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by tsuna <ts...@gmail.com>.
Resending to user@hadoop.apache.org now that I’m subscribed to that list.

On Thu, Jan 28, 2016 at 10:50 AM, tsuna <ts...@gmail.com> wrote:

> Just to close the loop on this ordeal…
>
> I started by clearing /hbase/splitWAL in ZK and restarting all the RS and
> the HM.  This didn’t change anything.
>
> On Wed, Jan 27, 2016 at 8:42 AM, tsuna <ts...@gmail.com> wrote:
> > 16/01/27 16:33:39 INFO namenode.FSNamesystem: Recovering [Lease.
> > Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
> > src=/hbase/WALs/r12s1.sjc.aristanetworks.com
> ,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com
> %2C9104%2C1452811288618.default.1453728791276
> > 16/01/27 16:33:39 WARN BlockStateChange: BLOCK*
> > BlockInfoUnderConstruction.initLeaseRecovery: No blocks found, lease
> > removed.
>
> I ran dfsadmin fsck -move to make sure that all the files that had lost
> blocks were moved to /lost+found, and this obviously didn’t help HBase,
> because as I stated earlier, only one WAL had lost a block, and 94% of the
> blocks lost affected the HFile of one of the regions.
>
> Yet, somehow, the error above appeared for every single one of the region
> servers, and I ended up having to move more WAL files manually to
> /lost+found:
>
> foo@r12s3:~/hadoop-2.7.1$ ./bin/hdfs dfs -ls /lost+found
> Found 15 items
> drwxr--r--   - foo supergroup          0 2016-01-28 05:56 /lost+found/hbase
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:33 /lost+found/
> r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:29 /lost+found/
> r12s10.sjc.aristanetworks.com%2C9104%2C1452811286704.default.1453728581434
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:31 /lost+found/
> r12s11.sjc.aristanetworks.com%2C9104%2C1452811286222.default.1453728710303
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
> r12s13.sjc.aristanetworks.com%2C9104%2C1452811287287.default.1453728621698
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
> r12s14.sjc.aristanetworks.com%2C9104%2C1452811286288.default.1453728336644
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
> r12s15.sjc.aristanetworks.com%2C9104%2C1453158959800.default.1453728342559
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
> r12s16.sjc.aristanetworks.com%2C9104%2C1452811286456.default.1453728374800
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
> r12s2.sjc.aristanetworks.com%2C9104%2C1452811286448.default.1453728137282
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
> r12s3.sjc.aristanetworks.com%2C9104%2C1452811286093.default.1453728393926
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:35 /lost+found/
> r12s4.sjc.aristanetworks.com%2C9104%2C1452811289547.default.1453728949397
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
> r12s5.sjc.aristanetworks.com%2C9104%2C1452811125084.default.1453728624262
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
> r12s6.sjc.aristanetworks.com%2C9104%2C1452811286154.default.1453728483550
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
> r12s7.sjc.aristanetworks.com%2C9104%2C1452811287528.default.1453728528180
> -rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
> r12s8.sjc.aristanetworks.com%2C9104%2C1452811287196.default.1453728125912
>
> After doing this and restarting the HMaster, everything came back up
> fine.  I don’t know if doing this caused any additional data loss – this is
> a dev cluster so data loss isn’t a big deal, but if I was to run into this
> issue in production, I would certainly be very nervous about this whole
> situation.
>
> This might turn more into an HDFS question at this point, so I’m Cc’ing
> hdfs-user@ just in case anybody has anything to say there.
>
> We’re going to upgrade to Hadoop 2.7.2 soon, just in case.
>

-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by tsuna <ts...@gmail.com>.
Just to close the loop on this ordeal…

I started by clearing /hbase/splitWAL in ZK and restarting all the RS and
the HM.  This didn’t change anything.

On Wed, Jan 27, 2016 at 8:42 AM, tsuna <ts...@gmail.com> wrote:
> 16/01/27 16:33:39 INFO namenode.FSNamesystem: Recovering [Lease.
> Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
> src=/hbase/WALs/r12s1.sjc.aristanetworks.com,9104,1452811288618-splitting/
r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
> 16/01/27 16:33:39 WARN BlockStateChange: BLOCK*
> BlockInfoUnderConstruction.initLeaseRecovery: No blocks found, lease
> removed.

I ran dfsadmin fsck -move to make sure that all the files that had lost
blocks were moved to /lost+found, and this obviously didn’t help HBase,
because as I stated earlier, only one WAL had lost a block, and 94% of the
blocks lost affected the HFile of one of the regions.

Yet, somehow, the error above appeared for every single one of the region
servers, and I ended up having to move more WAL files manually to
/lost+found:

foo@r12s3:~/hadoop-2.7.1$ ./bin/hdfs dfs -ls /lost+found
Found 15 items
drwxr--r--   - foo supergroup          0 2016-01-28 05:56 /lost+found/hbase
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:33 /lost+found/
r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:29 /lost+found/
r12s10.sjc.aristanetworks.com%2C9104%2C1452811286704.default.1453728581434
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:31 /lost+found/
r12s11.sjc.aristanetworks.com%2C9104%2C1452811286222.default.1453728710303
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
r12s13.sjc.aristanetworks.com%2C9104%2C1452811287287.default.1453728621698
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
r12s14.sjc.aristanetworks.com%2C9104%2C1452811286288.default.1453728336644
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:25 /lost+found/
r12s15.sjc.aristanetworks.com%2C9104%2C1453158959800.default.1453728342559
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
r12s16.sjc.aristanetworks.com%2C9104%2C1452811286456.default.1453728374800
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
r12s2.sjc.aristanetworks.com%2C9104%2C1452811286448.default.1453728137282
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:26 /lost+found/
r12s3.sjc.aristanetworks.com%2C9104%2C1452811286093.default.1453728393926
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:35 /lost+found/
r12s4.sjc.aristanetworks.com%2C9104%2C1452811289547.default.1453728949397
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:30 /lost+found/
r12s5.sjc.aristanetworks.com%2C9104%2C1452811125084.default.1453728624262
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
r12s6.sjc.aristanetworks.com%2C9104%2C1452811286154.default.1453728483550
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:28 /lost+found/
r12s7.sjc.aristanetworks.com%2C9104%2C1452811287528.default.1453728528180
-rw-r--r--   3 foo supergroup         83 2016-01-25 13:22 /lost+found/
r12s8.sjc.aristanetworks.com%2C9104%2C1452811287196.default.1453728125912

After doing this and restarting the HMaster, everything came back up fine.
I don’t know if doing this caused any additional data loss – this is a dev
cluster so data loss isn’t a big deal, but if I was to run into this issue
in production, I would certainly be very nervous about this whole situation.

This might turn more into an HDFS question at this point, so I’m Cc’ing
hdfs-user@ just in case anybody has anything to say there.

We’re going to upgrade to Hadoop 2.7.2 soon, just in case.

-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by Stack <st...@duboce.net>.
I need to study better but it looks like stuck on the NN trying to recover
lease. The WAL is missing blocks and NN lease recovery is skipping out.
St.Ack

On Wed, Jan 27, 2016 at 12:44 PM, tsuna <ts...@gmail.com> wrote:

> Some NN logs attached.
>
> On Wed, Jan 27, 2016 at 8:42 AM, tsuna <ts...@gmail.com> wrote:
> > On Wed, Jan 27, 2016 at 3:31 AM, Stack <st...@duboce.net> wrote:
> >> Can you try what Samir suggests? Can replay moved-aside logs using
> >> WALPlayer after cluster comes up.
> >
> > OK I’ll shut down the HM and all RS’es and clear the data in ZK and
> > restart everything and see how things go.
> >
> >> I see that at first we are making progress through the WALs ("... total
> >> tasks = 7..." ... it was 9 in your master log) but maybe they are
> getting
> >> stuck on particular WALs.  The stuck Master has same task list over
> last 24
> >> hours?
> >
> > It’s progressing veeerrryyyy slooowly.  I now have 34 regions online
> > (that’s 3 more than 10h ago! :P)
> >
> >> On the RS log, I see you are stuck trying to recover 1453728374800
> >>
> >> We are trying to recover the lease on the file from NN. Below looks bad.
> >> Over a minute and still not lease recovery.
> >>
> >> 140 2016-01-27 05:53:10,338 INFO  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
> >> util.FSHDFSUtils: recoverLease=false, attempt=2 on file=hdfs://
> >>
> 0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1452811286456-splitting/r12s16.sjc.aristanetworks.com%2C910
> >>    4%2C1452811286456.default.1453728374800 after 68218ms
> >>
> >> NN showing us stuck on lease recovery?
> >>
> >> NN went down during power outage?
> >
> > Yes, everything went down.
> >
> > The NN regularly shows this:
> >
> > 16/01/27 16:33:39 INFO namenode.FSNamesystem: recoverLease: [Lease.
> > Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
> > src=/hbase/WALs/r12s1.sjc.aristanetworks.com
> ,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com
> %2C9104%2C1452811288618.default.1453728791276
> > from client DFSClient_NONMAPREDUCE_174538359_1
> > 16/01/27 16:33:39 INFO namenode.FSNamesystem: Recovering [Lease.
> > Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
> > src=/hbase/WALs/r12s1.sjc.aristanetworks.com
> ,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com
> %2C9104%2C1452811288618.default.1453728791276
> > 16/01/27 16:33:39 WARN BlockStateChange: BLOCK*
> > BlockInfoUnderConstruction.initLeaseRecovery: No blocks found, lease
> > removed.
> > 16/01/27 16:33:39 WARN hdfs.StateChange: DIR*
> > NameSystem.internalReleaseLease: File
> > /hbase/WALs/r12s1.sjc.aristanetworks.com,9104,1452811288618-splitting/
> r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
> > has not been closed. Lease recovery is in progress. RecoveryId =
> > 1122295 for block blk_1074847505_1108430{UCState=UNDER_RECOVERY,
> > truncateBlock=null, primaryNodeIndex=-1, replicas=[]}
> >
> > I should perhaps mention at this point that as a result of the power
> > outage we lost a bunch of hard disk drives, and this caused us to
> > permanently lose approximately 0.3% of the blocks in HDFS.  I know
> > that at least one of WALs was affected by data loss
> > (/hbase/WALs/r12s12.sjc.aristanetworks.com,9104,1452811287315-splitting/
> r12s12.sjc.aristanetworks.com
> %2C9104%2C1452811287315.default.1453724924825).
> > Might be worth mentioning also that 94% of the 0.3% blocks we lost are
> > all from one unlucky region.
> >
> >> The BadVersion is probably because task has been assumed by another
> because
> >> lease recovery taking too long.
> >>
> >> I see 1453728374800 is still in the list you paste from Master later in
> >> this thread.
> >>
> >> Try moving it aside. Might have to restart master for it to notice the
> >> move-aside... Try not restarting first.
> >
> > --
> > Benoit "tsuna" Sigoure
>
>
>
> --
> Benoit "tsuna" Sigoure
>

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by tsuna <ts...@gmail.com>.
Some NN logs attached.

On Wed, Jan 27, 2016 at 8:42 AM, tsuna <ts...@gmail.com> wrote:
> On Wed, Jan 27, 2016 at 3:31 AM, Stack <st...@duboce.net> wrote:
>> Can you try what Samir suggests? Can replay moved-aside logs using
>> WALPlayer after cluster comes up.
>
> OK I’ll shut down the HM and all RS’es and clear the data in ZK and
> restart everything and see how things go.
>
>> I see that at first we are making progress through the WALs ("... total
>> tasks = 7..." ... it was 9 in your master log) but maybe they are getting
>> stuck on particular WALs.  The stuck Master has same task list over last 24
>> hours?
>
> It’s progressing veeerrryyyy slooowly.  I now have 34 regions online
> (that’s 3 more than 10h ago! :P)
>
>> On the RS log, I see you are stuck trying to recover 1453728374800
>>
>> We are trying to recover the lease on the file from NN. Below looks bad.
>> Over a minute and still not lease recovery.
>>
>> 140 2016-01-27 05:53:10,338 INFO  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
>> util.FSHDFSUtils: recoverLease=false, attempt=2 on file=hdfs://
>> 0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1452811286456-splitting/r12s16.sjc.aristanetworks.com%2C910
>>    4%2C1452811286456.default.1453728374800 after 68218ms
>>
>> NN showing us stuck on lease recovery?
>>
>> NN went down during power outage?
>
> Yes, everything went down.
>
> The NN regularly shows this:
>
> 16/01/27 16:33:39 INFO namenode.FSNamesystem: recoverLease: [Lease.
> Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
> src=/hbase/WALs/r12s1.sjc.aristanetworks.com,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
> from client DFSClient_NONMAPREDUCE_174538359_1
> 16/01/27 16:33:39 INFO namenode.FSNamesystem: Recovering [Lease.
> Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
> src=/hbase/WALs/r12s1.sjc.aristanetworks.com,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
> 16/01/27 16:33:39 WARN BlockStateChange: BLOCK*
> BlockInfoUnderConstruction.initLeaseRecovery: No blocks found, lease
> removed.
> 16/01/27 16:33:39 WARN hdfs.StateChange: DIR*
> NameSystem.internalReleaseLease: File
> /hbase/WALs/r12s1.sjc.aristanetworks.com,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
> has not been closed. Lease recovery is in progress. RecoveryId =
> 1122295 for block blk_1074847505_1108430{UCState=UNDER_RECOVERY,
> truncateBlock=null, primaryNodeIndex=-1, replicas=[]}
>
> I should perhaps mention at this point that as a result of the power
> outage we lost a bunch of hard disk drives, and this caused us to
> permanently lose approximately 0.3% of the blocks in HDFS.  I know
> that at least one of WALs was affected by data loss
> (/hbase/WALs/r12s12.sjc.aristanetworks.com,9104,1452811287315-splitting/r12s12.sjc.aristanetworks.com%2C9104%2C1452811287315.default.1453724924825).
> Might be worth mentioning also that 94% of the 0.3% blocks we lost are
> all from one unlucky region.
>
>> The BadVersion is probably because task has been assumed by another because
>> lease recovery taking too long.
>>
>> I see 1453728374800 is still in the list you paste from Master later in
>> this thread.
>>
>> Try moving it aside. Might have to restart master for it to notice the
>> move-aside... Try not restarting first.
>
> --
> Benoit "tsuna" Sigoure



-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by tsuna <ts...@gmail.com>.
On Wed, Jan 27, 2016 at 3:31 AM, Stack <st...@duboce.net> wrote:
> Can you try what Samir suggests? Can replay moved-aside logs using
> WALPlayer after cluster comes up.

OK I’ll shut down the HM and all RS’es and clear the data in ZK and
restart everything and see how things go.

> I see that at first we are making progress through the WALs ("... total
> tasks = 7..." ... it was 9 in your master log) but maybe they are getting
> stuck on particular WALs.  The stuck Master has same task list over last 24
> hours?

It’s progressing veeerrryyyy slooowly.  I now have 34 regions online
(that’s 3 more than 10h ago! :P)

> On the RS log, I see you are stuck trying to recover 1453728374800
>
> We are trying to recover the lease on the file from NN. Below looks bad.
> Over a minute and still not lease recovery.
>
> 140 2016-01-27 05:53:10,338 INFO  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
> util.FSHDFSUtils: recoverLease=false, attempt=2 on file=hdfs://
> 0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1452811286456-splitting/r12s16.sjc.aristanetworks.com%2C910
>    4%2C1452811286456.default.1453728374800 after 68218ms
>
> NN showing us stuck on lease recovery?
>
> NN went down during power outage?

Yes, everything went down.

The NN regularly shows this:

16/01/27 16:33:39 INFO namenode.FSNamesystem: recoverLease: [Lease.
Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
src=/hbase/WALs/r12s1.sjc.aristanetworks.com,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
from client DFSClient_NONMAPREDUCE_174538359_1
16/01/27 16:33:39 INFO namenode.FSNamesystem: Recovering [Lease.
Holder: DFSClient_NONMAPREDUCE_174538359_1, pendingcreates: 2],
src=/hbase/WALs/r12s1.sjc.aristanetworks.com,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
16/01/27 16:33:39 WARN BlockStateChange: BLOCK*
BlockInfoUnderConstruction.initLeaseRecovery: No blocks found, lease
removed.
16/01/27 16:33:39 WARN hdfs.StateChange: DIR*
NameSystem.internalReleaseLease: File
/hbase/WALs/r12s1.sjc.aristanetworks.com,9104,1452811288618-splitting/r12s1.sjc.aristanetworks.com%2C9104%2C1452811288618.default.1453728791276
has not been closed. Lease recovery is in progress. RecoveryId =
1122295 for block blk_1074847505_1108430{UCState=UNDER_RECOVERY,
truncateBlock=null, primaryNodeIndex=-1, replicas=[]}

I should perhaps mention at this point that as a result of the power
outage we lost a bunch of hard disk drives, and this caused us to
permanently lose approximately 0.3% of the blocks in HDFS.  I know
that at least one of WALs was affected by data loss
(/hbase/WALs/r12s12.sjc.aristanetworks.com,9104,1452811287315-splitting/r12s12.sjc.aristanetworks.com%2C9104%2C1452811287315.default.1453724924825).
Might be worth mentioning also that 94% of the 0.3% blocks we lost are
all from one unlucky region.

> The BadVersion is probably because task has been assumed by another because
> lease recovery taking too long.
>
> I see 1453728374800 is still in the list you paste from Master later in
> this thread.
>
> Try moving it aside. Might have to restart master for it to notice the
> move-aside... Try not restarting first.

-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by Stack <st...@duboce.net>.
Can you try what Samir suggests? Can replay moved-aside logs using
WALPlayer after cluster comes up.

I see that at first we are making progress through the WALs ("... total
tasks = 7..." ... it was 9 in your master log) but maybe they are getting
stuck on particular WALs.  The stuck Master has same task list over last 24
hours?

On the RS log, I see you are stuck trying to recover 1453728374800

We are trying to recover the lease on the file from NN. Below looks bad.
Over a minute and still not lease recovery.

140 2016-01-27 05:53:10,338 INFO  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
util.FSHDFSUtils: recoverLease=false, attempt=2 on file=hdfs://
0.namenode.sjc.aristanetworks.com:8820/hbase/WALs/r12s16.sjc.aristanetworks.com,9104,1452811286456-splitting/r12s16.sjc.aristanetworks.com%2C910
   4%2C1452811286456.default.1453728374800 after 68218ms

NN showing us stuck on lease recovery?

NN went down during power outage?

The BadVersion is probably because task has been assumed by another because
lease recovery taking too long.

I see 1453728374800 is still in the list you paste from Master later in
this thread.

Try moving it aside. Might have to restart master for it to notice the
move-aside... Try not restarting first.

St.Ack









On Wed, Jan 27, 2016 at 2:02 AM, tsuna <ts...@gmail.com> wrote:

> Hi,
> after a planned power outage one of our HBase clusters isn’t coming back
> up healthy.  The master shows the 16 region servers but zero regions.  All
> the RegionServers are experiencing the same problem, which is that they’re
> getting a BadVersion error from ZooKeeper.  This was with HBase 1.1.2 and
> I just upgraded all the nodes to 1.1.3 to see if this would make a
> difference, but it didn’t.
>
> 2016-01-27 05:54:02,402 WARN  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
> coordination.ZkSplitLogWorkerCoordination: BADVERSION failed to assert
> ownership for /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
> %252C9104%252C1452811286456.default.1453728374800
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode
> = BadVersion for /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
> %252C9104%252C1452811286456.default.1453728374800
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:429)
> at
> org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.attemptToOwnTask(ZkSplitLogWorkerCoordination.java:370)
> at
> org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$1.progress(ZkSplitLogWorkerCoordination.java:304)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.checkIfCancelled(FSHDFSUtils.java:329)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:244)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:162)
> at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:761)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:235)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2016-01-27 05:54:02,404 WARN  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
> coordination.ZkSplitLogWorkerCoordination: Failed to heartbeat the
> task/hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
> %252C9104%252C1452811286456.default.1453728374800
>
> I’m attaching the full log of the RS this was extracted from, which I just
> restarted on 1.1.3, in case that’s of any help.
>
> I’ve never seen this before and after a bit of digging, I’m not really
> going anywhere.  Any ideas / suggestions?
>
> --
> Benoit "tsuna" Sigoure
>

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by Samir Ahmic <ah...@gmail.com>.
Hi,

This looks like some WALs splitting operations are not able to finish
causing master to abort. First thing i would try is to stop cluster, clear
zk data and then try to start it again. If that does not solve issue you
probably have case of corrupted WALs files. In that case you will have to
locate corrupted WALs files on hdfs and remove them in order to start
master.

Cheers
Samir

On Wed, Jan 27, 2016 at 7:26 AM, tsuna <ts...@gmail.com> wrote:

> BTW, I already left the cluster in this state for 24h, so I know it’s
> stuck for good.
>
> Since I just restarted everything about half an hour ago, I can tell you
> that I now have 31 regions online, and 278 stuck offline, and things aren’t
> making progress.  The master is just waiting for log splitting to happen,
> and it regularly dies and restarts because it can’t complete initialization.
>
> In case that’s of any help, I’m attaching the last 1000 lines of the
> master log, which show that the master is just waiting for stuff to happen
> and then committing suicide and restarting and waiting again.
>
> 2016-01-27 06:21:30,871 INFO  [r12s3.sjc.aristanetworks.com,9102,1453874320326_splitLogManager__ChoreService_1]
> master.SplitLogManager: total tasks = 9 unassigned = 0
> tasks={/hbase/splitWAL/WALs%2Fr12s6.sjc.aristanetworks.com
> %2C9104%2C1452811286154-splitting%2Fr12s6.sjc.aristanetworks.com%252C9104%252C1452811286154.default.1453728483550=last_update
> = 1453875668700 last_version = 30 cur_worker_name =
> r12s2.sjc.aristanetworks.com,9104,1453873919713 status = in_progress
> incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0,
> /hbase/splitWAL/WALs%2Fr12s8.sjc.aristanetworks.com
> %2C9104%2C1452811287196-splitting%2Fr12s8.sjc.aristanetworks.com%252C9104%252C1452811287196.default.1453728125912=last_update
> = 1453875650945 last_version = 29 cur_worker_name =
> r12s13.sjc.aristanetworks.com,9104,1453873919678 status = in_progress
> incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0,
> /hbase/splitWAL/WALs%2Fr12s14.sjc.aristanetworks.com
> %2C9104%2C1452811286288-splitting%2Fr12s14.sjc.aristanetworks.com%252C9104%252C1452811286288.default.1453728336644=last_update
> = 1453875673936 last_version = 30 cur_worker_name =
> r12s11.sjc.aristanetworks.com,9104,1453873919623 status = in_progress
> incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0,
> /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com%252C9104%252C1452811286456.default.1453728374800=last_update
> = 1453875666968 last_version = 3593 cur_worker_name =
> r12s10.sjc.aristanetworks.com,9104,1453873919762 status = in_progress
> incarnation = 2 resubmits = 2 batch = installed = 1 done = 0 error = 0,
> /hbase/splitWAL/WALs%2Fr12s13.sjc.aristanetworks.com
> %2C9104%2C1452811287287-splitting%2Fr12s13.sjc.aristanetworks.com%252C9104%252C1452811287287.default.1453728621698=last_update
> = 1453875666969 last_version = 3650 cur_worker_name =
> r12s15.sjc.aristanetworks.com,9104,1453873919845 status = in_progress
> incarnation = 2 resubmits = 2 batch = null, /hbase/splitWAL/WALs%
> 2Fr12s4.sjc.aristanetworks.com%2C9104%2C1452811289547-splitting%
> 2Fr12s4.sjc.aristanetworks.com%252C9104%252C1452811289547.default.1453728949397=last_update
> = 1453875666969 last_version = 3658 cur_worker_name =
> r12s9.sjc.aristanetworks.com,9104,1453873919799 status = in_progress
> incarnation = 1 resubmits = 1 batch = installed = 1 done = 0 error = 0,
> /hbase/splitWAL/WALs%2Fr12s5.sjc.aristanetworks.com
> %2C9104%2C1452811125084-splitting%2Fr12s5.sjc.aristanetworks.com%252C9104%252C1452811125084.default.1453728624262=last_update
> = 1453875666968 last_version = 3437 cur_worker_name =
> r12s1.sjc.aristanetworks.com,9104,1453873919645 status = in_progress
> incarnation = 2 resubmits = 2 batch = null, /hbase/splitWAL/WALs%
> 2Fr12s1.sjc.aristanetworks.com%2C9104%2C1452811288618-splitting%
> 2Fr12s1.sjc.aristanetworks.com%252C9104%252C1452811288618.default.1453728791276=last_update
> = 1453875666968 last_version = 2280 cur_worker_name =
> r12s14.sjc.aristanetworks.com,9104,1453873919711 status = in_progress
> incarnation = 2 resubmits = 1 batch = null, /hbase/splitWAL/WALs%
> 2Fr12s2.sjc.aristanetworks.com%2C9104%2C1452811286448-splitting%
> 2Fr12s2.sjc.aristanetworks.com%252C9104%252C1452811286448.default.1453728137282=last_update
> = 1453875666968 last_version = 2219 cur_worker_name =
> r12s4.sjc.aristanetworks.com,9104,1453873919688 status = in_progress
> incarnation = 2 resubmits = 1 batch = null}
>
> On Tue, Jan 26, 2016 at 10:02 PM, tsuna <ts...@gmail.com> wrote:
>
>> Hi,
>> after a planned power outage one of our HBase clusters isn’t coming back
>> up healthy.  The master shows the 16 region servers but zero regions.  All
>> the RegionServers are experiencing the same problem, which is that they’re
>> getting a BadVersion error from ZooKeeper.  This was with HBase 1.1.2
>> and I just upgraded all the nodes to 1.1.3 to see if this would make a
>> difference, but it didn’t.
>>
>> 2016-01-27 05:54:02,402 WARN  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
>> coordination.ZkSplitLogWorkerCoordination: BADVERSION failed to assert
>> ownership for /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
>> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
>> %252C9104%252C1452811286456.default.1453728374800
>> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode
>> = BadVersion for /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
>> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
>> %252C9104%252C1452811286456.default.1453728374800
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
>> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
>> at
>> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:429)
>> at
>> org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.attemptToOwnTask(ZkSplitLogWorkerCoordination.java:370)
>> at
>> org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$1.progress(ZkSplitLogWorkerCoordination.java:304)
>> at
>> org.apache.hadoop.hbase.util.FSHDFSUtils.checkIfCancelled(FSHDFSUtils.java:329)
>> at
>> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:244)
>> at
>> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:162)
>> at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:761)
>> at
>> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
>> at
>> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:235)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
>> at
>> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
>> at
>> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> at java.lang.Thread.run(Thread.java:745)
>> 2016-01-27 05:54:02,404 WARN  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
>> coordination.ZkSplitLogWorkerCoordination: Failed to heartbeat the
>> task/hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
>> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
>> %252C9104%252C1452811286456.default.1453728374800
>>
>> I’m attaching the full log of the RS this was extracted from, which I
>> just restarted on 1.1.3, in case that’s of any help.
>>
>> I’ve never seen this before and after a bit of digging, I’m not really
>> going anywhere.  Any ideas / suggestions?
>>
>
> --
> Benoit "tsuna" Sigoure
>

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by tsuna <ts...@gmail.com>.
BTW, I already left the cluster in this state for 24h, so I know it’s stuck
for good.

Since I just restarted everything about half an hour ago, I can tell you
that I now have 31 regions online, and 278 stuck offline, and things aren’t
making progress.  The master is just waiting for log splitting to happen,
and it regularly dies and restarts because it can’t complete initialization.

In case that’s of any help, I’m attaching the last 1000 lines of the master
log, which show that the master is just waiting for stuff to happen and
then committing suicide and restarting and waiting again.

2016-01-27 06:21:30,871 INFO
[r12s3.sjc.aristanetworks.com,9102,1453874320326_splitLogManager__ChoreService_1]
master.SplitLogManager: total tasks = 9 unassigned = 0
tasks={/hbase/splitWAL/WALs%2Fr12s6.sjc.aristanetworks.com
%2C9104%2C1452811286154-splitting%2Fr12s6.sjc.aristanetworks.com%252C9104%252C1452811286154.default.1453728483550=last_update
= 1453875668700 last_version = 30 cur_worker_name =
r12s2.sjc.aristanetworks.com,9104,1453873919713 status = in_progress
incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0,
/hbase/splitWAL/WALs%2Fr12s8.sjc.aristanetworks.com
%2C9104%2C1452811287196-splitting%2Fr12s8.sjc.aristanetworks.com%252C9104%252C1452811287196.default.1453728125912=last_update
= 1453875650945 last_version = 29 cur_worker_name =
r12s13.sjc.aristanetworks.com,9104,1453873919678 status = in_progress
incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0,
/hbase/splitWAL/WALs%2Fr12s14.sjc.aristanetworks.com
%2C9104%2C1452811286288-splitting%2Fr12s14.sjc.aristanetworks.com%252C9104%252C1452811286288.default.1453728336644=last_update
= 1453875673936 last_version = 30 cur_worker_name =
r12s11.sjc.aristanetworks.com,9104,1453873919623 status = in_progress
incarnation = 0 resubmits = 0 batch = installed = 1 done = 0 error = 0,
/hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
%2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com%252C9104%252C1452811286456.default.1453728374800=last_update
= 1453875666968 last_version = 3593 cur_worker_name =
r12s10.sjc.aristanetworks.com,9104,1453873919762 status = in_progress
incarnation = 2 resubmits = 2 batch = installed = 1 done = 0 error = 0,
/hbase/splitWAL/WALs%2Fr12s13.sjc.aristanetworks.com
%2C9104%2C1452811287287-splitting%2Fr12s13.sjc.aristanetworks.com%252C9104%252C1452811287287.default.1453728621698=last_update
= 1453875666969 last_version = 3650 cur_worker_name =
r12s15.sjc.aristanetworks.com,9104,1453873919845 status = in_progress
incarnation = 2 resubmits = 2 batch = null, /hbase/splitWAL/WALs%
2Fr12s4.sjc.aristanetworks.com%2C9104%2C1452811289547-splitting%
2Fr12s4.sjc.aristanetworks.com%252C9104%252C1452811289547.default.1453728949397=last_update
= 1453875666969 last_version = 3658 cur_worker_name =
r12s9.sjc.aristanetworks.com,9104,1453873919799 status = in_progress
incarnation = 1 resubmits = 1 batch = installed = 1 done = 0 error = 0,
/hbase/splitWAL/WALs%2Fr12s5.sjc.aristanetworks.com
%2C9104%2C1452811125084-splitting%2Fr12s5.sjc.aristanetworks.com%252C9104%252C1452811125084.default.1453728624262=last_update
= 1453875666968 last_version = 3437 cur_worker_name =
r12s1.sjc.aristanetworks.com,9104,1453873919645 status = in_progress
incarnation = 2 resubmits = 2 batch = null, /hbase/splitWAL/WALs%
2Fr12s1.sjc.aristanetworks.com%2C9104%2C1452811288618-splitting%
2Fr12s1.sjc.aristanetworks.com%252C9104%252C1452811288618.default.1453728791276=last_update
= 1453875666968 last_version = 2280 cur_worker_name =
r12s14.sjc.aristanetworks.com,9104,1453873919711 status = in_progress
incarnation = 2 resubmits = 1 batch = null, /hbase/splitWAL/WALs%
2Fr12s2.sjc.aristanetworks.com%2C9104%2C1452811286448-splitting%
2Fr12s2.sjc.aristanetworks.com%252C9104%252C1452811286448.default.1453728137282=last_update
= 1453875666968 last_version = 2219 cur_worker_name =
r12s4.sjc.aristanetworks.com,9104,1453873919688 status = in_progress
incarnation = 2 resubmits = 1 batch = null}

On Tue, Jan 26, 2016 at 10:02 PM, tsuna <ts...@gmail.com> wrote:

> Hi,
> after a planned power outage one of our HBase clusters isn’t coming back
> up healthy.  The master shows the 16 region servers but zero regions.  All
> the RegionServers are experiencing the same problem, which is that they’re
> getting a BadVersion error from ZooKeeper.  This was with HBase 1.1.2 and
> I just upgraded all the nodes to 1.1.3 to see if this would make a
> difference, but it didn’t.
>
> 2016-01-27 05:54:02,402 WARN  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
> coordination.ZkSplitLogWorkerCoordination: BADVERSION failed to assert
> ownership for /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
> %252C9104%252C1452811286456.default.1453728374800
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode
> = BadVersion for /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
> %252C9104%252C1452811286456.default.1453728374800
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:429)
> at
> org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.attemptToOwnTask(ZkSplitLogWorkerCoordination.java:370)
> at
> org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$1.progress(ZkSplitLogWorkerCoordination.java:304)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.checkIfCancelled(FSHDFSUtils.java:329)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:244)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:162)
> at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:761)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:235)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2016-01-27 05:54:02,404 WARN  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
> coordination.ZkSplitLogWorkerCoordination: Failed to heartbeat the
> task/hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
> %252C9104%252C1452811286456.default.1453728374800
>
> I’m attaching the full log of the RS this was extracted from, which I just
> restarted on 1.1.3, in case that’s of any help.
>
> I’ve never seen this before and after a bit of digging, I’m not really
> going anywhere.  Any ideas / suggestions?
>

-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by tsuna <ts...@gmail.com>.
On Wed, Jan 27, 2016 at 2:21 AM, Ted Yu <yu...@gmail.com> wrote:
> Can you check your zookeeper quorum to see if there was some problem ?

ZooKeeper is working fine, we also have Kafka working with it.  The ZK
logs are just full of these BadVersion errors, e.g.:

core@r12s5 ~ $ docker logs --tail=1 zookeeper
2016-01-27 16:23:30,778 [myid:2] - INFO  [ProcessThread(sid:2
cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException
when processing sessionid:0x3527c36549f0279 type:setData cxid:0x11ef
zxid:0x20000d414 txntype:-1 reqpath:n/a Error
Path:/hbase/splitWAL/WALs%2Fr12s2.sjc.aristanetworks.com%2C9104%2C1452811286448-splitting%2Fr12s2.sjc.aristanetworks.com%252C9104%252C1452811286448.default.1453728137282
Error:KeeperErrorCode = BadVersion for
/hbase/splitWAL/WALs%2Fr12s2.sjc.aristanetworks.com%2C9104%2C1452811286448-splitting%2Fr12s2.sjc.aristanetworks.com%252C9104%252C1452811286448.default.1453728137282

-- 
Benoit "tsuna" Sigoure

Re: All RegionServers stuck on BadVersion from ZK after cluster restart

Posted by Ted Yu <yu...@gmail.com>.
Shortly before BadVersion occurred, I saw:

2016-01-27 05:52:36,048 INFO  [main-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r12s8.sjc.aristanetworks.com,9104,1453785783387
znode expired, triggering        replicatorRemoved event
2016-01-27 05:52:36,051 INFO  [main-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r12s3.sjc.aristanetworks.com,9104,1453785739822
znode expired, triggering        replicatorRemoved event
2016-01-27 05:52:40,028 INFO  [main-EventThread]
replication.ReplicationTrackerZKImpl:
/hbase/rs/r12s7.sjc.aristanetworks.com,9104,1453785743694
znode expired, triggering        replicatorRemoved event

Can you check your zookeeper quorum to see if there was some problem ?

Thanks

On Tue, Jan 26, 2016 at 10:02 PM, tsuna <ts...@gmail.com> wrote:

> Hi,
> after a planned power outage one of our HBase clusters isn’t coming back
> up healthy.  The master shows the 16 region servers but zero regions.  All
> the RegionServers are experiencing the same problem, which is that they’re
> getting a BadVersion error from ZooKeeper.  This was with HBase 1.1.2 and
> I just upgraded all the nodes to 1.1.3 to see if this would make a
> difference, but it didn’t.
>
> 2016-01-27 05:54:02,402 WARN  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
> coordination.ZkSplitLogWorkerCoordination: BADVERSION failed to assert
> ownership for /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
> %252C9104%252C1452811286456.default.1453728374800
> org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode
> = BadVersion for /hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
> %252C9104%252C1452811286456.default.1453728374800
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:115)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.setData(ZooKeeper.java:1270)
> at
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:429)
> at
> org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination.attemptToOwnTask(ZkSplitLogWorkerCoordination.java:370)
> at
> org.apache.hadoop.hbase.coordination.ZkSplitLogWorkerCoordination$1.progress(ZkSplitLogWorkerCoordination.java:304)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.checkIfCancelled(FSHDFSUtils.java:329)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:244)
> at
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:162)
> at org.apache.hadoop.hbase.wal.WALSplitter.getReader(WALSplitter.java:761)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:297)
> at
> org.apache.hadoop.hbase.wal.WALSplitter.splitLogFile(WALSplitter.java:235)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:104)
> at
> org.apache.hadoop.hbase.regionserver.handler.WALSplitterHandler.process(WALSplitterHandler.java:72)
> at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:129)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2016-01-27 05:54:02,404 WARN  [RS_LOG_REPLAY_OPS-r12s4:9104-0]
> coordination.ZkSplitLogWorkerCoordination: Failed to heartbeat the
> task/hbase/splitWAL/WALs%2Fr12s16.sjc.aristanetworks.com
> %2C9104%2C1452811286456-splitting%2Fr12s16.sjc.aristanetworks.com
> %252C9104%252C1452811286456.default.1453728374800
>
> I’m attaching the full log of the RS this was extracted from, which I just
> restarted on 1.1.3, in case that’s of any help.
>
> I’ve never seen this before and after a bit of digging, I’m not really
> going anywhere.  Any ideas / suggestions?
>
> --
> Benoit "tsuna" Sigoure
>