You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by Jean-Marc Spaggiari <je...@spaggiari.org> on 2013/08/29 18:03:22 UTC
Never ending "Doing distributed log split" task.,
I have restart my cluster and I'm now waiting for this task to end:
Doing distributed log split in
[hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting]
It's running fir now 30 minutes. There was nothing running on the cluster.
No reads, no writes, nothing, for days...
I got that on the logs:
2013-08-29 11:36:10,862 WARN
org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting/node1%2C60020%2C1377789460683.1377789462024
interrupted, resigning
java.io.InterruptedIOException
at
org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:136)
at
org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:54)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:780)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:414)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:381)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:112)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:280)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.InterruptedException: sleep interrupted
at java.lang.Thread.sleep(Native Method)
at
org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:118)
... 9 more
2013-08-29 11:36:10,950 WARN
org.apache.hadoop.hbase.regionserver.SplitLogWorker: Interrupted while
trying to assert ownership of
/hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1377789460683-splitting%2Fnode1%252C60020%252C1377789460683.1377789462024
java.lang.InterruptedException
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1160)
at
org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:361)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.attemptToOwnTask(SplitLogWorker.java:346)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
at java.lang.Thread.run(Thread.java:722)
I'm not 100% what is causing that. I have restarted it and still getting
the same result.
Any hint?
Thanks,
JM
Re: Never ending "Doing distributed log split" task.,
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
I have not yet found an easy way to upgrade to hadoop 1.2.1 so I'm waiting
for my 2nd cluster to be ready to install it with 1.2.1 and distcp to it.
But that's outside of the scope of this discussion ;)
2013/8/29 Ted Yu <yu...@gmail.com>
> So you have HBASE-8670 in your deployment.
>
> Suggest upgrading hadoop to newer release, e.g. 1.2.1 so that the new HDFS
> improvements can be utilized.
>
> Cheers
>
>
> On Thu, Aug 29, 2013 at 9:50 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > Hadoop 1.0.4 with HBase 0.94.12-SNAPSHOT
> >
> > The file name changed since I have restarted HBase but here is what I
> have:
> > hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls
> > hdfs://node3:9000/hbase/.logs/node1,60020,1377793020654/
> > Found 1 items
> > -rw-r--r-- 3 hbase supergroup 0 2013-08-29 12:17
> >
> >
> /hbase/.logs/node1,60020,1377793020654/node1%2C60020%2C1377793020654.1377793021892
> >
> > And I'm able to access it:
> > hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -get
> >
> >
> /hbase/.logs/node1,60020,1377793020654/node1%2C60020%2C1377793020654.1377793021892
> > .
> > hadoop@node3:~/hadoop-1.0.3$
> >
> > Oh. I just checked the UI again, and it's done. Wow! Took almost 1h. HBCK
> > report 0 inconsistencies detected. Status: OK
> >
> > So seems that I'm all fine.
> >
> > I don't know why it was so long. I will try to take a look at my
> Ganglia's
> > metrics to see if I can figure anything...
> >
> > JM
> >
> >
> >
> > 2013/8/29 Ted Yu <yu...@gmail.com>
> >
> > > What is your HBase / Hadoop version ?
> > >
> > > Can you check namenode log looking for lines related to
> > > hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-
> > > splitting/node1%2C60020%2C1377789460683.1377789462024 ?
> > >
> > > Thanks
> > >
> > >
> > > On Thu, Aug 29, 2013 at 9:03 AM, Jean-Marc Spaggiari <
> > > jean-marc@spaggiari.org> wrote:
> > >
> > > > I have restart my cluster and I'm now waiting for this task to end:
> > > >
> > > > Doing distributed log split in
> > > > [hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting]
> > > >
> > > > It's running fir now 30 minutes. There was nothing running on the
> > > cluster.
> > > > No reads, no writes, nothing, for days...
> > > >
> > > > I got that on the logs:
> > > >
> > > > 2013-08-29 11:36:10,862 WARN
> > > > org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
> > > >
> > > >
> > >
> >
> hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting/node1%2C60020%2C1377789460683.1377789462024
> > > > interrupted, resigning
> > > > java.io.InterruptedIOException
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:136)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:54)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:780)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:414)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:381)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:112)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:280)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
> > > > at java.lang.Thread.run(Thread.java:722)
> > > > Caused by: java.lang.InterruptedException: sleep interrupted
> > > > at java.lang.Thread.sleep(Native Method)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:118)
> > > > ... 9 more
> > > > 2013-08-29 11:36:10,950 WARN
> > > > org.apache.hadoop.hbase.regionserver.SplitLogWorker: Interrupted
> while
> > > > trying to assert ownership of
> > > >
> > > >
> > >
> >
> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1377789460683-splitting%2Fnode1%252C60020%252C1377789460683.1377789462024
> > > > java.lang.InterruptedException
> > > > at java.lang.Object.wait(Native Method)
> > > > at java.lang.Object.wait(Object.java:503)
> > > > at
> > > org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
> > > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
> > > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1160)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:361)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.attemptToOwnTask(SplitLogWorker.java:346)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
> > > > at
> > > >
> > > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
> > > > at java.lang.Thread.run(Thread.java:722)
> > > >
> > > >
> > > > I'm not 100% what is causing that. I have restarted it and still
> > getting
> > > > the same result.
> > > >
> > > > Any hint?
> > > >
> > > > Thanks,
> > > >
> > > > JM
> > > >
> > >
> >
>
Re: Never ending "Doing distributed log split" task.,
Posted by Ted Yu <yu...@gmail.com>.
So you have HBASE-8670 in your deployment.
Suggest upgrading hadoop to newer release, e.g. 1.2.1 so that the new HDFS
improvements can be utilized.
Cheers
On Thu, Aug 29, 2013 at 9:50 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> Hadoop 1.0.4 with HBase 0.94.12-SNAPSHOT
>
> The file name changed since I have restarted HBase but here is what I have:
> hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls
> hdfs://node3:9000/hbase/.logs/node1,60020,1377793020654/
> Found 1 items
> -rw-r--r-- 3 hbase supergroup 0 2013-08-29 12:17
>
> /hbase/.logs/node1,60020,1377793020654/node1%2C60020%2C1377793020654.1377793021892
>
> And I'm able to access it:
> hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -get
>
> /hbase/.logs/node1,60020,1377793020654/node1%2C60020%2C1377793020654.1377793021892
> .
> hadoop@node3:~/hadoop-1.0.3$
>
> Oh. I just checked the UI again, and it's done. Wow! Took almost 1h. HBCK
> report 0 inconsistencies detected. Status: OK
>
> So seems that I'm all fine.
>
> I don't know why it was so long. I will try to take a look at my Ganglia's
> metrics to see if I can figure anything...
>
> JM
>
>
>
> 2013/8/29 Ted Yu <yu...@gmail.com>
>
> > What is your HBase / Hadoop version ?
> >
> > Can you check namenode log looking for lines related to
> > hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-
> > splitting/node1%2C60020%2C1377789460683.1377789462024 ?
> >
> > Thanks
> >
> >
> > On Thu, Aug 29, 2013 at 9:03 AM, Jean-Marc Spaggiari <
> > jean-marc@spaggiari.org> wrote:
> >
> > > I have restart my cluster and I'm now waiting for this task to end:
> > >
> > > Doing distributed log split in
> > > [hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting]
> > >
> > > It's running fir now 30 minutes. There was nothing running on the
> > cluster.
> > > No reads, no writes, nothing, for days...
> > >
> > > I got that on the logs:
> > >
> > > 2013-08-29 11:36:10,862 WARN
> > > org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
> > >
> > >
> >
> hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting/node1%2C60020%2C1377789460683.1377789462024
> > > interrupted, resigning
> > > java.io.InterruptedIOException
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:136)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:54)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:780)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:414)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:381)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:112)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:280)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
> > > at java.lang.Thread.run(Thread.java:722)
> > > Caused by: java.lang.InterruptedException: sleep interrupted
> > > at java.lang.Thread.sleep(Native Method)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:118)
> > > ... 9 more
> > > 2013-08-29 11:36:10,950 WARN
> > > org.apache.hadoop.hbase.regionserver.SplitLogWorker: Interrupted while
> > > trying to assert ownership of
> > >
> > >
> >
> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1377789460683-splitting%2Fnode1%252C60020%252C1377789460683.1377789462024
> > > java.lang.InterruptedException
> > > at java.lang.Object.wait(Native Method)
> > > at java.lang.Object.wait(Object.java:503)
> > > at
> > org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
> > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
> > > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1160)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:361)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.attemptToOwnTask(SplitLogWorker.java:346)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
> > > at
> > >
> > >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
> > > at java.lang.Thread.run(Thread.java:722)
> > >
> > >
> > > I'm not 100% what is causing that. I have restarted it and still
> getting
> > > the same result.
> > >
> > > Any hint?
> > >
> > > Thanks,
> > >
> > > JM
> > >
> >
>
Re: Never ending "Doing distributed log split" task.,
Posted by Jean-Marc Spaggiari <je...@spaggiari.org>.
Hadoop 1.0.4 with HBase 0.94.12-SNAPSHOT
The file name changed since I have restarted HBase but here is what I have:
hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -ls
hdfs://node3:9000/hbase/.logs/node1,60020,1377793020654/
Found 1 items
-rw-r--r-- 3 hbase supergroup 0 2013-08-29 12:17
/hbase/.logs/node1,60020,1377793020654/node1%2C60020%2C1377793020654.1377793021892
And I'm able to access it:
hadoop@node3:~/hadoop-1.0.3$ bin/hadoop fs -get
/hbase/.logs/node1,60020,1377793020654/node1%2C60020%2C1377793020654.1377793021892
.
hadoop@node3:~/hadoop-1.0.3$
Oh. I just checked the UI again, and it's done. Wow! Took almost 1h. HBCK
report 0 inconsistencies detected. Status: OK
So seems that I'm all fine.
I don't know why it was so long. I will try to take a look at my Ganglia's
metrics to see if I can figure anything...
JM
2013/8/29 Ted Yu <yu...@gmail.com>
> What is your HBase / Hadoop version ?
>
> Can you check namenode log looking for lines related to
> hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-
> splitting/node1%2C60020%2C1377789460683.1377789462024 ?
>
> Thanks
>
>
> On Thu, Aug 29, 2013 at 9:03 AM, Jean-Marc Spaggiari <
> jean-marc@spaggiari.org> wrote:
>
> > I have restart my cluster and I'm now waiting for this task to end:
> >
> > Doing distributed log split in
> > [hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting]
> >
> > It's running fir now 30 minutes. There was nothing running on the
> cluster.
> > No reads, no writes, nothing, for days...
> >
> > I got that on the logs:
> >
> > 2013-08-29 11:36:10,862 WARN
> > org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
> >
> >
> hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting/node1%2C60020%2C1377789460683.1377789462024
> > interrupted, resigning
> > java.io.InterruptedIOException
> > at
> >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:136)
> > at
> >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:54)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:780)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:414)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:381)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:112)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:280)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
> > at java.lang.Thread.run(Thread.java:722)
> > Caused by: java.lang.InterruptedException: sleep interrupted
> > at java.lang.Thread.sleep(Native Method)
> > at
> >
> >
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:118)
> > ... 9 more
> > 2013-08-29 11:36:10,950 WARN
> > org.apache.hadoop.hbase.regionserver.SplitLogWorker: Interrupted while
> > trying to assert ownership of
> >
> >
> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1377789460683-splitting%2Fnode1%252C60020%252C1377789460683.1377789462024
> > java.lang.InterruptedException
> > at java.lang.Object.wait(Native Method)
> > at java.lang.Object.wait(Object.java:503)
> > at
> org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
> > at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1160)
> > at
> >
> >
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:361)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.attemptToOwnTask(SplitLogWorker.java:346)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
> > at java.lang.Thread.run(Thread.java:722)
> >
> >
> > I'm not 100% what is causing that. I have restarted it and still getting
> > the same result.
> >
> > Any hint?
> >
> > Thanks,
> >
> > JM
> >
>
Re: Never ending "Doing distributed log split" task.,
Posted by Ted Yu <yu...@gmail.com>.
What is your HBase / Hadoop version ?
Can you check namenode log looking for lines related to
hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-
splitting/node1%2C60020%2C1377789460683.1377789462024 ?
Thanks
On Thu, Aug 29, 2013 at 9:03 AM, Jean-Marc Spaggiari <
jean-marc@spaggiari.org> wrote:
> I have restart my cluster and I'm now waiting for this task to end:
>
> Doing distributed log split in
> [hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting]
>
> It's running fir now 30 minutes. There was nothing running on the cluster.
> No reads, no writes, nothing, for days...
>
> I got that on the logs:
>
> 2013-08-29 11:36:10,862 WARN
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
>
> hdfs://node3:9000/hbase/.logs/node1,60020,1377789460683-splitting/node1%2C60020%2C1377789460683.1377789462024
> interrupted, resigning
> java.io.InterruptedIOException
> at
>
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:136)
> at
>
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverFileLease(FSHDFSUtils.java:54)
> at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:780)
> at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:414)
> at
>
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:381)
> at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:112)
> at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:280)
> at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
> at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.InterruptedException: sleep interrupted
> at java.lang.Thread.sleep(Native Method)
> at
>
> org.apache.hadoop.hbase.util.FSHDFSUtils.recoverDFSFileLease(FSHDFSUtils.java:118)
> ... 9 more
> 2013-08-29 11:36:10,950 WARN
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: Interrupted while
> trying to assert ownership of
>
> /hbase/splitlog/hdfs%3A%2F%2Fnode3%3A9000%2Fhbase%2F.logs%2Fnode1%2C60020%2C1377789460683-splitting%2Fnode1%252C60020%252C1377789460683.1377789462024
> java.lang.InterruptedException
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:503)
> at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1253)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1129)
> at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1160)
> at
>
> org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.setData(RecoverableZooKeeper.java:361)
> at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.attemptToOwnTask(SplitLogWorker.java:346)
> at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:264)
> at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:211)
> at
>
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:179)
> at java.lang.Thread.run(Thread.java:722)
>
>
> I'm not 100% what is causing that. I have restarted it and still getting
> the same result.
>
> Any hint?
>
> Thanks,
>
> JM
>