You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by kun yan <ya...@gmail.com> on 2013/09/26 04:31:10 UTC
SplitLogManager issue
I use importtsv import data into HDFS, but during the power outage.
Horrible, and then I re-import the data.(hbase 0.94)
the exception
org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find
region for data2,04684015,99999999999999 after 10 tries.
I look HMaster logs as follows:
2013-09-26 10:21:17,874 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
= 0
2013-09-26 10:21:18,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
= 0
2013-09-26 10:21:19,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
= 0
2013-09-26 10:21:20,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
= 0
2013-09-26 10:21:21,876 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
= 0
2013-09-26 10:21:22,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
= 0
2013-09-26 10:21:23,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
= 0
2013-09-26 10:21:24,875 DEBUG
org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
= 0
2013-09-26 10:21:25,436 INFO
org.apache.hadoop.hbase.master.SplitLogManager: task
/hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
entered state err hydra0004,60020,1380159614688
2013-09-26 10:21:25,436 WARN
org.apache.hadoop.hbase.master.SplitLogManager: Error splitting
/hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
2013-09-26 10:21:25,436 WARN
org.apache.hadoop.hbase.master.SplitLogManager: error while splitting logs
in
[hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
installed = 2 but only 0 done
2013-09-26 10:21:25,436 WARN
org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting of
[hydra0003,60020,1379926447350, hydra0004,60020,1379926440171,
hydra0006,60020,1379926437471]
java.io.IOException: error or interrupted while splitting logs in
[hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
Task = installed = 2 done = 0 error = 2
at
org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:282)
at
org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:300)
at
org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:242)
at
org.apache.hadoop.hbase.master.HMaster.splitLogAfterStartup(HMaster.java:661)
at
org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:580)
at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:396)
at java.lang.Thread.run(Thread.java:722)
--
In the Hadoop world, I am just a novice, explore the entire Hadoop
ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com
Re: SplitLogManager issue
Posted by Julian Zhou <ju...@me.com>.
Seems that I also met a similar issue complaining about this:
org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to
find region for *t_system_rec,,99999999999999* after 10 tries.
at
org.apache.hadoop.hbase.client.HTableFactory.createHTableInterface(HTableFactory.java:38)
at
org.apache.hadoop.hbase.client.HTablePool.createHTable(HTablePool.java:265)
at
org.apache.hadoop.hbase.client.HTablePool.findOrCreateTable(HTablePool.java:195)
at
org.apache.hadoop.hbase.client.HTablePool.getTable(HTablePool.java:174)
But later after restart and rerunning application, it disappeared.
于 9/26/2013 11:03 AM, Ted Yu 写道:
> Can you check NameNode log ?
>
> What Hadoop / HBase releases are you using ?
>
> Thanks
>
> On Sep 25, 2013, at 7:52 PM, kun yan <ya...@gmail.com> wrote:
>
>> i check regionserver logs
>> What should I do, I only know a little bit HLog
>>
>> 2013-09-26 10:37:13,478 WARN org.apache.hadoop.hbase.util.FSHDFSUtils:
>> Cannot recoverLease after trying for 900000ms
>> (hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!;
>> attempt=16 on
>> file=hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
>> after 921109ms
>> 2013-09-26 10:37:13,519 WARN org.apache.hadoop.hbase.regionserver.wal.HLog:
>> Lease should have recovered. This is not expected. Will retry
>> java.io.IOException: Cannot obtain block length for
>> LocatedBlock{BP-1087715125-192.5.1.50-1378889582109:blk_-8658284328699269340_21570;
>> getBlockSize()=0; corrupt=false; offset=0; locs=[192.5.1.56:50010,
>> 192.5.1.52:50010, 192.5.1.55:50010]}
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:319)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:263)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
>> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1787)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:62)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1707)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
>> at java.lang.Thread.run(Thread.java:722)
>> 2013-09-26 10:40:05,900 DEBUG
>> org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=2.02 MB,
>> free=243.83 MB, max=245.84 MB, blocks=0, accesses=0, hits=0, hitRatio=0,
>> cachingAccesses=0, cachingHits=0, cachingHitsRatio=0, evictions=0,
>> evicted=0, evictedPerRun=NaN
>> 2013-09-26 10:40:07,770 DEBUG
>> org.apache.hadoop.hbase.regionserver.LogRoller: Hlog roll period 3600000ms
>> elapsed
>> 2013-09-26 10:42:14,404 ERROR
>> org.apache.hadoop.hbase.regionserver.wal.HLog: Can't open after 300
>> attempts and 300926ms for
>> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
>> 2013-09-26 10:42:14,404 INFO
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 0 edits
>> across 0 regions threw away edits for 0 regions; log
>> file=hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
>> is corrupted = false progress failed = false
>> 2013-09-26 10:42:14,404 WARN
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
>> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
>> failed, returning error
>> java.io.IOException: Cannot obtain block length for
>> LocatedBlock{BP-1087715125-192.5.1.50-1378889582109:blk_-8658284328699269340_21570;
>> getBlockSize()=0; corrupt=false; offset=0; locs=[192.5.1.56:50010,
>> 192.5.1.52:50010, 192.5.1.55:50010]}
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:319)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:263)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
>> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
>> at
>> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1787)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:62)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1707)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
>> at java.lang.Thread.run(Thread.java:722)
>> 2013-09-26 10:42:14,418 INFO
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully
>> transitioned task
>> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
>> to final state err
>> 2013-09-26 10:42:14,419 INFO
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker
>> hydra0006,60020,1380159605759 done with task
>> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
>> in 1222058ms
>> 2013-09-26 10:42:14,427 DEBUG
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker: tasks arrived or
>> departed
>> 2013-09-26 10:42:44,450 DEBUG
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker: tasks arrived or
>> departed
>> 2013-09-26 10:42:44,497 INFO
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker
>> hydra0006,60020,1380159605759 acquired task
>> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0003%2C60020%2C1379926447350-splitting%2Fhydra0003%252C60020%252C1379926447350.1380157487667
>> 2013-09-26 10:42:44,499 INFO
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog:
>> hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
>> length=63861780
>> 2013-09-26 10:42:44,506 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
>> Recovering lease on dfs file
>> hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667
>> 2013-09-26 10:42:44,507 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
>> recoverLease=true, attempt=0 on
>> file=hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667
>> after 1ms
>> 2013-09-26 10:42:44,512 WARN org.apache.hadoop.hdfs.DFSClient: Failed to
>> connect to /192.5.1.56:50010 for block, add to deadNodes and continue.
>> java.io.IOException: Got error for OP_READ_BLOCK, self=/192.5.1.56:37058,
>> remote=/192.5.1.56:50010, for file
>> /hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
>> for pool BP-1087715125-192.5.1.50-1378889582109 block
>> 785178848034028699_21541
>> java.io.IOException: Got error for OP_READ_BLOCK, self=/192.5.1.56:37058,
>> remote=/192.5.1.56:50010, for file
>> /hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
>> for pool BP-1087715125-192.5.1.50-1378889582109 block
>> 785178848034028699_21541
>> at
>> org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:429)
>> at
>> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:394)
>> at
>> org.apache.hadoop.hdfs.BlockReaderFactory.newBlockReader(BlockReaderFactory.java:137)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1103)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:538)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:750)
>> at
>> org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:794)
>> at java.io.DataInputStream.read(DataInputStream.java:149)
>> at java.io.DataInputStream.readFully(DataInputStream.java:195)
>> at java.io.DataInputStream.readFully(DataInputStream.java:169)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
>> at
>> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
>> at
>> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
>> at
>> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
>> at java.lang.Thread.run(Thread.java:722)
>>
>>
>> 2013/9/26 Jimmy Xiang <jx...@cloudera.com>
>>
>>> In the region server log, you should see the details about the failure.
>>>
>>>
>>> On Wed, Sep 25, 2013 at 7:31 PM, kun yan <ya...@gmail.com> wrote:
>>>
>>>> I use importtsv import data into HDFS, but during the power outage.
>>>> Horrible, and then I re-import the data.(hbase 0.94)
>>>>
>>>> the exception
>>>> org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find
>>>> region for data2,04684015,99999999999999 after 10 tries.
>>>>
>>>> I look HMaster logs as follows:
>>>>
>>>> 2013-09-26 10:21:17,874 DEBUG
>>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>>> unassigned
>>>> = 0
>>>> 2013-09-26 10:21:18,875 DEBUG
>>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>>> unassigned
>>>> = 0
>>>> 2013-09-26 10:21:19,875 DEBUG
>>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>>> unassigned
>>>> = 0
>>>> 2013-09-26 10:21:20,875 DEBUG
>>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>>> unassigned
>>>> = 0
>>>> 2013-09-26 10:21:21,876 DEBUG
>>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>>> unassigned
>>>> = 0
>>>> 2013-09-26 10:21:22,875 DEBUG
>>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>>> unassigned
>>>> = 0
>>>> 2013-09-26 10:21:23,875 DEBUG
>>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>>> unassigned
>>>> = 0
>>>> 2013-09-26 10:21:24,875 DEBUG
>>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>>> unassigned
>>>> = 0
>>>> 2013-09-26 10:21:25,436 INFO
>>>> org.apache.hadoop.hbase.master.SplitLogManager: task
>>> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
>>>> entered state err hydra0004,60020,1380159614688
>>>> 2013-09-26 10:21:25,436 WARN
>>>> org.apache.hadoop.hbase.master.SplitLogManager: Error splitting
>>> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
>>>> 2013-09-26 10:21:25,436 WARN
>>>> org.apache.hadoop.hbase.master.SplitLogManager: error while splitting
>>> logs
>>>> in
>>> [hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
>>> hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
>>> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
>>>> installed = 2 but only 0 done
>>>> 2013-09-26 10:21:25,436 WARN
>>>> org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting of
>>>> [hydra0003,60020,1379926447350, hydra0004,60020,1379926440171,
>>>> hydra0006,60020,1379926437471]
>>>> java.io.IOException: error or interrupted while splitting logs in
>>> [hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
>>> hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
>>> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
>>>> Task = installed = 2 done = 0 error = 2
>>>> at
>>> org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:282)
>>>> at
>>> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:300)
>>>> at
>>> org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:242)
>>>> at
>>> org.apache.hadoop.hbase.master.HMaster.splitLogAfterStartup(HMaster.java:661)
>>>> at
>>> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:580)
>>>> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:396)
>>>> at java.lang.Thread.run(Thread.java:722)
>>>>
>>>> --
>>>>
>>>> In the Hadoop world, I am just a novice, explore the entire Hadoop
>>>> ecosystem, I hope one day I can contribute their own code
>>>>
>>>> YanBit
>>>> yankunhadoop@gmail.com
>>
>>
>> --
>>
>> In the Hadoop world, I am just a novice, explore the entire Hadoop
>> ecosystem, I hope one day I can contribute their own code
>>
>> YanBit
>> yankunhadoop@gmail.com
--
Best Regards, Julian
Re: SplitLogManager issue
Posted by Ted Yu <yu...@gmail.com>.
Can you check NameNode log ?
What Hadoop / HBase releases are you using ?
Thanks
On Sep 25, 2013, at 7:52 PM, kun yan <ya...@gmail.com> wrote:
> i check regionserver logs
> What should I do, I only know a little bit HLog
>
> 2013-09-26 10:37:13,478 WARN org.apache.hadoop.hbase.util.FSHDFSUtils:
> Cannot recoverLease after trying for 900000ms
> (hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!;
> attempt=16 on
> file=hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
> after 921109ms
> 2013-09-26 10:37:13,519 WARN org.apache.hadoop.hbase.regionserver.wal.HLog:
> Lease should have recovered. This is not expected. Will retry
> java.io.IOException: Cannot obtain block length for
> LocatedBlock{BP-1087715125-192.5.1.50-1378889582109:blk_-8658284328699269340_21570;
> getBlockSize()=0; corrupt=false; offset=0; locs=[192.5.1.56:50010,
> 192.5.1.52:50010, 192.5.1.55:50010]}
> at
> org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:319)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:263)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
> at
> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
> at
> org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1787)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:62)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1707)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
> at java.lang.Thread.run(Thread.java:722)
> 2013-09-26 10:40:05,900 DEBUG
> org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=2.02 MB,
> free=243.83 MB, max=245.84 MB, blocks=0, accesses=0, hits=0, hitRatio=0,
> cachingAccesses=0, cachingHits=0, cachingHitsRatio=0, evictions=0,
> evicted=0, evictedPerRun=NaN
> 2013-09-26 10:40:07,770 DEBUG
> org.apache.hadoop.hbase.regionserver.LogRoller: Hlog roll period 3600000ms
> elapsed
> 2013-09-26 10:42:14,404 ERROR
> org.apache.hadoop.hbase.regionserver.wal.HLog: Can't open after 300
> attempts and 300926ms for
> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
> 2013-09-26 10:42:14,404 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 0 edits
> across 0 regions threw away edits for 0 regions; log
> file=hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
> is corrupted = false progress failed = false
> 2013-09-26 10:42:14,404 WARN
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
> failed, returning error
> java.io.IOException: Cannot obtain block length for
> LocatedBlock{BP-1087715125-192.5.1.50-1378889582109:blk_-8658284328699269340_21570;
> getBlockSize()=0; corrupt=false; offset=0; locs=[192.5.1.56:50010,
> 192.5.1.52:50010, 192.5.1.55:50010]}
> at
> org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:319)
> at
> org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:263)
> at
> org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
> at
> org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
> at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
> at
> org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1787)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:62)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1707)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
> at java.lang.Thread.run(Thread.java:722)
> 2013-09-26 10:42:14,418 INFO
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully
> transitioned task
> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
> to final state err
> 2013-09-26 10:42:14,419 INFO
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker
> hydra0006,60020,1380159605759 done with task
> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
> in 1222058ms
> 2013-09-26 10:42:14,427 DEBUG
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: tasks arrived or
> departed
> 2013-09-26 10:42:44,450 DEBUG
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: tasks arrived or
> departed
> 2013-09-26 10:42:44,497 INFO
> org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker
> hydra0006,60020,1380159605759 acquired task
> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0003%2C60020%2C1379926447350-splitting%2Fhydra0003%252C60020%252C1379926447350.1380157487667
> 2013-09-26 10:42:44,499 INFO
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog:
> hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
> length=63861780
> 2013-09-26 10:42:44,506 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
> Recovering lease on dfs file
> hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667
> 2013-09-26 10:42:44,507 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
> recoverLease=true, attempt=0 on
> file=hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667
> after 1ms
> 2013-09-26 10:42:44,512 WARN org.apache.hadoop.hdfs.DFSClient: Failed to
> connect to /192.5.1.56:50010 for block, add to deadNodes and continue.
> java.io.IOException: Got error for OP_READ_BLOCK, self=/192.5.1.56:37058,
> remote=/192.5.1.56:50010, for file
> /hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
> for pool BP-1087715125-192.5.1.50-1378889582109 block
> 785178848034028699_21541
> java.io.IOException: Got error for OP_READ_BLOCK, self=/192.5.1.56:37058,
> remote=/192.5.1.56:50010, for file
> /hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
> for pool BP-1087715125-192.5.1.50-1378889582109 block
> 785178848034028699_21541
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:429)
> at
> org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:394)
> at
> org.apache.hadoop.hdfs.BlockReaderFactory.newBlockReader(BlockReaderFactory.java:137)
> at
> org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1103)
> at
> org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:538)
> at
> org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:750)
> at
> org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:794)
> at java.io.DataInputStream.read(DataInputStream.java:149)
> at java.io.DataInputStream.readFully(DataInputStream.java:195)
> at java.io.DataInputStream.readFully(DataInputStream.java:169)
> at
> org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
> at
> org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
> at
> org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
> at
> org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
> at
> org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
> at java.lang.Thread.run(Thread.java:722)
>
>
> 2013/9/26 Jimmy Xiang <jx...@cloudera.com>
>
>> In the region server log, you should see the details about the failure.
>>
>>
>> On Wed, Sep 25, 2013 at 7:31 PM, kun yan <ya...@gmail.com> wrote:
>>
>>> I use importtsv import data into HDFS, but during the power outage.
>>> Horrible, and then I re-import the data.(hbase 0.94)
>>>
>>> the exception
>>> org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find
>>> region for data2,04684015,99999999999999 after 10 tries.
>>>
>>> I look HMaster logs as follows:
>>>
>>> 2013-09-26 10:21:17,874 DEBUG
>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>> unassigned
>>> = 0
>>> 2013-09-26 10:21:18,875 DEBUG
>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>> unassigned
>>> = 0
>>> 2013-09-26 10:21:19,875 DEBUG
>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>> unassigned
>>> = 0
>>> 2013-09-26 10:21:20,875 DEBUG
>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>> unassigned
>>> = 0
>>> 2013-09-26 10:21:21,876 DEBUG
>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>> unassigned
>>> = 0
>>> 2013-09-26 10:21:22,875 DEBUG
>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>> unassigned
>>> = 0
>>> 2013-09-26 10:21:23,875 DEBUG
>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>> unassigned
>>> = 0
>>> 2013-09-26 10:21:24,875 DEBUG
>>> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
>> unassigned
>>> = 0
>>> 2013-09-26 10:21:25,436 INFO
>>> org.apache.hadoop.hbase.master.SplitLogManager: task
>> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
>>> entered state err hydra0004,60020,1380159614688
>>> 2013-09-26 10:21:25,436 WARN
>>> org.apache.hadoop.hbase.master.SplitLogManager: Error splitting
>> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
>>> 2013-09-26 10:21:25,436 WARN
>>> org.apache.hadoop.hbase.master.SplitLogManager: error while splitting
>> logs
>>> in
>> [hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
>> hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
>> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
>>> installed = 2 but only 0 done
>>> 2013-09-26 10:21:25,436 WARN
>>> org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting of
>>> [hydra0003,60020,1379926447350, hydra0004,60020,1379926440171,
>>> hydra0006,60020,1379926437471]
>>> java.io.IOException: error or interrupted while splitting logs in
>> [hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
>> hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
>> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
>>> Task = installed = 2 done = 0 error = 2
>>> at
>> org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:282)
>>> at
>> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:300)
>>> at
>> org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:242)
>>> at
>> org.apache.hadoop.hbase.master.HMaster.splitLogAfterStartup(HMaster.java:661)
>>> at
>> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:580)
>>> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:396)
>>> at java.lang.Thread.run(Thread.java:722)
>>>
>>> --
>>>
>>> In the Hadoop world, I am just a novice, explore the entire Hadoop
>>> ecosystem, I hope one day I can contribute their own code
>>>
>>> YanBit
>>> yankunhadoop@gmail.com
>
>
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com
Re: SplitLogManager issue
Posted by kun yan <ya...@gmail.com>.
i check regionserver logs
What should I do, I only know a little bit HLog
2013-09-26 10:37:13,478 WARN org.apache.hadoop.hbase.util.FSHDFSUtils:
Cannot recoverLease after trying for 900000ms
(hbase.lease.recovery.timeout); continuing, but may be DATALOSS!!!;
attempt=16 on
file=hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
after 921109ms
2013-09-26 10:37:13,519 WARN org.apache.hadoop.hbase.regionserver.wal.HLog:
Lease should have recovered. This is not expected. Will retry
java.io.IOException: Cannot obtain block length for
LocatedBlock{BP-1087715125-192.5.1.50-1378889582109:blk_-8658284328699269340_21570;
getBlockSize()=0; corrupt=false; offset=0; locs=[192.5.1.56:50010,
192.5.1.52:50010, 192.5.1.55:50010]}
at
org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:319)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:263)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
at
org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
at
org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1787)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:62)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1707)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
at java.lang.Thread.run(Thread.java:722)
2013-09-26 10:40:05,900 DEBUG
org.apache.hadoop.hbase.io.hfile.LruBlockCache: Stats: total=2.02 MB,
free=243.83 MB, max=245.84 MB, blocks=0, accesses=0, hits=0, hitRatio=0,
cachingAccesses=0, cachingHits=0, cachingHitsRatio=0, evictions=0,
evicted=0, evictedPerRun=NaN
2013-09-26 10:40:07,770 DEBUG
org.apache.hadoop.hbase.regionserver.LogRoller: Hlog roll period 3600000ms
elapsed
2013-09-26 10:42:14,404 ERROR
org.apache.hadoop.hbase.regionserver.wal.HLog: Can't open after 300
attempts and 300926ms for
hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
2013-09-26 10:42:14,404 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Processed 0 edits
across 0 regions threw away edits for 0 regions; log
file=hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
is corrupted = false progress failed = false
2013-09-26 10:42:14,404 WARN
org.apache.hadoop.hbase.regionserver.SplitLogWorker: log splitting of
hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting/hydra0006%2C60020%2C1379926437471.1380157500804
failed, returning error
java.io.IOException: Cannot obtain block length for
LocatedBlock{BP-1087715125-192.5.1.50-1378889582109:blk_-8658284328699269340_21570;
getBlockSize()=0; corrupt=false; offset=0; locs=[192.5.1.56:50010,
192.5.1.52:50010, 192.5.1.55:50010]}
at
org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:319)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:263)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:205)
at
org.apache.hadoop.hdfs.DFSInputStream.<init>(DFSInputStream.java:198)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1117)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
at
org.apache.hadoop.io.SequenceFile$Reader.openFile(SequenceFile.java:1787)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.openFile(SequenceFileLogReader.java:62)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1707)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
at java.lang.Thread.run(Thread.java:722)
2013-09-26 10:42:14,418 INFO
org.apache.hadoop.hbase.regionserver.SplitLogWorker: successfully
transitioned task
/hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
to final state err
2013-09-26 10:42:14,419 INFO
org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker
hydra0006,60020,1380159605759 done with task
/hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
in 1222058ms
2013-09-26 10:42:14,427 DEBUG
org.apache.hadoop.hbase.regionserver.SplitLogWorker: tasks arrived or
departed
2013-09-26 10:42:44,450 DEBUG
org.apache.hadoop.hbase.regionserver.SplitLogWorker: tasks arrived or
departed
2013-09-26 10:42:44,497 INFO
org.apache.hadoop.hbase.regionserver.SplitLogWorker: worker
hydra0006,60020,1380159605759 acquired task
/hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0003%2C60020%2C1379926447350-splitting%2Fhydra0003%252C60020%252C1379926447350.1380157487667
2013-09-26 10:42:44,499 INFO
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter: Splitting hlog:
hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
length=63861780
2013-09-26 10:42:44,506 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
Recovering lease on dfs file
hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667
2013-09-26 10:42:44,507 INFO org.apache.hadoop.hbase.util.FSHDFSUtils:
recoverLease=true, attempt=0 on
file=hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667
after 1ms
2013-09-26 10:42:44,512 WARN org.apache.hadoop.hdfs.DFSClient: Failed to
connect to /192.5.1.56:50010 for block, add to deadNodes and continue.
java.io.IOException: Got error for OP_READ_BLOCK, self=/192.5.1.56:37058,
remote=/192.5.1.56:50010, for file
/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
for pool BP-1087715125-192.5.1.50-1378889582109 block
785178848034028699_21541
java.io.IOException: Got error for OP_READ_BLOCK, self=/192.5.1.56:37058,
remote=/192.5.1.56:50010, for file
/hbase/.logs/hydra0003,60020,1379926447350-splitting/hydra0003%2C60020%2C1379926447350.1380157487667,
for pool BP-1087715125-192.5.1.50-1378889582109 block
785178848034028699_21541
at
org.apache.hadoop.hdfs.RemoteBlockReader2.checkSuccess(RemoteBlockReader2.java:429)
at
org.apache.hadoop.hdfs.RemoteBlockReader2.newBlockReader(RemoteBlockReader2.java:394)
at
org.apache.hadoop.hdfs.BlockReaderFactory.newBlockReader(BlockReaderFactory.java:137)
at
org.apache.hadoop.hdfs.DFSInputStream.getBlockReader(DFSInputStream.java:1103)
at
org.apache.hadoop.hdfs.DFSInputStream.blockSeekTo(DFSInputStream.java:538)
at
org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:750)
at
org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:794)
at java.io.DataInputStream.read(DataInputStream.java:149)
at java.io.DataInputStream.readFully(DataInputStream.java:195)
at java.io.DataInputStream.readFully(DataInputStream.java:169)
at
org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1800)
at
org.apache.hadoop.io.SequenceFile$Reader.initialize(SequenceFile.java:1765)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1714)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1728)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader$WALReader.<init>(SequenceFileLogReader.java:55)
at
org.apache.hadoop.hbase.regionserver.wal.SequenceFileLogReader.init(SequenceFileLogReader.java:177)
at
org.apache.hadoop.hbase.regionserver.wal.HLog.getReader(HLog.java:713)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:846)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.getReader(HLogSplitter.java:759)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:403)
at
org.apache.hadoop.hbase.regionserver.wal.HLogSplitter.splitLogFile(HLogSplitter.java:371)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker$1.exec(SplitLogWorker.java:115)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.grabTask(SplitLogWorker.java:283)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.taskLoop(SplitLogWorker.java:214)
at
org.apache.hadoop.hbase.regionserver.SplitLogWorker.run(SplitLogWorker.java:182)
at java.lang.Thread.run(Thread.java:722)
2013/9/26 Jimmy Xiang <jx...@cloudera.com>
> In the region server log, you should see the details about the failure.
>
>
> On Wed, Sep 25, 2013 at 7:31 PM, kun yan <ya...@gmail.com> wrote:
>
> > I use importtsv import data into HDFS, but during the power outage.
> > Horrible, and then I re-import the data.(hbase 0.94)
> >
> > the exception
> > org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find
> > region for data2,04684015,99999999999999 after 10 tries.
> >
> > I look HMaster logs as follows:
> >
> > 2013-09-26 10:21:17,874 DEBUG
> > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
> unassigned
> > = 0
> > 2013-09-26 10:21:18,875 DEBUG
> > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
> unassigned
> > = 0
> > 2013-09-26 10:21:19,875 DEBUG
> > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
> unassigned
> > = 0
> > 2013-09-26 10:21:20,875 DEBUG
> > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
> unassigned
> > = 0
> > 2013-09-26 10:21:21,876 DEBUG
> > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
> unassigned
> > = 0
> > 2013-09-26 10:21:22,875 DEBUG
> > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
> unassigned
> > = 0
> > 2013-09-26 10:21:23,875 DEBUG
> > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
> unassigned
> > = 0
> > 2013-09-26 10:21:24,875 DEBUG
> > org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1
> unassigned
> > = 0
> > 2013-09-26 10:21:25,436 INFO
> > org.apache.hadoop.hbase.master.SplitLogManager: task
> >
> >
> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
> > entered state err hydra0004,60020,1380159614688
> > 2013-09-26 10:21:25,436 WARN
> > org.apache.hadoop.hbase.master.SplitLogManager: Error splitting
> >
> >
> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
> > 2013-09-26 10:21:25,436 WARN
> > org.apache.hadoop.hbase.master.SplitLogManager: error while splitting
> logs
> > in
> >
> [hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
> >
> hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
> >
> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
> > installed = 2 but only 0 done
> > 2013-09-26 10:21:25,436 WARN
> > org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting of
> > [hydra0003,60020,1379926447350, hydra0004,60020,1379926440171,
> > hydra0006,60020,1379926437471]
> > java.io.IOException: error or interrupted while splitting logs in
> >
> [hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
> >
> hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
> >
> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
> > Task = installed = 2 done = 0 error = 2
> > at
> >
> >
> org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:282)
> > at
> >
> >
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:300)
> > at
> >
> >
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:242)
> > at
> >
> >
> org.apache.hadoop.hbase.master.HMaster.splitLogAfterStartup(HMaster.java:661)
> > at
> >
> >
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:580)
> > at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:396)
> > at java.lang.Thread.run(Thread.java:722)
> >
> > --
> >
> > In the Hadoop world, I am just a novice, explore the entire Hadoop
> > ecosystem, I hope one day I can contribute their own code
> >
> > YanBit
> > yankunhadoop@gmail.com
> >
>
--
In the Hadoop world, I am just a novice, explore the entire Hadoop
ecosystem, I hope one day I can contribute their own code
YanBit
yankunhadoop@gmail.com
Re: SplitLogManager issue
Posted by Jimmy Xiang <jx...@cloudera.com>.
In the region server log, you should see the details about the failure.
On Wed, Sep 25, 2013 at 7:31 PM, kun yan <ya...@gmail.com> wrote:
> I use importtsv import data into HDFS, but during the power outage.
> Horrible, and then I re-import the data.(hbase 0.94)
>
> the exception
> org.apache.hadoop.hbase.client.NoServerForRegionException: Unable to find
> region for data2,04684015,99999999999999 after 10 tries.
>
> I look HMaster logs as follows:
>
> 2013-09-26 10:21:17,874 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
> = 0
> 2013-09-26 10:21:18,875 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
> = 0
> 2013-09-26 10:21:19,875 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
> = 0
> 2013-09-26 10:21:20,875 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
> = 0
> 2013-09-26 10:21:21,876 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
> = 0
> 2013-09-26 10:21:22,875 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
> = 0
> 2013-09-26 10:21:23,875 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
> = 0
> 2013-09-26 10:21:24,875 DEBUG
> org.apache.hadoop.hbase.master.SplitLogManager: total tasks = 1 unassigned
> = 0
> 2013-09-26 10:21:25,436 INFO
> org.apache.hadoop.hbase.master.SplitLogManager: task
>
> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
> entered state err hydra0004,60020,1380159614688
> 2013-09-26 10:21:25,436 WARN
> org.apache.hadoop.hbase.master.SplitLogManager: Error splitting
>
> /hbase/splitlog/hdfs%3A%2F%2Fhydra0001%3A8020%2Fhbase%2F.logs%2Fhydra0006%2C60020%2C1379926437471-splitting%2Fhydra0006%252C60020%252C1379926437471.1380157500804
> 2013-09-26 10:21:25,436 WARN
> org.apache.hadoop.hbase.master.SplitLogManager: error while splitting logs
> in
> [hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
> hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
> installed = 2 but only 0 done
> 2013-09-26 10:21:25,436 WARN
> org.apache.hadoop.hbase.master.MasterFileSystem: Failed splitting of
> [hydra0003,60020,1379926447350, hydra0004,60020,1379926440171,
> hydra0006,60020,1379926437471]
> java.io.IOException: error or interrupted while splitting logs in
> [hdfs://hydra0001:8020/hbase/.logs/hydra0003,60020,1379926447350-splitting,
> hdfs://hydra0001:8020/hbase/.logs/hydra0004,60020,1379926440171-splitting,
> hdfs://hydra0001:8020/hbase/.logs/hydra0006,60020,1379926437471-splitting]
> Task = installed = 2 done = 0 error = 2
> at
>
> org.apache.hadoop.hbase.master.SplitLogManager.splitLogDistributed(SplitLogManager.java:282)
> at
>
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:300)
> at
>
> org.apache.hadoop.hbase.master.MasterFileSystem.splitLogAfterStartup(MasterFileSystem.java:242)
> at
>
> org.apache.hadoop.hbase.master.HMaster.splitLogAfterStartup(HMaster.java:661)
> at
>
> org.apache.hadoop.hbase.master.HMaster.finishInitialization(HMaster.java:580)
> at org.apache.hadoop.hbase.master.HMaster.run(HMaster.java:396)
> at java.lang.Thread.run(Thread.java:722)
>
> --
>
> In the Hadoop world, I am just a novice, explore the entire Hadoop
> ecosystem, I hope one day I can contribute their own code
>
> YanBit
> yankunhadoop@gmail.com
>