You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/06/08 17:40:07 UTC
[jira] Created: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
IllegalArgumentException in halfhfilereader#next
------------------------------------------------
Key: HBASE-1495
URL: https://issues.apache.org/jira/browse/HBASE-1495
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.20.0
>From posix4e up on IRC
{code}
#
2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
#
java.lang.IllegalArgumentException
#
at java.nio.Buffer.position(Buffer.java:218)
#
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
#
at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
#
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
#
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
#
at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
#
reScanner.java:101)
#
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
#
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
#
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
#
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
#
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
{code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-1495:
----------------------------
Assignee: stack
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717747#action_12717747 ]
stack commented on HBASE-1495:
------------------------------
I see this in my testing now. Looks like issue around split; parent is being cleaned up -- deleted -- though daughters are still out on cluster being served.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717462#action_12717462 ]
stack commented on HBASE-1495:
------------------------------
>From posix4e:
{code}
#
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.position():67551
#
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.capacity():67559
#
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.String():java.nio.HeapByteBuffer[pos=67551 lim=67551 cap=67559]
#
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.array():[B@77932b46
#
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.array().length:67559
#
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: currKeyLen:37
#
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: currValueLen:3557
{code}
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717461#action_12717461 ]
stack commented on HBASE-1495:
------------------------------
Here are some numbers from posix4e:
{code}
2009-06-08 15:46:59,325 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.position(): 65648
2009-06-08 15:46:59,325 WARN org.apache.hadoop.hbase.io.hfile.HFile: currKeyLen: 37
2009-06-08 15:46:59,325 WARN org.apache.hadoop.hbase.io.hfile.HFile: currValueLen: 95
{code}
Here is code:
{code}
public boolean next() throws IOException {
// LOG.deug("rem:" + block.remaining() + " p:" + block.position() +
// " kl: " + currKeyLen + " kv: " + currValueLen);
if (block == null) {
throw new IOException("Next called on non-seeked scanner");
}
block.position(block.position() + currKeyLen + currValueLen);
{code}
Here is the Buffer position code:
{code}
public final Buffer position(int newPosition) {
if ((newPosition > limit) || (newPosition < 0))
throw new IllegalArgumentException();
{code}
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1495:
-------------------------------
Attachment: HBASE-1495.patch
compactions are fucking up - the bottom half gets both, and the top half gets the top part.
This is a fix to that problem.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719084#action_12719084 ]
stack commented on HBASE-1495:
------------------------------
Here is example from logs of item 1. that would seem to indicate the issue is in Scanners. I've instrumented meta scanner so that on each next, I print out what was found. Immediately, I do a get on the row and I print that too. Usually the two match. When they don't is when I have issue:
{code}
794 2009-06-12 22:41:18,377 [RegionManager.metaScanner] INFO org.apache.hadoop.hbase.master.BaseScanner: REMOVE NEXT VALUES keyvalues={TestTable,,1244846310969/info:regioninfo/1244846397895/Put/vlen=277, TestTable,,1244846310969/info:server/1244846319501/Put/vle n=19, TestTable,,1244846310969/info:serverstartcode/1244846319501/Put/vlen=8, TestTable,,1244846310969/info:splitA/1244846397895/Put/vlen=277, TestTable,,1244846310969/info:splitB/1244846397895/Put/vlen=297}
795 2009-06-12 22:41:18,379 [RegionManager.metaScanner] INFO org.apache.hadoop.hbase.master.BaseScanner: REMOVE GET keyvalues={TestTable,,1244846310969/historian:assignment/1244846315138/Put/vlen=71, TestTable,,1244846310969/historian:compaction/1244846393353/Pu t/vlen=36, TestTable,,1244846310969/historian:open/1244846319510/Put/vlen=51, TestTable,,1244846310969/historian:split/1244846312701/Put/vlen=43, TestTable,,1244846310969/info:regioninfo/1244846397895/Put/vlen=277, TestTable,,1244846310969/info:server/124484 6319501/Put/vlen=19, TestTable,,1244846310969/info:serverstartcode/1244846319501/Put/vlen=8, TestTable,,1244846310969/info:splitA/1244846397895/Put/vlen=277, TestTable,,1244846310969/info:splitB/1244846397895/Put/vlen=297}
{code}
When above scan finishes, is when we remove splitA. On next meta scan, I see this:
{code}
840 2009-06-12 22:41:28,390 [RegionManager.metaScanner] INFO org.apache.hadoop.hbase.master.BaseScanner: REMOVE NEXT VALUES keyvalues={TestTable,,1244846310969/info:regioninfo/1244846397895/Put/vlen=277, TestTable,,1244846310969/info:server/1244846319501/Put/vle n=19, TestTable,,1244846310969/info:serverstartcode/1244846319501/Put/vlen=8}
841 2009-06-12 22:41:28,403 [RegionManager.metaScanner] INFO org.apache.hadoop.hbase.master.BaseScanner: REMOVE GET keyvalues={TestTable,,1244846310969/historian:assignment/1244846315138/Put/vlen=71, TestTable,,1244846310969/historian:compaction/1244846393353/Pu t/vlen=36, TestTable,,1244846310969/historian:open/1244846319510/Put/vlen=51, TestTable,,1244846310969/historian:split/1244846312701/Put/vlen=43, TestTable,,1244846310969/info:regioninfo/1244846397895/Put/vlen=277, TestTable,,1244846310969/info:server/124484 6319501/Put/vlen=19, TestTable,,1244846310969/info:serverstartcode/1244846319501/Put/vlen=8, TestTable,,1244846310969/info:splitB/1244846397895/Put/vlen=297}
{code}
Above, the splitB is present in the Get but not in the Scan next.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "Alex Newman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717494#action_12717494 ]
Alex Newman commented on HBASE-1495:
------------------------------------
try {
block.position(block.position() + currKeyLen + currValueLen);
} catch (IllegalArgumentException e){
LOG.warn(e.getMessage());
LOG.warn("block.position():"+block.position());
LOG.warn("block.capacity():"+block.capacity());
LOG.warn("block.String():"+block.toString());
LOG.warn("block.array():"+block.array());
LOG.warn("block.array().length:"+block.array().length);
LOG.warn("currKeyLen:"+currKeyLen);
LOG.warn("currValueLen:"+currValueLen);
}
if (block.remaining() <= 0) {
results in
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.position():67551
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.capacity():67559
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.String():java.nio.HeapByteBuffer[pos=67551 lim=67551 cap=67559]
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.array():[B@77932b46
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.array().length:67559
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: currKeyLen:37
2009-06-08 17:45:48,706 WARN org.apache.hadoop.hbase.io.hfile.HFile: currValueLen:3557
2009-06-08 17:45:48,706 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2009-06-08 17:45:48,709 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_-8600012014215763562_6667788 from any node: java.io.IOException: No live nodes contain current block
2009-06-08 17:45:51,714 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_-8600012014215763562_6667788 from any node: java.io.IOException: No live nodes contain current block
2009-06-08 17:45:54,716 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io.IOException: Cannot open filename /hbase/t3/299987382/block/2277509959628818760
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1444)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1769)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1585)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1712)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:96)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:86)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:950)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:906)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1093)
at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingStoreScanner.java:101)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:852)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:717)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "Alex Newman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717469#action_12717469 ]
Alex Newman commented on HBASE-1495:
------------------------------------
2009-06-08 18:04:36,722 WARN org.apache.hadoop.hbase.io.hfile.HFile:
2009-06-08 18:04:36,722 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.position():66795
2009-06-08 18:04:36,722 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.capacity():66803
2009-06-08 18:04:36,722 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.String():java.nio.HeapByteBuffer[pos=66795 lim=66795 cap=66803]
2009-06-08 18:04:36,722 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.array():[B@5c74b704
2009-06-08 18:04:36,722 WARN org.apache.hadoop.hbase.io.hfile.HFile: block.array().length:66803
2009-06-08 18:04:36,722 WARN org.apache.hadoop.hbase.io.hfile.HFile: currKeyLen:37
2009-06-08 18:04:36,722 WARN org.apache.hadoop.hbase.io.hfile.HFile: currValueLen:23965
2009-06-08 18:04:36,722 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2009-06-08 18:04:36,724 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_626364962828652414_6670983 from any node: java.io.IOException: No live nodes contain current block
2009-06-08 18:04:39,728 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_626364962828652414_6670983 from any node: java.io.IOException: No live nodes contain current block
2009-06-08 18:04:42,731 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io.IOException: Cannot open filename /hbase/t3/1992779812/block/3201548317092254781
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1444)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1769)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1585)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1712)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:96)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:86)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:950)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:906)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1093)
at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingStoreScanner.java:101)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:852)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:717)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719223#action_12719223 ]
ryan rawson commented on HBASE-1495:
------------------------------------
my IDE says this.row = row is self assigning 'row' to itself. It is a noop essentially.
I thought I was removing the System.out.printlns, but I'll check again.
I found that LOG.debug(location) was kicking out 1 line for every item I am committing, making my client stupidly noisy. Maybe the problem is my client has debug on, but i dont know how to solve having client and server on different log settings if they are using the same classpath on the same machine.
I'll try to create some new patches.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson updated HBASE-1495:
-------------------------------
Attachment: (was: HBASE-1495.patch)
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719412#action_12719412 ]
ryan rawson commented on HBASE-1495:
------------------------------------
I've figured out this bug, and it has been fixed in the guise of HBASE-1520, HBASE-1522.
So to outline the problems:
HBASE-1522 would cause the problem whereby a split child would have it's parent removed from under it.
Then we'd go to next() in hfile, this it the code:
public boolean next() throws IOException {
if (block == null) {
throw new IOException("Next called on non-seeked scanner");
}
block.position(block.position() + currKeyLen + currValueLen);
if (block.remaining() <= 0) {
// load next block
}
So we'd call block.position(...), and get to the end of the block, thus triggering the 'load next block' code.
But at this point, the HDFS code would throw an exception - reading a deleted file fails.
But due to HBASE-1520, the scan code would ignore the exception.
Then, we'd call next() again. Only this time the block.position() is at the end, so attempts to push it forward would throw the InvalidArgumentException indicated above.
The fixes are 2 fold:
- don't ignore IOExceptions
- don't delete a split parent if there are children pointing to it.
Both of these have been fixed in other issues.
I am very satisfied that the bug has been nailed to the wall. Both these bugs were committed to trunk. Please verify if you can and let me know if it's fixed.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719172#action_12719172 ]
stack commented on HBASE-1495:
------------------------------
And things below should be inside try/finallys:
{code}
+ MinorCompactingStoreScanner scanner = new MinorCompactingStoreScanner(this, scanners);
+ while ( scanner.next(writer) ) { }
+
+ scanner.close();
{code}
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719217#action_12719217 ]
stack commented on HBASE-1495:
------------------------------
I'm going out. As of this evening, item 1 in comment '12/Jun/09 10:15 PM' above is still happening in spite of a bunch of Jon work (hbase-1516, etc.). Will try again tomorrow... but probably not till monday.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "Alex Newman (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12717493#action_12717493 ]
Alex Newman commented on HBASE-1495:
------------------------------------
computation.
Regions On FS 20 Number of regions on FileSystem. Rough count.
: servers: 7 requests=0, regions=13
node 5
2009-06-07 20:14:59,702 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_8054294304707090727_6626586 from any node: java.io.IOException: No live nodes contain current block
2009-06-07 20:15:23,007 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -8896201081780238142 lease expired
2009-06-07 20:15:27,879 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 1596079652788274335 lease expired
2009-06-07 20:15:40,059 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -2733198405792644962 lease expired
2009-06-07 20:16:10,248 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -7860104214020620019 lease expired
2009-06-07 20:16:25,149 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_8054294304707090727_6626586 from any node: java.io.IOException: No live nodes contain current block
2009-06-07 20:16:40,440 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 6217323943013460033 lease expired
2009-06-07 20:17:10,680 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -6711331116393695890 lease expired
2009-06-07 20:17:40,835 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 3871771505311058479 lease expired
2009-06-07 20:18:10,069 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_8054294304707090727_6626586 from any node: java.io.IOException: No live nodes contain current block
2009-06-07 20:18:11,037 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner -8973011642755732600 lease expired
2009-06-07 20:18:41,201 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 3807906792818606834 lease expired
2009-06-07 20:19:13,095 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: Scanner 579239649970047854 lease expired
2009-06-07 20:19:14,773 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io.IOException: Could not obtain block: blk_8054294304707090727_6626586 file=/hbase/.META./1028785192/info/8853829816968996247
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1757)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1585)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1712)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:950)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:906)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.loadBlock(HFile.java:1222)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.seekTo(HFile.java:1105)
at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.getStoreFile(StoreFileGetScan.java:80)
at org.apache.hadoop.hbase.regionserver.StoreFileGetScan.get(StoreFileGetScan.java:65)
at org.apache.hadoop.hbase.regionserver.Store.get(Store.java:1480)
at org.apache.hadoop.hbase.regionserver.HRegion.getClosestRowBefore(HRegion.java:1037)
at org.apache.hadoop.hbase.regionserver.HRegionServer.getClosestRowBefore(HRegionServer.java:1706)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:643)
at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:913)
....
node 1
2009-06-07 20:22:33,342 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io.IOException: Canno
t open filename /hbase/t3/601977017/block/4848245505122296259
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1444)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1769)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1585)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1712)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:96)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:86)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:950)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:906)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1082)
at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingStoreScanner.java:101)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
reScanner.java:101)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
2009-06-07 20:22:33,368 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region t3,,1244420449037
2009-06-07 20:22:33,368 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region t3,,1244420449037
2009-06-07 20:22:33,373 INFO org.apache.hadoop.io.compress.CodecPool: Got brand-new decompressor
2009-06-07 20:25:01,766 INFO org.apache.hadoop.hbase.regionserver.HRegionServer: compactions no longer limited
2009-06-07 20:27:47,938 INFO org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region t3,,1244420449037 in 5mins, 14sec
2009-06-07 20:27:47,938 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting split of region t3,,1244420449037
2009-06-07 20:27:47,942 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed t3,,1244420449037
2009-06-07 20:27:48,319 INFO org.apache.hadoop.hbase.regionserver.HRegion: region t3,,1244420867941/2082155757 available; sequence id is 9633520
2009-06-07 20:27:48,319 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed t3,,1244420867941
2009-06-07 20:27:48,465 INFO org.apache.hadoop.hbase.regionserver.HRegion: region t3,******************,1244420867941/1057048264 available; sequence id is 9633521
2009-06-07 20:27:48,465 INFO org.apache.hadoop.hbase.regionserver.HRegion: Closed t3,U*****************,1244420867941
2009-06-07 20:27:48,470 INFO org.apache.hadoop.hbase.regionserver.CompactSplitThread: region split, META updated, and report to master all successful. Old region=REGION => {NAME => 't3,,1244420449037', STARTKEY => '', ENDKEY => '*****************', ENCODED => 2120511499, OFFLINE => true, SPLIT => true, TABLE => {{NAME => 't3', FAMILIES => [{NAME => '*****************', COMPRESSION => 'GZ', VERSIONS => '3', TTL => '2147483647', BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}}, new regions: t3,,1244420867941, t3,*****************,1244420867941. Split took 0sec
2009-06-07 20:27:48,470 INFO org.apache.hadoop.hbase.regionserver.HRegion: Starting compaction on region t3,*****************,1244420449037
2009-06-07 20:27:58,106 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7797131986764001105_6627592 from any node: java.io.IOException: No live nodes contain current block
2009-06-07 20:28:01,111 INFO org.apache.hadoop.hdfs.DFSClient: Could not obtain block blk_7797131986764001105_6627592 from any node: java.io.IOException: No live nodes contain current block
2009-06-07 20:28:04,113 WARN org.apache.hadoop.hdfs.DFSClient: DFS Read: java.io.IOException: Cannot open filename /hbase/t3/82566562/block/7165913658187924750
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.openInfo(DFSClient.java:1444)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.chooseDataNode(DFSClient.java:1769)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.blockSeekTo(DFSClient.java:1585)
at org.apache.hadoop.hdfs.DFSClient$DFSInputStream.read(DFSClient.java:1712)
at java.io.DataInputStream.read(DataInputStream.java:132)
at org.apache.hadoop.hbase.io.hfile.BoundedRangeFileInputStream.read(BoundedRangeFileInputStream.java:99)
at org.apache.hadoop.io.compress.DecompressorStream.getCompressedData(DecompressorStream.java:96)
at org.apache.hadoop.io.compress.DecompressorStream.decompress(DecompressorStream.java:86)
at org.apache.hadoop.io.compress.DecompressorStream.read(DecompressorStream.java:74)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:100)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.decompress(HFile.java:950)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader.readBlock(HFile.java:906)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1082)
at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingStoreScanner.java:101)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
2009-06-07 20:28:04,124 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*****************,1244420449037
java.lang.IllegalArgumentException
at java.nio.Buffer.position(Buffer.java:218)
at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingStoreScanner.java:101)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
2009-06-07 20:07:36,330 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=*****,***** ip=/***.**.**.110 cmd=mkdirs src=/hbase/.META./1028785192/historian dst=null perm=ts:ticker:rwxr-xr-x
2009-06-07 20:07:36,335 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=*****,***** ip=/***.**.**.110 cmd=create src=/hbase/.META./1028785192/historian/3542463731429695625 dst=null perm=ts:ticker:
rw-r--r--
2009-06-07 20:07:36,340 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /hbase/.META./1028785192/historian/3542463731429695625. blk_-4350004983998257175_6626584
2009-06-07 20:07:36,348 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.111:50010 is added to blk_7773188474590228904_6626583 size 778360
2009-06-07 20:07:36,348 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.107:50010 is added to blk_7773188474590228904_6626583 size 778360
2009-06-07 20:07:36,349 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.112:50010 is added to blk_7773188474590228904_6626583 size 778360
2009-06-07 20:07:36,354 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=*****,***** ip=/***.**.**.112 cmd=create src=/hbase/.logs/*************,60020,1244418229161/hlog.dat.1244419656351 dst=null
perm=ts:ticker:rw-r--r--
2009-06-07 20:07:36,365 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.111:50010 is added to blk_-4350004983998257175_6626584 size 3190
2009-06-07 20:07:36,365 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.107:50010 is added to blk_-4350004983998257175_6626584 size 3190
2009-06-07 20:07:36,365 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.110:50010 is added to blk_-4350004983998257175_6626584 size 3190
2009-06-07 20:07:36,373 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=*****,***** ip=/***.**.**.110 cmd=open src=/hbase/.META./1028785192/historian/3542463731429695625 dst=null perm=null
2009-06-07 20:07:36,379 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=*****,***** ip=/***.**.**.110 cmd=mkdirs src=/hbase/.META./1028785192/info dst=null perm=ts:ticker:rwxr-xr-x
2009-06-07 20:07:36,382 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=*****,***** ip=/***.**.**.110 cmd=create src=/hbase/.META./1028785192/info/8853829816968996247 dst=null perm=ts:ticker:rw-r--r-
-
2009-06-07 20:07:36,384 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /hbase/.META./1028785192/info/8853829816968996247. blk_8054294304707090727_6626586
2009-06-07 20:07:36,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.107:50010 is added to blk_8054294304707090727_6626586 size 7296
2009-06-07 20:07:36,387 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.111:50010 is added to blk_8054294304707090727_6626586 size 7296
2009-06-07 20:07:36,388 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.110:50010 is added to blk_8054294304707090727_6626586 size 7296
2009-06-07 20:07:36,392 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=*****,***** ip=/***.**.**.110 cmd=open src=/hbase/.META./1028785192/info/8853829816968996247 dst=null perm=null
2009-06-07 20:07:36,409 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /hbase/.logs/**************,60020,1244418229161/hlog.dat.1244419656351. blk_-249701538603762950_6626586
2009-06-07 20:07:37,238 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.110:50010 is added to blk_-249701538603762950_6626586 size 783411
2009-06-07 20:07:37,239 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.107:50010 is added to blk_-249701538603762950_6626586 size 783411
2009-06-07 20:07:37,239 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.112:50010 is added to blk_-249701538603762950_6626586 size 783411
2009-06-07 20:07:37,244 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=*****,***** ip=/***.**.**.112 cmd=create src=/hbase/.logs/************,60020,1244418229161/hlog.dat.1244419657242 dst=null
perm=ts:ticker:rw-r--r--
2009-06-07 20:07:37,354 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.allocateBlock: /hbase/.logs/************,60020,1244418229161/hlog.dat.1244419657242. blk_-4367384643419626092_6626587
2009-06-07 20:07:37,830 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.29:50010 is added to blk_4530508892235409778_6626487 size 778943
2009-06-07 20:07:37,830 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.112:50010 is added to blk_4530508892235409778_6626487 size 778943
2009-06-07 20:07:37,831 INFO org.apache.hadoop.hdfs.StateChange: BLOCK* NameSystem.addStoredBlock: blockMap updated: ***.**.**.109:50010 is added to blk_4530508892235409778_6626487 size 778943
2009-06-07 20:07:37,834 INFO org.apache.hadoop.hdfs.server.namenode.FSNamesystem.audit: ugi=*****,***** ip=/***.**.**.109 cmd=create src=/hbase/.logs/************,60020,1244418229042/hlog.dat.1244419657833 dst=null
perm=ts:ticker:rw-r--r--
less /home/fds/ts/logs/*datanode*.log.2009-06-07
node 6
2009-06-07 20:07:36,384 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_8054294304707090727_6626586 src: /***.**.**.110:57200 dest: /***.**.**.111:50010
2009-06-07 20:07:36,387 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /***.**.**.110:57200, dest: /***.**.**.111:50010, bytes: 7296, op: HDFS_WRITE, cliID: DFSClient_-930264054, srvID: DS-118466857-***.**.**.111-50
010-1234138704820, blockid: blk_8054294304707090727_6626586
2009-06-07 20:07:36,387 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 1 for block blk_8054294304707090727_6626586 terminating
....
2009-06-07 20:14:28,313 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /***.**.**.111:50010, dest: /***.**.**.110:54059, bytes: 7356, op: HDFS_READ, cliID: DFSClient_-930264054, srvID: DS-118466857-***.**.**.111-500
10-1234138704820, blockid: blk_8054294304707090727_6626586
repeated thousands of times
....
2009-06-07 21:02:02,894 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_8054294304707090727_6626586 file /data/2/hadoop/current/subdir42/blk_8054294304707090727
node 5
2009-06-07 20:07:36,384 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_8054294304707090727_6626586 src: /***.**.**.110:50289 dest: /***.**.**.110:50010
2009-06-07 20:07:36,388 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /***.**.**.110:50289, dest: /***.**.**.110:50010, bytes: 7296, op: HDFS_WRITE, cliID: DFSClient_-930264054, srvID: DS-793422389-***.**.**.110-50
010-1234138704958, blockid: blk_8054294304707090727_6626586
2009-06-07 20:07:36,388 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 2 for block blk_8054294304707090727_6626586 terminating
2009-06-07 20:07:36,394 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /***.**.**.110:50010, dest: /***.**.**.110:50291, bytes: 132, op: HDFS_READ, cliID: DFSClient_-930264054, srvID: DS-793422389-***.**.**.110-5001
0-1234138704958, blockid: blk_8054294304707090727_6626586
node 2
2009-06-07 20:07:36,385 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Receiving block blk_8054294304707090727_6626586 src: /***.**.**.111:35284 dest: /***.**.**.107:50010
2009-06-07 20:07:36,386 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /***.**.**.111:35284, dest: /***.**.**.107:50010, bytes: 7296, op: HDFS_WRITE, cliID: DFSClient_-930264054, srvID: DS-274843024-***.**.**.107-50
010-1234138705859, blockid: blk_8054294304707090727_6626586
2009-06-07 20:07:36,386 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: PacketResponder 0 for block blk_8054294304707090727_6626586 terminating
....
2009-06-07 20:14:03,364 INFO org.apache.hadoop.hdfs.server.datanode.DataNode.clienttrace: src: /***.**.**.107:50010, dest: /***.**.**.110:55798, bytes: 7356, op: HDFS_READ, cliID: DFSClient_-930264054, srvID: DS-274843024-***.**.**.107-500
10-1234138705859, blockid: blk_8054294304707090727_6626586
repeated thousands of times
2009-06-07 21:02:07,245 INFO org.apache.hadoop.hdfs.server.datanode.DataNode: Deleting block blk_8054294304707090727_6626586 file /data/1/hadoop/current/blk_8054294304707090727
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719170#action_12719170 ]
stack commented on HBASE-1495:
------------------------------
Excellent Ryan. Testing now.
Comments on patch:
Why remove this:
{code}
- LOG.debug(location);
{code}
IIRC, its useful emitting when region when its added cache... some hope of figuring whats going on client-side?
The half-file reader fix looks great.
Don't commit the System.out.printlns
You do this:
{code}
- this.row = row;
{code}
Is that right?
Patch mixes in compacting fix but thats fine.
Patch looks good.
Trying it now.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719224#action_12719224 ]
ryan rawson commented on HBASE-1495:
------------------------------------
my patch is bogus. it's out of date now.
I'll be submitting a combined patch to 1513 that fixes Half HFile issues and compaction speed.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HBASE-1495) IllegalArgumentException
in halfhfilereader#next
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719171#action_12719171 ]
Andrew Purtell edited comment on HBASE-1495 at 6/13/09 11:50 AM:
-----------------------------------------------------------------
Fixed up patch for trunk. Testing against HBASE-1500
was (Author: apurtell):
Fixed up patch for trunk.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ryan rawson resolved HBASE-1495.
--------------------------------
Resolution: Fixed
Release Note: Fixed by HBASE-1520, HBASE-1522
feel free to file a new bug if you still experience this.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12719169#action_12719169 ]
Andrew Purtell commented on HBASE-1495:
---------------------------------------
Patch HBASE-1495 does not apply cleanly to trunk. Store.java edits are rejected.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1495:
-------------------------
Attachment: 1495.patch
Attachment fixes two issues: 1. splitting, we were updating .META. using one side of split for both splitA and splitB, and 2., on processing of split removals, we were getting NPE.
Still to do are the following phenomeon:
1. On remove of a info:splitA or info:splitB from a parent regions' row, on the next meta scan, splitA and splitB are missing even though only one side has been removed. Because we think both removed, we'll remove the parent injuring the daughter split still being served.
2. I'm seeing store files that have keys in excess of the start and end row of hosting region.
Still working on 1 and 2.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1495) IllegalArgumentException in
halfhfilereader#next
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-1495:
----------------------------------
Attachment: HBASE-1495.patch
Fixed up patch for trunk.
> IllegalArgumentException in halfhfilereader#next
> ------------------------------------------------
>
> Key: HBASE-1495
> URL: https://issues.apache.org/jira/browse/HBASE-1495
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 0.20.0
>
> Attachments: 1495.patch, HBASE-1495.patch, HBASE-1495.patch
>
>
> From posix4e up on IRC
> {code}
> #
> 2009-06-07 20:22:33,367 ERROR org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction failed for region t3,*******************,1244420117045
> #
> java.lang.IllegalArgumentException
> #
> at java.nio.Buffer.position(Buffer.java:218)
> #
> at org.apache.hadoop.hbase.io.hfile.HFile$Reader$Scanner.next(HFile.java:1072)
> #
> at org.apache.hadoop.hbase.io.HalfHFileReader$1.next(HalfHFileReader.java:108)
> #
> at org.apache.hadoop.hbase.regionserver.StoreFileScanner.next(StoreFileScanner.java:52)
> #
> at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:79)
> #
> at org.apache.hadoop.hbase.regionserver.MinorCompactingStoreScanner.next(MinorCompactingSto
> #
> reScanner.java:101)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:849)
> #
> at org.apache.hadoop.hbase.regionserver.Store.compact(Store.java:714)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:766)
> #
> at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:723)
> #
> at org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.