You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2008/10/28 23:20:46 UTC
[jira] Created: (HBASE-969) Won't split under load
Won't split under load
----------------------
Key: HBASE-969
URL: https://issues.apache.org/jira/browse/HBASE-969
Project: Hadoop HBase
Issue Type: Bug
Reporter: stack
Fix For: 0.19.0
Looks like a new bug where we won't split when under load. Some recent refactoring seems to have dropped our split provoker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-969) Won't when storefile > 2G (WAS ->
Won't split under load)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-969.
-------------------------
Resolution: Fixed
Committed.
> Won't when storefile > 2G (WAS -> Won't split under load)
> ---------------------------------------------------------
>
> Key: HBASE-969
> URL: https://issues.apache.org/jira/browse/HBASE-969
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: badmath.patch
>
>
> Looks like a new bug where we won't split when under load. Some recent refactoring seems to have dropped our split provoker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-969) Won't when storefile > 2G (WAS -> Won't
split under load)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-969:
------------------------
Summary: Won't when storefile > 2G (WAS -> Won't split under load) (was: Won't split under load)
> Won't when storefile > 2G (WAS -> Won't split under load)
> ---------------------------------------------------------
>
> Key: HBASE-969
> URL: https://issues.apache.org/jira/browse/HBASE-969
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: badmath.patch
>
>
> Looks like a new bug where we won't split when under load. Some recent refactoring seems to have dropped our split provoker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-969) Won't split under load
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-969:
------------------------
Priority: Blocker (was: Major)
Adding to 0.19.0. Need to check it. In a subsequent run, regions split fine. Need to try and recreate.
> Won't split under load
> ----------------------
>
> Key: HBASE-969
> URL: https://issues.apache.org/jira/browse/HBASE-969
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.19.0
>
>
> Looks like a new bug where we won't split when under load. Some recent refactoring seems to have dropped our split provoker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-969) Won't split under load
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643745#action_12643745 ]
stack commented on HBASE-969:
-----------------------------
Regionserver eventually OOME'd, probably because store got so big.
{code}
2008-10-29 23:03:02,306 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Completed compaction of 1118057853/info store size is 48.3g; time since last major compaction: 5834 seconds
2008-10-29 23:03:02,420 WARN org.apache.hadoop.hbase.regionserver.HStore: Failed getting store size for 1118057853/info
java.io.EOFException: End of file reached before reading fully.
at org.apache.hadoop.fs.FSInputStream.readFully(FSInputStream.java:68)
at org.apache.hadoop.fs.FSDataInputStream.readFully(FSDataInputStream.java:51)
at org.apache.hadoop.hbase.io.BlockFSInputStream.blockSeekTo(BlockFSInputStream.java:163)
at org.apache.hadoop.hbase.io.BlockFSInputStream.read(BlockFSInputStream.java:124)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at org.apache.hadoop.io.SequenceFile$Reader.readRecordLength(SequenceFile.java:1898)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1928)
at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:1833)
at org.apache.hadoop.io.MapFile$Reader.finalKey(MapFile.java:391)
at org.apache.hadoop.hbase.regionserver.HStore.checkSplit(HStore.java:1886)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:917)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:902)
at org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:860)
{code}
> Won't split under load
> ----------------------
>
> Key: HBASE-969
> URL: https://issues.apache.org/jira/browse/HBASE-969
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.19.0
>
>
> Looks like a new bug where we won't split when under load. Some recent refactoring seems to have dropped our split provoker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-969) Won't split under load
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-969:
------------------------
Attachment: badmath.patch
What was actually producing the exception was bad math; when integer math overflowed int, we were trying to do negative offsets.
> Won't split under load
> ----------------------
>
> Key: HBASE-969
> URL: https://issues.apache.org/jira/browse/HBASE-969
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.19.0
>
> Attachments: badmath.patch
>
>
> Looks like a new bug where we won't split when under load. Some recent refactoring seems to have dropped our split provoker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-969) Won't split under load
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack reassigned HBASE-969:
---------------------------
Assignee: stack
> Won't split under load
> ----------------------
>
> Key: HBASE-969
> URL: https://issues.apache.org/jira/browse/HBASE-969
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.19.0
>
>
> Looks like a new bug where we won't split when under load. Some recent refactoring seems to have dropped our split provoker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-969) Won't split under load
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643692#action_12643692 ]
stack commented on HBASE-969:
-----------------------------
Just saw the 'no split' happen. One thing to notice is that with our new batch uploading, we're overwhelming compaction. I'm seeing long-running compactions up in the minutes again. Maybe our algorithm needs tuning -- if lots of store files, pick up more than the smallest -- especially if all close in size as is what I'm seeing here (flushes are 128M, 2x64M configured size). Need to still figure why no split is happening:
{code}
2008-10-29 23:03:02,663 DEBUG org.apache.hadoop.hbase.regionserver.HStore: Compaction size of 1118057853/info: 16.4g; Skipped 139 files , size: 12677739481
{code}
> Won't split under load
> ----------------------
>
> Key: HBASE-969
> URL: https://issues.apache.org/jira/browse/HBASE-969
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Priority: Blocker
> Fix For: 0.19.0
>
>
> Looks like a new bug where we won't split when under load. Some recent refactoring seems to have dropped our split provoker.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.