You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by Andrew Purtell <ap...@apache.org> on 2010/12/07 00:16:33 UTC
Re: [jira] Commented: (HBASE-3312) Region fails to split
I'm catching a flight to Narita, then another to Los Angeles today, and will look at this when I get home.
Just so you don't think I dropped this on jira and ran off. :-)
I also have that OOME issue to look at also, and I wonder if this split problem is related.
- Andy
-----Original Message-----
From: "stack (JIRA)" <ji...@apache.org>
Date: Mon, 6 Dec 2010 14:01:27
To: <ap...@apache.org>
Subject: [jira] Commented: (HBASE-3312) Region fails to split
[ https://issues.apache.org/jira/browse/HBASE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968365#action_12968365 ]
stack commented on HBASE-3312:
------------------------------
This is the formatter for the message you see:
{code}
LOG.info("Split row is not inside region key range or is equal to " +
"startkey: " + Bytes.toString(this.splitrow));
{code}
So, when it logs:
{code}
Split row is not inside region key range or is equal to startkey:
DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF
{code}
... we are actually printing out the proposed 'splitrow' which indeed is same as the start row.
Why do we keep picking the same splitrow over and over though region is large? Must be something dumb we're doing w/ mutliple families or so?
Good on you Andrew.
> Region fails to split
> ---------------------
>
> Key: HBASE-3312
> URL: https://issues.apache.org/jira/browse/HBASE-3312
> Project: HBase
> Issue Type: Bug
> Reporter: Andrew Purtell
> Priority: Critical
> Fix For: 0.90.0
>
>
> "SplitTransaction: Split row is not inside region key range or is equal to startkey".
> I stopped writers after realizing one region of the table was growing unbounded:
> {noformat}
> webtable,DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF,1291557809928.
> deccb20bcbf8e634008cf093105c4fc5.
> stores=3, storefiles=10, storefileSizeMB=6753, memstoreSizeMB=2,
> storefileIndexSizeMB=2
> {noformat}
> In the regionserver log, every compaction of this region fails to split with the following message:
> {noformat}
> 2010-12-05 09:04:50,156 INFO org.apache.hadoop.hbase.regionserver.SplitTransaction:
> Split row is not inside region key range or is equal to startkey:
> DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF
> {noformat}
> However there are many rows in the region:
> {noformat}
> 10/12/05 09:33:33 DEBUG client.HTable$ClientScanner: Advancing internal scanner
> to startKey at 'DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF'
> [...]
> Current count: 258000, row: DE424FBDBD15FF3B3E9D0C3DB149ECD29B0F615B
> Current count: 259000, row: DF27251479D6C91B27AA9B1561070A53011A6D1E
> 10/12/05 09:33:36 DEBUG client.HTable$ClientScanner: Finished with region REGION =>
> {NAME => 'webtable,DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF,
> 1291557809928.deccb20bcbf8e634008cf093105c4fc5.',
> STARTKEY => 'DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF',
> ENDKEY => 'DF76CF458433DB5D0CB2C50042452B296E3721A7',
> ENCODED => deccb20bcbf8e634008cf093105c4fc5, TABLE => {{NAME => 'webtable',
> FAMILIES => [{NAME => 'content', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
> VERSIONS => '2147483647', COMPRESSION => 'LZO', TTL => '2147483647',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
> {NAME => 'info', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
> VERSIONS => '2147483647', COMPRESSION => 'LZO', TTL => '2147483647',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
> {NAME => 'url', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
> VERSIONS => '1', COMPRESSION => 'LZO', TTL => '2147483647',
> BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
> 10/12/05 09:33:36 DEBUG client.HTable$ClientScanner: Advancing internal scanner
> to startKey at 'DF76CF458433DB5D0CB2C50042452B296E3721A7'
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
Re: [jira] Commented: (HBASE-3312) Region fails to split
Posted by Ted Yu <yu...@gmail.com>.
Andrew:
I guess you don't see the following (from line 1356 in Store.checkSplit())
in your log:
if (LOG.isDebugEnabled()) {
LOG.debug("cannot split because midkey is the same as first or "
+
"last row");
}
Was it possible that the largest StoreFile grew (symmetrically) on both
sides of the midkey (which happened to be the start row) ?
On Mon, Dec 6, 2010 at 3:16 PM, Andrew Purtell <ap...@apache.org> wrote:
> I'm catching a flight to Narita, then another to Los Angeles today, and
> will look at this when I get home.
>
> Just so you don't think I dropped this on jira and ran off. :-)
>
> I also have that OOME issue to look at also, and I wonder if this split
> problem is related.
>
> - Andy
>
> -----Original Message-----
> From: "stack (JIRA)" <ji...@apache.org>
> Date: Mon, 6 Dec 2010 14:01:27
> To: <ap...@apache.org>
> Subject: [jira] Commented: (HBASE-3312) Region fails to split
>
>
> [
> https://issues.apache.org/jira/browse/HBASE-3312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968365#action_12968365]
>
> stack commented on HBASE-3312:
> ------------------------------
>
> This is the formatter for the message you see:
>
> {code}
> LOG.info("Split row is not inside region key range or is equal to " +
> "startkey: " + Bytes.toString(this.splitrow));
> {code}
>
> So, when it logs:
>
> {code}
> Split row is not inside region key range or is equal to startkey:
> DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF
> {code}
>
> ... we are actually printing out the proposed 'splitrow' which indeed is
> same as the start row.
>
> Why do we keep picking the same splitrow over and over though region is
> large? Must be something dumb we're doing w/ mutliple families or so?
>
> Good on you Andrew.
>
>
>
> > Region fails to split
> > ---------------------
> >
> > Key: HBASE-3312
> > URL: https://issues.apache.org/jira/browse/HBASE-3312
> > Project: HBase
> > Issue Type: Bug
> > Reporter: Andrew Purtell
> > Priority: Critical
> > Fix For: 0.90.0
> >
> >
> > "SplitTransaction: Split row is not inside region key range or is equal
> to startkey".
> > I stopped writers after realizing one region of the table was growing
> unbounded:
> > {noformat}
> > webtable,DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF,1291557809928.
> > deccb20bcbf8e634008cf093105c4fc5.
> > stores=3, storefiles=10, storefileSizeMB=6753,
> memstoreSizeMB=2,
> > storefileIndexSizeMB=2
> > {noformat}
> > In the regionserver log, every compaction of this region fails to split
> with the following message:
> > {noformat}
> > 2010-12-05 09:04:50,156 INFO
> org.apache.hadoop.hbase.regionserver.SplitTransaction:
> > Split row is not inside region key range or is equal to startkey:
> > DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF
> > {noformat}
> > However there are many rows in the region:
> > {noformat}
> > 10/12/05 09:33:33 DEBUG client.HTable$ClientScanner: Advancing internal
> scanner
> > to startKey at 'DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF'
> > [...]
> > Current count: 258000, row: DE424FBDBD15FF3B3E9D0C3DB149ECD29B0F615B
> > Current count: 259000, row: DF27251479D6C91B27AA9B1561070A53011A6D1E
> > 10/12/05 09:33:36 DEBUG client.HTable$ClientScanner: Finished with region
> REGION =>
> > {NAME => 'webtable,DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF,
> > 1291557809928.deccb20bcbf8e634008cf093105c4fc5.',
> > STARTKEY => 'DE0CBA1D6CDFCDD6CBC1065D2C9C1CA17BDA0FAF',
> > ENDKEY => 'DF76CF458433DB5D0CB2C50042452B296E3721A7',
> > ENCODED => deccb20bcbf8e634008cf093105c4fc5, TABLE => {{NAME =>
> 'webtable',
> > FAMILIES => [{NAME => 'content', BLOOMFILTER => 'NONE', REPLICATION_SCOPE
> => '0',
> > VERSIONS => '2147483647', COMPRESSION => 'LZO', TTL => '2147483647',
> > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
> > {NAME => 'info', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
> > VERSIONS => '2147483647', COMPRESSION => 'LZO', TTL => '2147483647',
> > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'},
> > {NAME => 'url', BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0',
> > VERSIONS => '1', COMPRESSION => 'LZO', TTL => '2147483647',
> > BLOCKSIZE => '65536', IN_MEMORY => 'false', BLOCKCACHE => 'false'}]}}
> > 10/12/05 09:33:36 DEBUG client.HTable$ClientScanner: Advancing internal
> scanner
> > to startKey at 'DF76CF458433DB5D0CB2C50042452B296E3721A7'
> > {noformat}
>
> --
> This message is automatically generated by JIRA.
> -
> You can reply to this email to add a comment to the issue online.
>
>