You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "stack (JIRA)" <ji...@apache.org> on 2009/08/12 02:40:14 UTC

[jira] Created: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
-----------------------------------------------------------------------------------------------------------------------

                 Key: HBASE-1761
                 URL: https://issues.apache.org/jira/browse/HBASE-1761
             Project: Hadoop HBase
          Issue Type: Bug
            Reporter: stack
             Fix For: 0.20.0


getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.

Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743545#action_12743545 ] 

stack commented on HBASE-1761:
------------------------------

Ryan wants to run some tests.  Assigning issue to him (Assignment will expire soon since he's off on holidays).

I ran tests on my small cluster to ensure this doesn't slow things down.  It doesn't.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v10.patch, 1761-v11.patch, 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761-v9.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761.patch

Unfinished patch

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743731#action_12743731 ] 

stack commented on HBASE-1761:
------------------------------

Hmm. You are right on the early-out not working.  Let me fix that.  I think its something we want.  Let me add:

if catalog table, parse key for table and row component and do compares minus region id component.  A key only becomes a candidate if its from requested table.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v10.patch, 1761-v11.patch, 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761-v9.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v2.patch

Memstore refactoring done with some tests passing.  Now to go at StoreFiles.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v2.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12743556#action_12743556 ] 

ryan rawson commented on HBASE-1761:
------------------------------------

some issues:

Store.java:1066 (in v9):

        for (Map.Entry<Long, StoreFile> e: m.entrySet()) {
          // Update the candidate keys from the current map file
          rowAtOrBeforeFromStoreFile(e.getValue(), state);
          // Early out if we can.
          if (state.hasMatch()) break;
        }


The state.hasMatch looks like so:

  boolean hasMatch() {
    if (!hasCandidate()) return false;
    return this.kvcomparator.compareRows(getCandidate(), this.targetkey) == 0;
  }


this will never return true. ever.  the targetKey is 'table,row,9999999...'.  Thus it can never match anything ever.

So the early out doesn't work. do we want early out?  If we changed it to be 'hasMatch() { return hasCandidate(); }', then the problem i described where
we can return a candidate from a file that doesnt have any entries for our table at all.  That'd be bad obviously.

If we have the following hfiles in sequence order:
- hfile contains entries for tables 'a,b,c' but not 'target_table'
- hfile contains entries for 'target_table'.

then we could id something from 'c' as the candidate.  if we short circuted then we'd be in trouble.

We might need to to constructor a concept of 'first on table' row key, but I don't know yet.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v10.patch, 1761-v11.patch, 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761-v9.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Assigned: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jean-Daniel Cryans reassigned HBASE-1761:
-----------------------------------------

    Assignee: stack

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>            Assignee: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v10.patch, 1761-v11.patch, 1761-v12.patch, 1761-v13.patch, 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761-v9.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v11.patch

Fix another issue found by Ryan's IDE

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v10.patch, 1761-v11.patch, 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761-v9.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v9.patch

Great review by Ryan (on IRC).

I had some pollution from another patch.   He also made me clean up stuff like remove the incorrectly named cloneRow and use constructors instead.  All tests now pass for me.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761-v9.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v12.patch

Adding new tests.

Removing optimization that would allow early-out.  After addressing Ryan comment, turns out that only if client asks for first row in region can we early-out.  getClosest needs to carry around the HRegionInfo so can know if candidate is within a regions name.  Then can early-out.

Added test to see if we are within the target table so we know when we back up into another.

Not done yet.  Need to fill out a few more tests.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v10.patch, 1761-v11.patch, 1761-v12.patch, 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761-v9.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v10.patch

Add missing files.


> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v10.patch, 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761-v9.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v4.patch

This patch works.  Review please though its kinda tough to do.

Added tests.  It simplifies the whole getClosestAtOrBefore cutting out loads of superfluous (seeming) code.  Now we don't keep open a wide range of possible candidates and deletes as we worm down through memstore and storefiles, instead we go for the wanted row and stop trying if we go past current candidate.  We only replace candidate if new one is better (close).  

Tries to make the logic same in memstore as it is in store for storefile but the scanner in hfile fights against us.  We need to flip it so it scans to asked-for-row or just after rather than just before as it does now..  The just before was done to support getClosest originally and it was optimal when old style deletes where a delete shadowed the thing it was deleting -- same r/c/ts coordinates -- but with new deletes where delete family is first-in-row, then we have to back up too far to be sure we get all elements in the row.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v5.patch

Includes early-out optimzation.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742910#action_12742910 ] 

ryan rawson commented on HBASE-1761:
------------------------------------

i will review this in depth today. Get ready to cry :-)

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742146#action_12742146 ] 

stack commented on HBASE-1761:
------------------------------

>From Bradford:

{code}
"org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to
contact region server Some server for region , row
'E3652782193BC8D66A0BA1629D0FAAAB', but failed after 11 attempts.
Exceptions:
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.
java.io.IOException: HRegionInfo was null or empty in .META.

      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionLocationForRowWithRetries(HConnectionManager.java:995)
      at org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1025)
      at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:584)
      at org.apache.hadoop.hbase.client.HTable.put(HTable.java:450)
      at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1738)
      at org.apache.hadoop.hbase.client.HTable.commit(HTable.java:1719)
      at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:74)
      at org.apache.hadoop.hbase.mapred.TableOutputFormat$TableRecordWriter.write(TableOutputFormat.java:54)
      at org.apache.hadoop.mapred.MapTask$DirectMapOutputCollector.collect(MapTask.java:576)
      at com.visible.generatefromcsv.FromCSVMapper.map(FromCSVMapper.java:81)
      at com.visible.generatefromcsv.FromCSVMapper.map(FromCSVMapper.java:1)
      at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
      at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:356)
      at org.apache.hadoop.mapred.MapTask.run(MapTask.java:305)
      at org.apache.hadoop.mapred.Child.main(Child.java:170)
{code}

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v13.patch

Will apply this patch in an hour or so unless objections.

This last version adds more testing to the new testclosest junit class.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v10.patch, 1761-v11.patch, 1761-v12.patch, 1761-v13.patch, 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761-v9.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v7.patch

Fix for failing TestEmptyMetaInfo test.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v6.patch

Cleanup.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "Erik Holstad (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742891#action_12742891 ] 

Erik Holstad commented on HBASE-1761:
-------------------------------------

Just some small comments on the patch:
HRegionServer.getClosestrowBefore()
1. We should probably add a family check, like the one we have in the other get//put methods.

GetClosestRowBEforeTracker.addDelete()
1. Could add a check to see if the kv is "bigger than the current candidate, if not don't add it

GetClosestRowBEforeTracker.isDeleted(KeyValue, NavigableSet)
1. Could move the kvts out of the for loop, same thing with the get calls from kv, like getQualifierOffset() etc
2. The line under //check column is a little bit long

The only other thing I would add is to have a structure more like in the getDeleteTracker where we have to List that we merge, which I think would work,
instead of the Map<Map>. But hopefully we can remove all of this if/when we move Meta to ZooKeeper.

Other than that I think it looks really good.

Regards Erik


> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Resolved: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack resolved HBASE-1761.
--------------------------

      Resolution: Fixed
    Hadoop Flags: [Reviewed]

Committed trunk and branch.  Here's commit message:

{code}
Rewritten getclosestatorbefore that works with new style deletes (delete
family, delete column and just delete.  Doesn't carry as much load as it
goes from memstore across storefiles (e.g. only one candidate is
carried rather than all as previous) and always goes for the target row
rather than zig-zag in so should perform a bit better.  Includes fix
that should make it so we no longer return rows from wrong table.  Also
add flush on drop of table.

M src/test/org/apache/hadoop/hbase/HBaseTestCase.java
  Formatting.
A src/test/org/apache/hadoop/hbase/regionserver/TestGetClosestAtOrBefore.java
  Added testing of getClosestAtOrBefore.  Most of this test was in hbase before
  hbase-1304 went in.  Also dropped what was in TestMemStore testing 
  getClosest and moved it here instead.
M src/test/org/apache/hadoop/hbase/regionserver/TestMemStore.java
  Made a little more stringent asserting row count same as column count.
  Added new test to assert we can see new values as they come in as we scan
  (More hbase-1738).
  MemStore constructor no longer takes ttl.  Removed the testGetClosest from
  here and moved up into TestGetClosestAtOrBefore.
M src/test/org/apache/hadoop/hbase/regionserver/TestHRegion.java
  Remove unused imports.
M src/test/org/apache/hadoop/hbase/regionserver/TestCompaction.java
  Formatting.
A src/java/org/apache/hadoop/hbase/regionserver/GetClosestRowBeforeTracker.java
  New class that holds getclosest state and utility.  Like ScanDeleteTracker
  and GetDeleteTracker but doens't implement the DeleteTracker interface
  as it needs to span rows.
M src/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
  Don't allocate ArrayList unless we're going to use it.
M  src/java/org/apache/hadoop/hbase/regionserver/GetDeleteTracker.java
  Fix the isEmpty.
M src/java/org/apache/hadoop/hbase/regionserver/MemStore.java
  No longer takes a ttl in constructor (It now lives in GetClosestRowBeforeTracker).
  Reimplemented getClosestRowAtOrBefore.
M  src/java/org/apache/hadoop/hbase/regionserver/HRegion.java
  Reimplemented getClosestRowAtOrBefore.
M  src/java/org/apache/hadoop/hbase/regionserver/Store.java
  MemStore doesn't take a ttl.  Reimplemented getClosestRowAtOrBefore.
M  src/java/org/apache/hadoop/hbase/KeyValue.java
  Added comparator that only looks at row portion of a KV.  Useful for
  case where want to use kv rather than extract its row as row key in map.
  Changed the type on KVs that do not have values to be Type.Maximum by default.
  (Was Type.Put by default).  Removed cloneRow.  Not a clone as Ryan pointed
  out and can use constructor instead.
  (isDelete, isDeleteFamily): Added.
M src/java/org/apache/hadoop/hbase/util/Bytes.java
  Formatting.
M src/java/org/apache/hadoop/hbase/client/HConnectionManager.java
  Log message.
M src/java/org/apache/hadoop/hbase/client/Put.java
  Was allocating a Set twice on each Put deserialization.
M  bin/HBase.rb
  Flush before major compacting when dropping table.
{code}

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v10.patch, 1761-v11.patch, 1761-v12.patch, 1761-v13.patch, 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761-v5.patch, 1761-v6.patch, 1761-v7.patch, 1761-v9.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Commented: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
    [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12742743#action_12742743 ] 

stack commented on HBASE-1761:
------------------------------

TODO: One optimization would be to stop searching if we hit jackpot -- if we turn up the wanted row as a candidate in memstore or in an early storefile.

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v2.patch, 1761-v3.patch, 1761-v4.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


[jira] Updated: (HBASE-1761) getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem

Posted by "stack (JIRA)" <ji...@apache.org>.
     [ https://issues.apache.org/jira/browse/HBASE-1761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

stack updated HBASE-1761:
-------------------------

    Attachment: 1761-v3.patch

Still not done (bit of a pain --- NavigableSets go to asked for row or just after... HFileScanner goes to asked for row or just before).

> getclosest doesn't understand delete family; manifests as "HRegionInfo was null or empty in .META" A.K.A the BS problem
> -----------------------------------------------------------------------------------------------------------------------
>
>                 Key: HBASE-1761
>                 URL: https://issues.apache.org/jira/browse/HBASE-1761
>             Project: Hadoop HBase
>          Issue Type: Bug
>            Reporter: stack
>             Fix For: 0.20.0
>
>         Attachments: 1761-v2.patch, 1761-v3.patch, 1761.patch
>
>
> getclosestatorbefore was not converted to deal with the new delete types.  It only knows how to process old style deletes.  Usually all is well as edits come in but its possible to get into state where you have persisted in one file a deletefamily for all in meta and in the file behind it, there are entries on the info family.  Since closest doesn't understand deletefamily, it will return the Put rows only for the subsequent getfull, which knows how to work with deletefamilies fail.
> Once this happens, table is hosed.  Seen on Bradford Stephens upload and at Powerset.  "Fix" is flush and major compact.  Gives impression that hbase is 'delicate'.  Fixing.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.