You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2009/07/07 01:39:14 UTC
[jira] Created: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
HBASE-1597 introduced a bug when compacting after a split
---------------------------------------------------------
Key: HBASE-1615
URL: https://issues.apache.org/jira/browse/HBASE-1615
Project: Hadoop HBase
Issue Type: Bug
Reporter: Jonathan Gray
Assignee: stack
Fix For: 0.20.0
The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1615:
---------------------------------
Attachment: (was: HBASE-1615-v3.patch)
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, HBASE-1615-v4.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1615:
---------------------------------
Attachment: HBASE-1615-v4.patch
Correct filename.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, HBASE-1615-v4.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1615:
-------------------------
Attachment: 1615-v2.patch
Adds logging and toStringBinary on exceptions as well as the CompactionReader fix.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1615:
---------------------------------
Attachment: HBASE-1615-v5.patch
Changes implementation of how we're going to not cache blocks.
No more CompactionReader. We now pass a boolean in when we create the HFile.Reader.Scanner as to whether we want to cache blocks or not. That is passed in to the HFile.Reader.readBlock().
Much more simple and avoids code duplication and unnecessary complexity. If we decide to make more optimizations to the reader or scanner, we will revisit this down the road. Need to stabilize trunk!
Passes new unit test introduced in 1616.
This patch takes out a bunch of debugging stuff added by stack's patches. Only for clarity purposes and so I could track where I was at.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, HBASE-1615-v4.patch, HBASE-1615-v5.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727881#action_12727881 ]
Jonathan Gray commented on HBASE-1615:
--------------------------------------
Issue is, we have a local readBlock in CompactionReader but that the delegate Reader does not use our cacheBlock method.
I think this might be much easier if we just upgrade "block should be cached" to a passed boolean to HFile Reader and Scanner instantation, rather than all this subclassing, delegating, and such.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, HBASE-1615-v4.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack resolved HBASE-1615.
--------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Thanks for patch Jon.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, HBASE-1615-v4.patch, HBASE-1615-v5.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1615:
-------------------------
Attachment: hfile.patch
File that delegates to passed Reader rather than create a new one with attributes copied from the passed in Reader.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727898#action_12727898 ]
stack commented on HBASE-1615:
------------------------------
+1 on patch. Cleaner than new CompactedReader class.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, HBASE-1615-v4.patch, HBASE-1615-v5.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1615:
---------------------------------
Attachment: HBASE-1615-v3.patch
The patch didn't get any prettier.
Issue is that there is a special HFileScanner inside of HalfHFileReader.
This patch breaks that into an actual class (rather than added on construction). And since it's a separate class, you now pass in the HalfHFileReader, and it sets a local top/splitKey.
Also adds HFile.Reader.Scanner instantiation based on a passed in Reader. So now we can have an instantiated Scanner from HFile.Reader.Scanner than takes our subclassed CompactionReader (so that we can call our own cacheBlock method). All other methods are passed along to the delegate (in this case, the HalfHFileReader). But then the HalfHFileScanner goes atop the HFileScanner (which has our CompactionReader as the Reader, passing along to HalfHFileReader).
Yeah i think i repeated myself there. Anyways, it ain't pretty but it seems to work.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, HBASE-1615-v4.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727880#action_12727880 ]
Jonathan Gray commented on HBASE-1615:
--------------------------------------
Hmm, well that patch passes tests but up on cluster it does weird stuff. And I think I'm still seeing stuff get cached.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, HBASE-1615-v4.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727856#action_12727856 ]
Andrew Purtell commented on HBASE-1615:
---------------------------------------
Especially nice are the detailed exception messages. Sorry I didn't catch the reader versus half file reader issue on review of 1597.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1615:
-------------------------
Attachment: 1615-v3.patch
Was being too clever in my test of split key -- wasn't allowing for '' key.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1615) HBASE-1597 introduced a bug when
compacting after a split
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12727890#action_12727890 ]
Jonathan Gray commented on HBASE-1615:
--------------------------------------
My tests also pass with v5 patch.
> HBASE-1597 introduced a bug when compacting after a split
> ---------------------------------------------------------
>
> Key: HBASE-1615
> URL: https://issues.apache.org/jira/browse/HBASE-1615
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Jonathan Gray
> Assignee: stack
> Fix For: 0.20.0
>
> Attachments: 1615-v2.patch, 1615-v3.patch, HBASE-1615-v4.patch, HBASE-1615-v5.patch, hfile.patch
>
>
> The way HBASE-1597 instantiated the CompactionReader ignored whether the Reader it came from was a half reader or not.
> Need to reimplement CompactionReader.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.