You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hbase.apache.org by "Jonathan Gray (JIRA)" <ji...@apache.org> on 2009/07/07 09:30:15 UTC
[jira] Created: (HBASE-1620) Caching blocks during major compaction
when we should not
Caching blocks during major compaction when we should not
---------------------------------------------------------
Key: HBASE-1620
URL: https://issues.apache.org/jira/browse/HBASE-1620
Project: Hadoop HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.20.0
Reporter: Jonathan Gray
Assignee: Jonathan Gray
Fix For: 0.20.0
Should not cache blocks during major compactions like with minor compactions. Patch tomorrow.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1620) Need to use special StoreScanner
constructor for major compactions (passed sf, no caching, etc)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-1620:
-------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Thanks for patch Jon.
> Need to use special StoreScanner constructor for major compactions (passed sf, no caching, etc)
> -----------------------------------------------------------------------------------------------
>
> Key: HBASE-1620
> URL: https://issues.apache.org/jira/browse/HBASE-1620
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1620-v1.patch
>
>
> Should not cache blocks during major compactions like with minor compactions.
> Also, need to only work on passed StoreFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1620) Need to use special StoreScanner
constructor for major compactions (passed sf, no caching, etc)
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1620:
---------------------------------
Description:
Should not cache blocks during major compactions like with minor compactions.
Also, need to only work on passed StoreFiles.
was:Should not cache blocks during major compactions like with minor compactions. Patch tomorrow.
Summary: Need to use special StoreScanner constructor for major compactions (passed sf, no caching, etc) (was: Caching blocks during major compaction when we should not)
> Need to use special StoreScanner constructor for major compactions (passed sf, no caching, etc)
> -----------------------------------------------------------------------------------------------
>
> Key: HBASE-1620
> URL: https://issues.apache.org/jira/browse/HBASE-1620
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
>
> Should not cache blocks during major compactions like with minor compactions.
> Also, need to only work on passed StoreFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1620) Need to use special StoreScanner
constructor for major compactions (passed sf, no caching, etc)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728260#action_12728260 ]
stack commented on HBASE-1620:
------------------------------
Jon explained whats going on here to me in IRC.
Adds constructor that takes the scanners setup doing major compaction rather than create new ones. The new ones will not have the "do not cache" flag set.
> Need to use special StoreScanner constructor for major compactions (passed sf, no caching, etc)
> -----------------------------------------------------------------------------------------------
>
> Key: HBASE-1620
> URL: https://issues.apache.org/jira/browse/HBASE-1620
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1620-v1.patch
>
>
> Should not cache blocks during major compactions like with minor compactions.
> Also, need to only work on passed StoreFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1620) Need to use special StoreScanner
constructor for major compactions (passed sf, no caching, etc)
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1620:
---------------------------------
Attachment: HBASE-1620-v1.patch
Adds new StoreScanner constructor that takes a list of StoreFiles.
Other changes include not adding to changed readers observer list and we automatically will not cache blocks now since we use the right thing.
> Need to use special StoreScanner constructor for major compactions (passed sf, no caching, etc)
> -----------------------------------------------------------------------------------------------
>
> Key: HBASE-1620
> URL: https://issues.apache.org/jira/browse/HBASE-1620
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1620-v1.patch
>
>
> Should not cache blocks during major compactions like with minor compactions.
> Also, need to only work on passed StoreFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1620) Need to use special StoreScanner
constructor for major compactions (passed sf, no caching, etc)
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728197#action_12728197 ]
stack commented on HBASE-1620:
------------------------------
Whats this patch adding?
You have yet to add the observer?
> Need to use special StoreScanner constructor for major compactions (passed sf, no caching, etc)
> -----------------------------------------------------------------------------------------------
>
> Key: HBASE-1620
> URL: https://issues.apache.org/jira/browse/HBASE-1620
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1620-v1.patch
>
>
> Should not cache blocks during major compactions like with minor compactions.
> Also, need to only work on passed StoreFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1620) Need to use special StoreScanner
constructor for major compactions (passed sf, no caching, etc)
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1620:
---------------------------------
Status: Patch Available (was: Open)
Also does not add to changed readers observer list. We don't want to add newly flushed storefiles midway through compaction. In the future, we could snapshot the memcache and include it in the major compaction. Would be a good way to help keep storefiles low.
Ready for commit.
> Need to use special StoreScanner constructor for major compactions (passed sf, no caching, etc)
> -----------------------------------------------------------------------------------------------
>
> Key: HBASE-1620
> URL: https://issues.apache.org/jira/browse/HBASE-1620
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1620-v1.patch
>
>
> Should not cache blocks during major compactions like with minor compactions.
> Also, need to only work on passed StoreFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1620) Need to use special StoreScanner
constructor for major compactions (passed sf, no caching, etc)
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728549#action_12728549 ]
ryan rawson commented on HBASE-1620:
------------------------------------
it is possible this patch ups the perf of hbase... I now see smaller GC pauses, and I am achieving higher hbase perf than before.
> Need to use special StoreScanner constructor for major compactions (passed sf, no caching, etc)
> -----------------------------------------------------------------------------------------------
>
> Key: HBASE-1620
> URL: https://issues.apache.org/jira/browse/HBASE-1620
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1620-v1.patch
>
>
> Should not cache blocks during major compactions like with minor compactions.
> Also, need to only work on passed StoreFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-1620) Need to use special StoreScanner
constructor for major compactions (passed sf, no caching, etc)
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12728746#action_12728746 ]
Jonathan Gray commented on HBASE-1620:
--------------------------------------
Definitely possible, though it's really from HBASE-1597 and then the reimplementation in HBASE-1615 not this issue (which is for major).
I'm unsure why this would not have impacted the old LRU... I would expect the new one to be faster not slower.
> Need to use special StoreScanner constructor for major compactions (passed sf, no caching, etc)
> -----------------------------------------------------------------------------------------------
>
> Key: HBASE-1620
> URL: https://issues.apache.org/jira/browse/HBASE-1620
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1620-v1.patch
>
>
> Should not cache blocks during major compactions like with minor compactions.
> Also, need to only work on passed StoreFiles.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-1620) Caching blocks during major compaction
when we should not
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Gray updated HBASE-1620:
---------------------------------
Priority: Blocker (was: Major)
Broadening scope and making a blocker. There is at least one nasty bug in major compactions.
> Caching blocks during major compaction when we should not
> ---------------------------------------------------------
>
> Key: HBASE-1620
> URL: https://issues.apache.org/jira/browse/HBASE-1620
> Project: Hadoop HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.20.0
> Reporter: Jonathan Gray
> Assignee: Jonathan Gray
> Priority: Blocker
> Fix For: 0.20.0
>
>
> Should not cache blocks during major compactions like with minor compactions. Patch tomorrow.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.