You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Karthick Sankarachary (JIRA)" <ji...@apache.org> on 2011/06/10 02:34:00 UTC
[jira] [Created] (HBASE-3976) Disable Block Cache On Compactions
Disable Block Cache On Compactions
----------------------------------
Key: HBASE-3976
URL: https://issues.apache.org/jira/browse/HBASE-3976
Project: HBase
Issue Type: Improvement
Components: regionserver
Affects Versions: 0.90.3
Reporter: Karthick Sankarachary
Assignee: Karthick Sankarachary
Priority: Minor
Fix For: 0.90.4
Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Karthick Sankarachary (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051440#comment-13051440 ]
Karthick Sankarachary commented on HBASE-3976:
----------------------------------------------
If "hbase.rs.cacheblocksonwrite" is used sparingly, then I guess this a non-issue. It'd be nice though to have different knobs to turn block caching on for the "flush" case as opposed to the "compaction" case, considering that the blocks in the snapshop being flushed are probably hotter than those in the files being compacted.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Mikhail Bautin (Reopened) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mikhail Bautin reopened HBASE-3976:
-----------------------------------
Assignee: Mikhail Bautin (was: Nicolas Spiegelberg)
Re-opening until we add a unit test and implement a proper fix.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Mikhail Bautin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448936#comment-13448936 ]
Mikhail Bautin commented on HBASE-3976:
---------------------------------------
Lars: thanks for double-checking this!
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: HBASE-3976.patch, HBASE-3976-unconditional.patch, HBASE-3976-V3.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Karthick Sankarachary (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthick Sankarachary updated HBASE-3976:
-----------------------------------------
Attachment: HBASE-3976-V3.patch
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Blocker
> Fix For: 0.90.4, 0.92.0
>
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3976) Disable Block Cache On Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3976:
-------------------------
Fix Version/s: 0.92.0
Back out of branch and trunk?
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Blocker
> Fix For: 0.90.4, 0.92.0
>
> Attachments: HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13444264#comment-13444264 ]
Lars Hofhansl commented on HBASE-3976:
--------------------------------------
In a typical read/write type load you'd want "hbase.rs.cacheblocksonwrite" on, no?
The data in the memstore is the newest, which will typically be the most interesting data in the future. Without "hbase.rs.cacheblocksonwrite" a memflush will make that data cold and it needs to be loaded into the cache upon the next set of reads.
The same is actually true (to a lesser extend) for compactions: We take the latest data and make it cold.
"hbase.rs.cacheblocksonwrite" actually does not mean much, what writes are we talking about?
I can see see three different options here: (1) cache on flush, (2) cache on minor compaction, and *maybe* (3) cache on major compaction... Maybe that's overkill (especially the last one)?
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: HBASE-3976.patch, HBASE-3976-unconditional.patch, HBASE-3976-V3.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Jason Rutherglen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13046944#comment-13046944 ]
Jason Rutherglen commented on HBASE-3976:
-----------------------------------------
This makes sense! I think for local data blocks this can further be solved by preventing the read side of the compacted blocks from entering the system IO cache, ala LUCENE-2500.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Minor
> Fix For: 0.90.4
>
> Attachments: HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Zhihong Yu (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13182732#comment-13182732 ]
Zhihong Yu commented on HBASE-3976:
-----------------------------------
This was integrated on 17/Jun/11 11:00 and the status is Open.
That's confusing.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13051603#comment-13051603 ]
stack commented on HBASE-3976:
------------------------------
bq. If "hbase.rs.cacheblocksonwrite" is used sparingly...
My guess is that its not used at all, not at least until we commit fix for hbase-3417
bq. It'd be nice though to have different knobs to turn block caching on for the "flush" case as opposed to the "compaction" case, considering that the blocks in the snapshop being flushed are probably hotter than those in the files being compacted.
Yes. Good idea (IIRC, this idea or a version of it has come up before but I don't recall the issue if any to implement -- we should make one).
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3976) Disable Block Cache On Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3976:
-------------------------
Assignee: Nicolas Spiegelberg (was: Karthick Sankarachary)
Assigning Nicolas for review. Punt to me N if you don't have time (np). Thanks.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.4, 0.92.0
>
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Mikhail Bautin (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188784#comment-13188784 ]
Mikhail Bautin commented on HBASE-3976:
---------------------------------------
The last commit I can see for this JIRA is the following:
{quote}
Author: stack <st...@13f79535-47bb-0310-9956-ffa450edef68>
Date: Tue Jun 14 16:47:37 2011
HBASE-3976 Disable Block Cache On Compactions -- REVERT
{quote}
This makes me think this path was actually reverted in trunk. Besides, when Jonathan's HBASE-4422 (CacheConfig) went in, it completely changed the way this feature would be implemented. This is why it is not possible to verify whether the patch has been applied or not anymore from looking at the current version of Store.java in trunk. I can't say off the top of my head whether we cache blocks on write during compactions in HBase trunk these days with the new logic implemented by HBASE-4422 in effect. However, I am planning to write a new unit test for both trunk and 89-fb to verify that, and address this JIRA if necessary in a consistent way across the two branches.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047162#comment-13047162 ]
Andrew Purtell commented on HBASE-3976:
---------------------------------------
+1
Except I don't think it needs be configurable, i.e.
bq. Is there a good reason to believe that caching blocks during compactions is beneficial?
I fail to think of one.
Will commit unless I hear otherwise.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Minor
> Fix For: 0.90.4
>
> Attachments: HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13048317#comment-13048317 ]
Andrew Purtell commented on HBASE-3976:
---------------------------------------
Reverted
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.4, 0.92.0
>
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Nicolas Spiegelberg (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047783#comment-13047783 ]
Nicolas Spiegelberg commented on HBASE-3976:
--------------------------------------------
I don't think this patch is doing what you're expecting. Store.completeCompaction() is at the end of a compaction. You give the new StoreFile a pointer to the BlockCache so it is possible for that file to cache blocks. This StoreFile (with associated reader) is put in Store.storefiles, which is utilized for all gets/scans. It seems like your patch will unconditionally disable the block cache for all StoreFiles that are the result of a compaction.
The actual decision on whether you put scan requests for StoreFiles in the block cache during compaction is handled by StoreFileScanner.getScannersForStoreFiles() in Store.compactStores(). That is unconditionally set to false.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Minor
> Fix For: 0.90.4
>
> Attachments: HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13050990#comment-13050990 ]
Hudson commented on HBASE-3976:
-------------------------------
Integrated in HBase-TRUNK #1976 (See [https://builds.apache.org/job/HBase-TRUNK/1976/])
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3976) Disable Block Cache On Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3976:
-------------------------
Priority: Blocker (was: Minor)
Seems like we need to back out the patch? Made it a blocker so it doesn't get forgotten.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Blocker
> Fix For: 0.90.4, 0.92.0
>
> Attachments: HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047304#comment-13047304 ]
Jean-Daniel Cryans commented on HBASE-3976:
-------------------------------------------
I can't believe we're block caching during compactions, huge +1 from me.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Minor
> Fix For: 0.90.4
>
> Attachments: HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Karthick Sankarachary (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Karthick Sankarachary updated HBASE-3976:
-----------------------------------------
Attachment: HBASE-3976.patch
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Minor
> Fix For: 0.90.4
>
> Attachments: HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047423#comment-13047423 ]
Andrew Purtell commented on HBASE-3976:
---------------------------------------
Committed to trunk and 0.90 branch. Tests pass locally.
Thanks a lot for bringing this to our attention Karthick!
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Minor
> Fix For: 0.90.4
>
> Attachments: HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047907#comment-13047907 ]
Andrew Purtell commented on HBASE-3976:
---------------------------------------
@Nicholas: You're correct, my bad.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.4, 0.92.0
>
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-3976:
----------------------------------
Attachment: HBASE-3976-unconditional.patch
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Minor
> Fix For: 0.90.4
>
> Attachments: HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Nicolas Spiegelberg (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nicolas Spiegelberg resolved HBASE-3976.
----------------------------------------
Resolution: Incomplete
its very hard to tell if this issue is corrected in trunk or not. Talked offline with Mikhail about strategies for getting proper resolution here. HBASE-5230 has been made to verify that this issue is indeed fixed & will encompass a fix if other subtle bugs remain.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047909#comment-13047909 ]
Andrew Purtell commented on HBASE-3976:
---------------------------------------
I have to catch a flight. My apologies again for putting something through rushed last time. Won't make the mistake again. Will pick up upon landing unless already backed out.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.4, 0.92.0
>
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13049527#comment-13049527 ]
stack commented on HBASE-3976:
------------------------------
I applied v3 after looking at it but then reverted it. Its making it so we always skip cache writing out compacted file. But by default we will skip the cache writing the compacted file. There is also a config. for the rare case where we actually do not want to skip writing cache -- this patch was undoing that facility.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Blocker
> Fix For: 0.90.4, 0.92.0
>
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448131#comment-13448131 ]
Lars Hofhansl commented on HBASE-3976:
--------------------------------------
Any comment on my comment :) ?
The use case we're the most interested in is transferring the "hotness" of the memstore to the blockcache (i.e. cache on flush).
If there's interest I'll look into implementing the flags I mentioned in my previous comment.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: HBASE-3976.patch, HBASE-3976-unconditional.patch, HBASE-3976-V3.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Karthick Sankarachary (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13047823#comment-13047823 ]
Karthick Sankarachary commented on HBASE-3976:
----------------------------------------------
My bad. We do want to be able to cache blocks in the reader associated with the compacted file. Moreover, we do disable caching while reading from the files being compacted, as Nicolas pointed out.
Having said that, it looks like we're still caching blocks while writing to the compacted file. In particular, when we create its writer in {{Store#compactStore}}, we don't seem to be disabling the cache, as we should be doing.
To see what I mean, please take a look at the V3 version of the patch. I apologize in advance if this turns out to be a false alarm.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Karthick Sankarachary
> Priority: Blocker
> Fix For: 0.90.4, 0.92.0
>
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lars Hofhansl resolved HBASE-3976.
----------------------------------
Resolution: Fixed
Closing... Please reopen if this should be kept open.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: HBASE-3976.patch, HBASE-3976-unconditional.patch, HBASE-3976-V3.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Mikhail Bautin (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448521#comment-13448521 ]
Mikhail Bautin commented on HBASE-3976:
---------------------------------------
Lars: I agree, cache-on-flush is definitely the most useful. This is what we are now using in production for some workloads.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: HBASE-3976.patch, HBASE-3976-unconditional.patch, HBASE-3976-V3.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3976) Disable Block Cache On Compactions
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13448920#comment-13448920 ]
Lars Hofhansl commented on HBASE-3976:
--------------------------------------
Hmm... Looking at the code in trunk, this is (mostly) what is currently happening anyway.
HStore.createWriterInTmp using the configured cacheOnWrite setting unless this is a compaction (in which case cacheOnWrite is set to false).
There is also a test for this in TestCacheOnWrite.
I think we can close this issue. Agreed?
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Mikhail Bautin
> Priority: Minor
> Attachments: HBASE-3976.patch, HBASE-3976-unconditional.patch, HBASE-3976-V3.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3976) Disable Block Cache On Compactions
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
stack updated HBASE-3976:
-------------------------
Priority: Minor (was: Blocker)
Fix Version/s: (was: 0.90.4)
(was: 0.92.0)
Undid its blocker billing.
> Disable Block Cache On Compactions
> ----------------------------------
>
> Key: HBASE-3976
> URL: https://issues.apache.org/jira/browse/HBASE-3976
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
> Affects Versions: 0.90.3
> Reporter: Karthick Sankarachary
> Assignee: Nicolas Spiegelberg
> Priority: Minor
> Attachments: HBASE-3976-V3.patch, HBASE-3976-unconditional.patch, HBASE-3976.patch
>
>
> Is there a good reason to believe that caching blocks during compactions is beneficial? Currently, if block cache is enabled on a certain family, then every time it's compacted, we load all of its blocks into the (LRU) cache, at the expense of the legitimately hot ones.
> As a matter of fact, this concern was raised earlier in HBASE-1597, which rightly points out that, "we should not bog down the LRU with unneccessary blocks" during compaction. Even though that issue has been marked as "fixed", it looks like it ought to be reopened.
> Should we err on the side of caution and not cache blocks during compactions period (as illustrated in the attached patch)? Or, can we be selectively aggressive about what blocks do get cached during compaction (e.g., only cache those blocks from the recent files)?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira