You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Lars Hofhansl (JIRA)" <ji...@apache.org> on 2012/07/19 06:29:34 UTC
[jira] [Created] (HBASE-6428) Pluggable Compaction policies
Lars Hofhansl created HBASE-6428:
------------------------------------
Summary: Pluggable Compaction policies
Key: HBASE-6428
URL: https://issues.apache.org/jira/browse/HBASE-6428
Project: HBase
Issue Type: New Feature
Reporter: Lars Hofhansl
For some usecases is useful to allow more control over how KVs get compacted.
For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
In addition these date ranged HFile can be easily used for backups while maintaining historical data.
This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481118#comment-13481118 ]
Lars Hofhansl commented on HBASE-6428:
--------------------------------------
I would like to revive this issue. If Store.createWriterInTmp and Store.completeCompaction were public and Store.completeCompaction would allow also a list of new writers to be passed in, it would be possible a "poor man's tiered compaction" like this:
RegionObserver.preCompact(Store, Scanner) would traverse through the scanner, and write the (say) 3 newest version of each KV into a writer and all others into another writer, (or one could tier based on multiple ranges of timestamp and hence keep historical data separate from current data).
Upon completion it would call completeCompaction passing all created writers.
RegionObserver.preCompact would probably also need to know the compaction type (so that this tiering could be different between major and minor compactions).
Does this make sense? I have a need to store historical data life in HBase for occasional historical queries, but would like to avoid a big impact to scanners who only care for the newest KVs.
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419695#comment-13419695 ]
Lars Hofhansl commented on HBASE-6428:
--------------------------------------
That is an excellent point.
Should also think about HBASE-6427 with this in mind.
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482879#comment-13482879 ]
Lars Hofhansl commented on HBASE-6428:
--------------------------------------
I'll have a demo patch of what I have in mind in a bit.
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420741#comment-13420741 ]
Andrew Purtell commented on HBASE-6428:
---------------------------------------
Is there a use case where there could be multiple receivers for a getSmallestReadPoint call? Where you want that kind of stacking, the CP API upcall approach is good; where you have a global behavior that you might want to override, a plug makes sense. For consistency's sake, those pluggable points should all work the same way. We have HBASE-4050's ServiceLoader, but I think we should also look at Guice (HBASE-6407).
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419279#comment-13419279 ]
Andrew Purtell commented on HBASE-6428:
---------------------------------------
bq. For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data. In addition these date ranged HFile can be easily used for backups while maintaining historical data.
I'd be curious if you think the Coprocessor API for compactions cannot be reworked to handle this.
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421182#comment-13421182 ]
Lars Hofhansl commented on HBASE-6428:
--------------------------------------
Yep. I think we agree that the CP framework is good match for this issue (HBASE-6428) and then HBASE-6427 needs a different approach. The CP might need a bit more information in order to be able to determine the desired location of new HFiles, but otherwise "should" (famous last words) be good.
Speaking of CPs. I find that the apis for flushes and compaction are different in the sense that pre/post flush is per region whereas pre/postCompaction is per store.
Can't change that now, but maybe there should be a {pre|post}StoreFlush CP hook that is passed the store and the scanner, so it can interfere with flush in the same it can with a compaction.
For HBASE-6427 I was thinking something simple like RegionSplitPolicy, which just loads a class by reflection.
A generalized plugin loading mechanism would be nice, though.
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424485#comment-13424485 ]
Lars Hofhansl commented on HBASE-6428:
--------------------------------------
To be complete, this needs the new postFlush hook suggested in HBASE-6427.
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6428) Pluggable Compaction policies
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420741#comment-13420741 ]
Andrew Purtell edited comment on HBASE-6428 at 7/23/12 4:18 PM:
----------------------------------------------------------------
Is there a use case where there could be multiple receivers for a getSmallestReadPoint call? Where you want that kind of stacking, the CP API upcall approach is good; where you have a global behavior that you might want to override, a plug makes sense. For consistency's sake, those pluggable points should all work the same way. We have HBASE-4050's ServiceLoader, but I think we should also look at Guice (HBASE-6407).
Edit: Another consideration is performance. I'd like to draw a distinction between coarse grained extension and fine grained extension. The former is where large sections of functionality are replaced, for example using CPs to replace default compaction selection with another strategy. The latter certainly applies to single method calls on hot code paths. For the latter, ideally we want an extension mechanism that will still allow the JVM to inline what's plugged in.
was (Author: apurtell):
Is there a use case where there could be multiple receivers for a getSmallestReadPoint call? Where you want that kind of stacking, the CP API upcall approach is good; where you have a global behavior that you might want to override, a plug makes sense. For consistency's sake, those pluggable points should all work the same way. We have HBASE-4050's ServiceLoader, but I think we should also look at Guice (HBASE-6407).
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13481639#comment-13481639 ]
Andrew Purtell commented on HBASE-6428:
---------------------------------------
It does make sense, yes.
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13424484#comment-13424484 ]
Lars Hofhansl commented on HBASE-6428:
--------------------------------------
Just looked at the current compaction code.
At the very least the Store.completeCompaction and Store.createWriterInTmp need to be made public. In addition Store.completeCompaction has accept Collection<StoreFile.Writer> instead of just a single StoreFile.Writer.
Need to dig a bit deeper, but these changes should allow a coprocessor to take over compactions, write a new set of store files (maybe leveled, or time based) and hook these new HFiles up with HBase.
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419710#comment-13419710 ]
Lars Hofhansl commented on HBASE-6428:
--------------------------------------
As it stands the Coprocessor could at best function as a post filter. In this case that might be good enough, though. preCompact could skip the normal processing and use the passed internalScanner and then write to multiple store files; for example based on TS.
HBASE-6427 is different in that there I want "unfilter" something, so any coprocessor hooks would have to be deeper in the stack.
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13482163#comment-13482163 ]
Lars Hofhansl commented on HBASE-6428:
--------------------------------------
Hmm... This actually would be much simpler if the entire Compactor class could be pluggable; along with some refactoring of that class into an extensible interface.
I guess that's what HBASE-5334 has in mind.
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6428) Pluggable Compaction policies
Posted by "Lars Hofhansl (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-6428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13418924#comment-13418924 ]
Lars Hofhansl commented on HBASE-6428:
--------------------------------------
Another way of looking at this is a possible policy that considers all HFile in terms of a baseline + changes on top of that baseline.
(For the record: I am not saying that I will do this any time soon, just recording this as an idea).
> Pluggable Compaction policies
> -----------------------------
>
> Key: HBASE-6428
> URL: https://issues.apache.org/jira/browse/HBASE-6428
> Project: HBase
> Issue Type: New Feature
> Reporter: Lars Hofhansl
>
> For some usecases is useful to allow more control over how KVs get compacted.
> For example one could envision storing old versions of a KV separate HFiles, which then rarely have to be touched/cached by queries querying for new data.
> In addition these date ranged HFile can be easily used for backups while maintaining historical data.
> This would be a major change, allowing compactions to provide multiple targets (not just a filter).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira