[jira] [Issue Comment Edited] (HBASE-3842) Refactor Coprocessor Compaction API

["Gary Helmling (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-3842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13075985#comment-13075985 ] 

Gary Helmling edited comment on HBASE-3842 at 8/2/11 12:57 AM:
---------------------------------------------------------------

I think the stacking issue is key here:  are we expecting the common case to be loading a single "CompactionObserver" that overrides the compaction implementation, or loading multiple that each override/customize compaction policy but not necessarily behavior?

I agree on the one hand that having a {{KeyValue}} oriented interface for {{preCompactWrite()}} and {{postCompactWrite()}} may not be sufficient.  At the same time, I don't think we want to force the implementations to write their own {{StoreFiles}} though, as that will be massively inefficient -- for N loaded coprocessors this becomes N compactions being written (assuming we bypass the core compaction code at the end of chaining).

One alternative would be to have {{preCompact}} take the scanner to be used as a parameter, as suggested, and return a scanner instance that would allow overriding policy and mutating KVs, while still relying on the core writer functionality.  This would allow wrapping the store scanner with a custom scanner that inspects and emits KVs as needed on the fly.  In this case, {{preCompact}} would look like:

{code}
StoreScanner preCompact(ObserverContext<~> context, Store store, StoreScanner scanner);
{code}

Wrapping the scanner seems much easier for chaining multiple observers.  On the other hand we lose the clean {{boolean}} return to indicate that core compaction processing should be skipped.  Are there cases that would still want to handling the store file writing portion of the implementation entirely in the coprocessor?  If so, can we still emit a flag to skip normal processing another way?  We could skip normal processing if {{null}} is returned.  Seems a little clunky, but it could work with appropriate documentation.

      was (Author: ghelmling):
    I think the stacking issue is key here:  are we expecting the common case to be loading a single "CompactionObserver" that overrides the compaction implementation, or loading multiple that each override/customize compaction policy but not necessarily behavior?

I agree on the one hand that having a {{KeyValue}} oriented interface for {{preCompactWrite()}} and {{postCompactWrite()}} may not be sufficient.  At the same time, I don't think we want to force the implementations to write their own {{StoreFiles}} though, as that will be massively inefficient -- for N loaded coprocessors this becomes N compactions being written (assuming we bypass the core compaction code at the end of chaining).

One alternative would be to have {{preCompact}} take the scanner to be used as a parameter, as suggested, and return a scanner instance that would allow overriding policy and mutating KVs, while still relying on the core writer functionality.  This would allow wrapping the store scanner with a custom scanner that inspects and emits KVs as needed on the fly.  In this case, {{preCompact}} would look like:

{{code}}
StoreScanner preCompact(ObserverContext<~> context, Store store, StoreScanner scanner);
{{code}}

Wrapping the scanner seems much easier for chaining multiple observers.  On the other hand we lose the clean {{boolean}} return to indicate that core compaction processing should be skipped.  Are there cases that would still want to handling the store file writing portion of the implementation entirely in the coprocessor?  If so, can we still emit a flag to skip normal processing another way?  We could skip normal processing if {{null}} is returned.  Seems a little clunky, but it could work with appropriate documentation.
  
> Refactor Coprocessor Compaction API
> -----------------------------------
>
>                 Key: HBASE-3842
>                 URL: https://issues.apache.org/jira/browse/HBASE-3842
>             Project: HBase
>          Issue Type: Improvement
>          Components: coprocessors, regionserver
>    Affects Versions: 0.92.0
>            Reporter: Nicolas Spiegelberg
>            Assignee: Nicolas Spiegelberg
>              Labels: compaction
>             Fix For: 0.92.0
>
>
> After HBASE-3797, the compaction logic flow has been significantly altered.  Because of this, the current compaction coprocessor API is insufficient for gaining full insight into compaction requests/results.  Refactor coprocessor API after HBASE-3797 is committed to be more extensible and increase visibility.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira