You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Andrew Purtell (JIRA)" <ji...@apache.org> on 2010/11/22 21:54:13 UTC
[jira] Created: (HBASE-3260) Coprocessors: Lifecycle management
Coprocessors: Lifecycle management
----------------------------------
Key: HBASE-3260
URL: https://issues.apache.org/jira/browse/HBASE-3260
Project: HBase
Issue Type: Sub-task
Reporter: Andrew Purtell
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Gary Helmling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935096#action_12935096 ]
Gary Helmling commented on HBASE-3260:
--------------------------------------
I agree we should standardize the terminology between here and server lifecycle management.
I remember in the RS and master lifecycle discussion seeing a mention of the Google guava Service interface:
http://guava-libraries.googlecode.com/svn/trunk/javadoc/com/google/common/base/Service.html
It uses similar start() and stop() methods, with a very similar set of states, should RS and master go that route.
I'm hesitant to bring in anything too heavy weight (really more than feather weight) to assist with this. I'm not super familiar with the OSGi landscape but a lot of what I'm seeing seems to be big "component framework servers" that load and run everything, so I'm not optimistic we'll find an easy lib that fits our bill. But I'm open to any suggestions from others more knowledgeable.
The DI thought is interesting, but I'm not sure coprocessors would be a great fit for those tools. We currently provide access to needed services through the CoprocessorEnvironment interface, which DI could certainly replace. But each coprocessor is currently a separate island the relationship between them seems more akin to chained servlet filters on a request than independent components. For the moment we're trying to avoid direct class references between coprocessors so that we can better isolate the class loading, allowing us to do code weaving on class loading to enforce resource utilization throttling.
Are there other aspects you can see DI providing? Maybe I'm seeing things the wrong way? And any suggestions on very light weight libs for assisting?
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Gary Helmling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972639#action_12972639 ]
Gary Helmling commented on HBASE-3260:
--------------------------------------
As part of this change, I'm also converting the CoprocessorHost {{bypass}} and {{complete}} variables that were added in HBASE-3348 from {{AtomicBoolean}} to {{ThreadLocal<Boolean>}}. Without being thread local, it's up to the coprocessor implementor to setting of these values between multiple invoking threads, which is non-obvious. With this change the framework handles synchronization, which seems more as expected.
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12966007#action_12966007 ]
Andrew Purtell commented on HBASE-3260:
---------------------------------------
I would like to take a step back and consider a key motivation of the coprocessor design, or, at least present here an analogy: We have treated HBase like an operating system kernel (a highly concurrent performance critical thing) and coprocessors as the addition of an extension surface in the manner usually employed with respect to OS kernels -- callouts -- the lightest weight possible in terms of performance impact. As stated earlier on this issue, chained servlet filters is another good analogy.
We are not considering something like generic DI, nor perhaps OSGi, though honestly I'm not convinced either way on the latter point if it is useful for coprocessor lifecycle management or not. (I thought... why not just a couple of additional upcalls? That would be enough for my needs.) We want to provide a flexible extension surface with well defined semantics. We do not want to force refactorings for the sake of producing the necessary layering/abstraction for a DI point, or to separate some function into the OSGi notion of service. So then DI and perhaps most of what benefit OSGi might bring to an application engineering context is not applicable here.
This approach can of course change if refactoring of HBase code to be more amenable to generic DI and component model is the consensus way forward. Or if the core devs feel amenable to satisfying requests for code refactoring when someone might want to extend something via the DI approach.
Personally I think the coprocessors approach as is has compelling benefits:
- Simplicity
- Well defined interface and semantics -- a fixed framework of extension points
- No need to involve core devs or trigger code refactoring
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935183#action_12935183 ]
stack commented on HBASE-3260:
------------------------------
Thanks for entertaining my random ramble Gary and Andrew. Your reasoning seems good to me.
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-3260:
----------------------------------
Description:
Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
A coprocessor transitions between the below states over its lifetime:
||State||Description||
|UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
|INSTALLED|The coprocessor implementation has been successfully installed|
|STARTING|A coprocessor instance is being started.|
|ACTIVE|The coprocessor instance has been successfully activated and is running.|
|STOPPING|A coprocessor instance is being stopped.|
See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
{code:java}
public interface CoprocessorLifecycle {
void start(CoprocessorEnvironment env) throws IOException;
void stop(CoprocessorEnvironment env) throws IOException;
}
{code}
Fix Version/s: 0.92.0
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965919#action_12965919 ]
ryan rawson commented on HBASE-3260:
------------------------------------
We are fairly thrifty on classes, for example the HBase read path touches 10-20 classes or so.
Reading the OSGi wikipedia page wasn't very encouraging, "industry standard" generally implies "sucky". You should come to the coprocessor hackathon in a week or so in palo alto and advocate for OSGi.
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Leen Toelen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969744#action_12969744 ]
Leen Toelen commented on HBASE-3260:
------------------------------------
Nuxeo uses a custom version of an OSGi container. This might be interesting as well. They are also adding OSGi metadata on their bundles, which makes sense even if not using OSGi at runtime.
(Slide 41)
http://blogs.nuxeo.com/fermigier/2010/12/lessons-learned-building-nuxeo-ep-the-slides-are-now-available.html
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972312#action_12972312 ]
HBase Review Board commented on HBASE-3260:
-------------------------------------------
Message from: "Gary Helmling" <gh...@gmail.com>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1306/
-----------------------------------------------------------
Review request for hbase, stack and Andrew Purtell.
Summary
-------
This patch adds explicit start() and stop() methods for lifecycle management to the Coprocessor interface and refactors some of the Coprocessor/RegionObserver distinction, moving the region-related pre/post hooks that were previously in Coprocessor to RegionObserver.
Coprocessor is now the base interface, containing only:
- start()
- stop()
- Priority enum
- State enum
RegionObserver extends Coprocessor, and now contains the additional pre/post hooks, moved from Coprocessor:
- pre/postOpen
- pre/postClose
- pre/postFlush
- pre/postCompact
- pre/postSplit
This will allow cleaner extension in the future, to allow addition of a MasterObserver interface, for example.
As shown above, I've also added a new Coprocessor.State enum consisting of the states:
UNINSTALLED -> INSTALLED -> STARTING -> ACTIVE -> STOPPING -> STOPPED
However, the UNINSTALLED/INSTALLED distinction is not particularly useful at the moment. I'd appreciate other feedback on what's necessary here. The current handling could make do with:
UNINSTALLED -> STARTING -> ACTIVE -> STOPPING -> UNINSTALLED (4 total states)
However, the UNINSTALLED/INSTALLED distinction may be useful if we want to add class level initialization in the future...
This addresses bug HBASE-3260.
http://issues.apache.org/jira/browse/HBASE-3260
Diffs
-----
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java b81a465
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseRegionObserverCoprocessor.java f022598
src/main/java/org/apache/hadoop/hbase/coprocessor/Coprocessor.java 7ea1c5e
src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java 1792290
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java f028525
src/test/java/org/apache/hadoop/hbase/coprocessor/SimpleRegionObserver.java 3db4c36
src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java 81cb75d
Diff: http://review.cloudera.org/r/1306/diff
Testing
-------
Added tests for start() and stop() method invocation in org.apache.hadoop.hbase.coprocessor.TestCoprocessorInterface
The existing TestCoprocessorEndpoint, TestCoprocessorInterface, TestRegionObserverInterface, TestRegionObserverStacking tests continue to work. I'm not seeing any new failures in the rest of the tests, but TestReplication is timing out for me, preventing all tests from executing.
Thanks,
Gary
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Gary Helmling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Helmling reassigned HBASE-3260:
------------------------------------
Assignee: Gary Helmling
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Gary Helmling
> Fix For: 0.92.0
>
> Attachments: HBASE-3260_final.patch, statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935111#action_12935111 ]
Andrew Purtell commented on HBASE-3260:
---------------------------------------
We could use something like Guice as a lightweight DI framework within HBase code in general, but I think this is orthogonal to what coprocessors tries to achieve.
bq. But each coprocessor is currently a separate island the relationship between them seems more akin to chained servlet filters on a request than independent components.
Chained servlet filters is a good analogy. Need to add client-transparent compression support to your webapp? Register a compression filter on the chain. Need to add client-transparent value compression to your table? Register a value compression coprocessor on the region.
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965848#action_12965848 ]
ryan rawson commented on HBASE-3260:
------------------------------------
Perhaps you could explain the advantages more, the 15 interfaces, 16 class argument sounds kind of opposite from "lightweight".
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Neil Bartlett (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965924#action_12965924 ]
Neil Bartlett commented on HBASE-3260:
--------------------------------------
If industry standard implies sucky then I guess Java REALLY sucks ;-)
Don't get me wrong, I understand where you're coming from. But not all standards suck merely because they are standard.
Thanks for the invitation to the Hackathon, unfortunately I'm not in the area (I'm in London, UK). I will ping some folks who may be able to attend.
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Neil Bartlett (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965914#action_12965914 ]
Neil Bartlett commented on HBASE-3260:
--------------------------------------
Really? Wow. I suppose I should clarify that only about 3 of those classes are actual surface area API that you might interact with, and most of the interfaces are listeners with one or two methods that you might want to implement if you want to hook into specific parts of the lifecycle.
To go any further with this heavyweight vs lightweight argument though, I'd need to understand what this community defines as heavyweight. Note though that OSGi's roots are in embedded software. It can run on very small devices indeed.
The advantages as I see them would be:
1) Not reinventing the wheel, but instead inheriting a very robust and production-tested solution that appears to be close to your requirements already.
2) Ability to use existing tools for OSGi bundle creation: e.g. Bnd, Maven's bundle plugin, extensive tooling in both Eclipse and IntelliJ.
3) Familiarity for a large number of developers, particularly within the Apache community since many other Apache projects are using OSGi
4) Modularity. Users can structure their extensions as fine-grained modules and even allow their extensions to be further extended by others.
5) Dynamic deployment and "hotswappability"
6) It's cool :-)
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12974488#action_12974488 ]
Hudson commented on HBASE-3260:
-------------------------------
Integrated in HBase-TRUNK #1697 (See [https://hudson.apache.org/hudson/job/HBase-TRUNK/1697/])
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Gary Helmling
> Fix For: 0.92.0
>
> Attachments: HBASE-3260_final.patch, statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Purtell updated HBASE-3260:
----------------------------------
Attachment: statechart.png
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Gary Helmling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Helmling updated HBASE-3260:
---------------------------------
Attachment: HBASE-3260_final.patch
Final patch that was committed.
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Gary Helmling
> Fix For: 0.92.0
>
> Attachments: HBASE-3260_final.patch, statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Neil Bartlett (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965845#action_12965845 ]
Neil Bartlett commented on HBASE-3260:
--------------------------------------
I was directed to this JIRA by a Twitter conversation between Gary and @asynchronaut (Holger Hofstätte)
If you are looking for a lightweight library that can help to provide OSGi-like lifecycle management, there is an obvious answer: OSGi itself. Apache Felix is a fully certified OSGi implementation and it is just a 300k JAR file. It can be embedded in a Java application with literally five lines of code.
You seem to be concerned that OSGi is heavyweight. While it's true that it is used in many heavyweight servers including e.g. IBM WebSphere, the core of OSGi is very tight indeed, consisting of only 15 interfaces and 16 classes (of which 9 are "Permission" classes for Java 2 security).
By embedding OSGi you will benefit from existing tools and skills, in addition to opening up the possibility for users to use many higher level frameworks that build on top of OSGi. Also you will be using lifecycle management code that is very well tested and proven.
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972336#action_12972336 ]
HBase Review Board commented on HBASE-3260:
-------------------------------------------
Message from: "Gary Helmling" <gh...@gmail.com>
bq. On 2010-12-16 17:09:44, Andrew Purtell wrote:
bq. > src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java, line 66
bq. > <http://review.cloudera.org/r/1306/diff/1/?file=18379#file18379line66>
bq. >
bq. > What are these arguments about?
Those are:
- String protocol
- long clientVersion
from org.apache.hadoop.ipc.VersionedProtocol.
Will fix these up.
bq. On 2010-12-16 17:09:44, Andrew Purtell wrote:
bq. > src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java, line 289
bq. > <http://review.cloudera.org/r/1306/diff/1/?file=18383#file18383line289>
bq. >
bq. > Should be a WARN?
Yeah, agree. Will fix.
bq. On 2010-12-16 17:09:44, Andrew Purtell wrote:
bq. > src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java, line 305
bq. > <http://review.cloudera.org/r/1306/diff/1/?file=18383#file18383line305>
bq. >
bq. > Should be a WARN?
Yeah, will fix.
bq. On 2010-12-16 17:09:44, Andrew Purtell wrote:
bq. > src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java, line 385
bq. > <http://review.cloudera.org/r/1306/diff/1/?file=18383#file18383line385>
bq. >
bq. > Since you are committing a change set in this area, Ryan suggested no need for AtomicBoolean here, could just be plain volatile boolean. I think that's right.
Ok will change this to a volatile boolean and repost.
- Gary
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1306/#review2102
-----------------------------------------------------------
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Gary Helmling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965921#action_12965921 ]
Gary Helmling commented on HBASE-3260:
--------------------------------------
@Neil
The coprocessor framework code here is involved in the critical read and write paths for HBase and also in a sense becomes part of the user facing client API. Any overhead in either direction is a big concern.
If you're in the area and free, would be great if you can come to the hackathon:
http://www.meetup.com/hackathon/calendar/15597555/
and explain other benefits in more detail. Particularly what OSGi might bring in classloading advantages, security/sandboxing and resource throttling of user code.
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "Gary Helmling (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Helmling resolved HBASE-3260.
----------------------------------
Resolution: Fixed
Committed to trunk after review by Andy.
As noted in this issue, I also tweaked the bypass and complete vars from HBASE-3348 to be thread local.
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Assignee: Gary Helmling
> Fix For: 0.92.0
>
> Attachments: HBASE-3260_final.patch, statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935031#action_12935031 ]
stack commented on HBASE-3260:
------------------------------
I'm good with borrowing the nomenclature. Can we borrow libraries that will manage the lifecycle for us? Would it make sense implementing CPs atop some lifecycle supporting framework? Would it make sense using, say, any of the DI containers wiring up CPs?
The regionserver and master have similar need of a lifecycle as has been discussed elsewhere. Would be grand if same lifecycle nomenclature was used throughout -- for hbase daemons and CP.
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-3260) Coprocessors: Lifecycle management
Posted by "HBase Review Board (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12972328#action_12972328 ]
HBase Review Board commented on HBASE-3260:
-------------------------------------------
Message from: "Andrew Purtell" <ap...@apache.org>
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
http://review.cloudera.org/r/1306/#review2102
-----------------------------------------------------------
Ship it!
src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEndpointCoprocessor.java
<http://review.cloudera.org/r/1306/#comment6557>
What are these arguments about?
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/1306/#comment6558>
Should be a WARN?
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/1306/#comment6559>
Should be a WARN?
src/main/java/org/apache/hadoop/hbase/regionserver/CoprocessorHost.java
<http://review.cloudera.org/r/1306/#comment6560>
Since you are committing a change set in this area, Ryan suggested no need for AtomicBoolean here, could just be plain volatile boolean. I think that's right.
- Andrew
> Coprocessors: Lifecycle management
> ----------------------------------
>
> Key: HBASE-3260
> URL: https://issues.apache.org/jira/browse/HBASE-3260
> Project: HBase
> Issue Type: Sub-task
> Reporter: Andrew Purtell
> Fix For: 0.92.0
>
> Attachments: statechart.png
>
>
> Considering extending CPs to the master, we have no equivalent to pre/postOpen and pre/postClose as on the regionserver. We also should consider how to resolve dependencies and initialization ordering if loading coprocessors that depend on others.
> OSGi (http://en.wikipedia.org/wiki/OSGi) has a lifecycle API and is familiar to many Java programmers, so we propose to borrow its terminology and state machine.
> A lifecycle layer manages coprocessors as they are dynamically installed, started, stopped, updated and uninstalled. Coprocessors rely on the framework for dependency resolution and class loading. In turn, the framework calls up to lifecycle management methods in the coprocessor as needed.
> A coprocessor transitions between the below states over its lifetime:
> ||State||Description||
> |UNINSTALLED|The coprocessor implementation is not installed. This is the default implicit state.|
> |INSTALLED|The coprocessor implementation has been successfully installed|
> |STARTING|A coprocessor instance is being started.|
> |ACTIVE|The coprocessor instance has been successfully activated and is running.|
> |STOPPING|A coprocessor instance is being stopped.|
> See attached state diagram. Transitions to STOPPING will only happen as the region is being closed. If a coprocessor throws an unhandled exception, this will cause the RegionServer to close the region, stopping all coprocessor instances on it.
> Transitions from INSTALLED->STARTING and ACTIVE->STOPPING would go through upcall methods into the coprocessor via the CoprocessorLifecycle interface:
> {code:java}
> public interface CoprocessorLifecycle {
> void start(CoprocessorEnvironment env) throws IOException;
> void stop(CoprocessorEnvironment env) throws IOException;
> }
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.