You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jonathan Hsieh (JIRA)" <ji...@apache.org> on 2012/11/22 20:08:57 UTC
[jira] [Created] (HBASE-7212) Globally Barriered Procedure
mechanism
Jonathan Hsieh created HBASE-7212:
-------------------------------------
Summary: Globally Barriered Procedure mechanism
Key: HBASE-7212
URL: https://issues.apache.org/jira/browse/HBASE-7212
Project: HBase
Issue Type: Sub-task
Components: snapshots
Affects Versions: hbase-6055
Reporter: Jonathan Hsieh
Fix For: hbase-6055
This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Enis Soztutar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509425#comment-13509425 ]
Enis Soztutar commented on HBASE-7212:
--------------------------------------
What happens when the coordinator dies (in this case hmaster). Does the new HMaster discover the prev procedure and abort?
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508096#comment-13508096 ]
stack commented on HBASE-7212:
------------------------------
Doc looks great.
Do you have to write it yourself? Anything already available that you could use? Say we used the zk curator client, it has a few barriers implemented already: https://github.com/Netflix/curator/wiki/Recipes If we were using curator say, could you use these receipes as building blocks so you didn't have to write this yourself? (This feature has to be backportable to 0.94?)
Reading the diagram, I"m not sure what receivedreached is. Or sendReached. sendReached is the coordinator saying all participants responded/are participating?
On your barrier you say "...but does not support ACID semantics" and thats ok because the 'transactions' we'll be running over this mechanism do not require it? Because they can proceed and complete in any order and result will come off the same?
You say "....Does not recover on failures" ... because the operation just FAILs. Right?
Only one of these precedures can be ongoingn at any one time? Is that right?
How do I read these set of slides? There is a 'Barrier Procedure Coordination' and then there is 'Procedure Coordination'? So, the PC makes use of a BPC? BPC is the skeleton you hang PC on?
Why you say this 'If we aren’t doing proper 2PC do we need all this infrastructure?'? Are you making a case for our not needing 2PC given what is being implemented?
Coordinator can be any client? Does not have to be master?
What is ProcedureCoordinateComms?
Does this barrier acquistion have any relation to zk barrier receipe? http://zookeeper.apache.org/doc/trunk/recipes.html#sc_recipes_eventHandles
What is 'class' in the zk node hierarchy? Class of procedure?
Procedure looks good to me.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh reassigned HBASE-7212:
-------------------------------------
Assignee: Jonathan Hsieh
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509465#comment-13509465 ]
Jonathan Hsieh commented on HBASE-7212:
---------------------------------------
bq. What happens when the coordinator dies (in this case hmaster). Does the new HMaster discover the prev procedure and abort?
The new HMaster will delete all znodes associated with the procedure class (all znodes associated with snapshotting procedures), all members still using them should timeout and fail, and new operations need to be issued. For snapshots in particular, there isn't really a chance for a partial snapshot being present when taking one because all the snapshot work is done in a temp dir and atomically put into place with a dir rename op after the coordinator realizes all the members have released/leave'd successfully. There will be junk in these tmp dirs left over but they get cleaned up on the next take snapshot attempt, or when the new master starts.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508125#comment-13508125 ]
Jonathan Hsieh commented on HBASE-7212:
---------------------------------------
Thanks for taking a look. I'll get another rev out with cleaned up documentation in a day or two. Answers below.
bq. Do you have to write it yourself? Anything already available that you could use? Say we used the zk curator client, it has a few barriers implemented already: https://github.com/Netflix/curator/wiki/Recipes If we were using curator say, could you use these receipes as building blocks so you didn't have to write this yourself? (This feature has to be backportable to 0.94?)
This is a simplified version of Jesse's patch. I just gave curator a quick it is similar to the double barrier (https://github.com/Netflix/curator/wiki/Double-barrier). If it is implemented as the recipe you pointed out, I think we'd still need to add in the ability for cancellation/abort to come from any of the members.
bq. Reading the diagram, I"m not sure what receivedreached is. Or sendReached. sendReached is the coordinator saying all participants responded/are participating?
Yes -- reached is sent when the coordinator figures out that it has "reached" the global barrier point because all members have taken their part of the global barrier.
Basically, zk is being used for its async notifications and as the RPC mechanism. Arrows into the ZK column are calls writing to ZK, arrows out of ZK are callbacks being called at the target. So the red coordinator writes to zk via sendStart, zk node creation triggers a startNewOpearion callback on the the blue member1, and similarly on the the green member2. These names are short hand for the names in the review was posted -- now sendStart -> sendBarrierStart, sendReached -> sendBarrierReached, startNewOperation -> Subprocedure's consturctor + acquireBarrier, receiveReached -> receiveReachedGlobalBarrier
bq. On your barrier you say "...but does not support ACID semantics" and thats ok because the 'transactions' we'll be running over this mechanism do not require it? Because they can proceed and complete in any order and result will come off the same?
Previously, this code was called TwoPhaseCommit (2pc). While it had two phases, the code did not implement true two phase commit. The purpose of this explicit comparison is to make clear 2pc's purpose (distributed ACID guarantees), to point out that we don't have 2pc here, to point out that we don't need 2pc here, and to point out that we just need a global barrier.
The online snapshot coordination does not need all of what 2pc provides. The first cut will have "only on a sunny day" semantics -- e.g. it will only succeed if everything succeeds and if anything fails along the way whole attempt will be aborted. This is ok because the durable work that snapshots does goes into tmp dir (/hbase/.snapshots/.tmp/xxx) that is "commited" at the end atomically via HDFS dir rename, and that durable intermediate operation (e.g. new files from forcing a hlog roll or hlog flush) don't need to be undone to remain correct.
bq. You say "....Does not recover on failures" ... because the operation just FAILs. Right?
Yup.
bq. Only one of these precedures can be ongoingn at any one time? Is that right?
True for this first cut implementation, but not a fundamental limitation. This actually gets enforced at the snapshot manager level which may be visible in HBASE-7208 and definitely in HBASE-6866 when that gets posted. I believe as implemented if we picked a different class we could have multiple different kinds of procedure concurrently running on a different znode dir hierarchy.
bq. How do I read these set of slides? There is a 'Barrier Procedure Coordination' and then there is 'Procedure Coordination'? So, the PC makes use of a BPC? BPC is the skeleton you hang PC on?
All those are synonymous -- I've bee using procedure as a shorthand. The code implements one framework for a globally barriered procedure, and I've just tried to call it 'procedure' and 'subprocedure' everywhere (though from review I missed spots where it was called task, operation, or commit). This 'procedure' takes care of the global barrier coordination and cross process error propagation.
bq. Why you say this 'If we aren’t doing proper 2PC do we need all this infrastructure?'? Are you making a case for our not needing 2PC given what is being implemented?
I could probably remove that line -- I'm now convinced why we need what this code does.
The main questions I had when I was initially understanding the previous implementation was "Is this 2pc?" and "Do we need 2pc?". The answers are: what we have implemented here has two phases but is *not* true two-phase commit. 2pc, as defined in the literature (http://www.cs.berkeley.edu/~brewer/cs262/Aries.pdf), requires that once the coordinator says something is committed, any failures at a member or coordinator must be recover by failing forward and completing it. The key point here is that while we will need a global barrier for one of the snapshot flavors (global), it don't need full 2PC because 1) the we don't need to undo work (like a log roll or flush) if some sub part of the first phase (our acquire/2pc's prepare) fails, and because 2) we don't need to recover failing forward if anything fails in the second phase (our release/2pc's commit). In the latter case we just fail and delete .snapshot/.tmp reminants in the fs, and carry on with extra flushed/rolled hlogs.
bq. Coordinator can be any client? Does not have to be master?
It could be anywhere, but currently for snapshots the coordinator lives on the master.
bq. What is ProcedureCoordinateComms?
This is actually a layer that separate the zk code (the rpc communications or comms code) from specific execution (snapshotting specific code). I could probably remove it, but the abstraction allows for testing the core pieces without zk.
bq. Does this barrier acquistion have any relation to zk barrier receipe? http://zookeeper.apache.org/doc/trunk/recipes.html#sc_recipes_eventHandles
Yes. It is very similar to the double barrier. The main thing different here is this code allows for any member or coordinator to abort/cancel the whole shebang while the recipe doesn't seem to. From the recipe it seems that we could be a little bit more clever about how we use our znodes. (we might have one extra set).
bq. What is 'class' in the zk node hierarchy? Class of procedure?
The online-snapshots is a 'class' (e.g. all online snapshots) while a procedure name is an actual name for a particular snapshotting request (snapshot121201, snapshot121202 etc). Off the top of my head I can't think of any other HBase processes that are ok with the procedure mechanism's semantics (other operations like enabling, disabling, schema change, splitting, merging probably want 2pc and its recovery requirements). I think this extra znode dir could probably get removed.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509344#comment-13509344 ]
stack commented on HBASE-7212:
------------------------------
On curator double-barrier, it would seem there is no 'abort' as you say. They do have timeouts on barrier enter and leave. Would that be enough See http://www.jarvana.com/jarvana/view/com/netflix/curator/curator-recipes/0.6.4/curator-recipes-0.6.4-javadoc.jar!/com/netflix/curator/framework/recipes/barriers/DistributedDoubleBarrier.html#leave()
Rather than 'abort', you could just timeout? That might be simpler still? i.e. your "Need to be able to force failure after a specified
timeout elapses"
double-barrier does not seem to be enough though. There needs to be a means of telling cluster members to go for a particular snapshot barrier. To this end, I suppose all members need to be watching a snapshot dir and when a new snapshot appears, all try to 'enter' its barrier?
bq. Yes -- reached is sent when the coordinator figures out that it has "reached" the global barrier point because all members have taken their part of the global barrier.
Is it true that you do not want members to start 'snapshotting' until ALL participants have 'entered' the barrier? Does it matter if they start doing their work soon as they 'enter' the barrier (using curator/zk receipe terms). Reading on, it seems like its fine if members just go about their merry way....working on their part of snapshot. If not all members complete, the coordinator will clean up the incomplete.
What do you think of the terms in the zk receipe: i.e. rather than 'reach' a barrier, 'enter' it?
Some of the answers you give above should go into doc of this feature. They are quality.
I buy your argument for going w/ the more basic barrier rather than 2pc function for snapshots (Yeah, 2pc would be useful for other distributed ops like table enable/disable w/ us 'failing forward' an interrupted table enable or disable)
On 'Comms', it was just unclear to me what it was. Makes sense now.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509463#comment-13509463 ]
Jonathan Hsieh commented on HBASE-7212:
---------------------------------------
I need to take a look at the source implementation of the curator double barrier and examples of its use to do a better job of comparing. Based on the api and the zk recipes, I'm going to make some assumptions here.
As another analogy, it seems that our procedure mechanism is similar to a monitor (synchronized in java) that guarantees enter/acquire and leave/release of the barrier parts, while the curator one is lower level and leaves it to the implementer to enforce that invariant.
bq. Rather than 'abort', you could just timeout? That might be simpler still? i.e. your "Need to be able to force failure after a specified timeout elapses"
So in this patch, the time-based abort trigger and a potential user-induced cancellation uses the same mechanism to notify all members (and the coordinator) that the procedure has aborted.
I'm speculating but with think one assumption with this mechanism has vs the double barrier's is that we assume that the actions on the members may be slow (one implementation waits for a memstore flush per region) and may need to be interrupted before completion. The curator double barrier api doesn't have such a mechanism and we may have to wait for all operations to complete before we can abort them.
bq. double-barrier does not seem to be enough though. There needs to be a means of telling cluster members to go for a particular snapshot barrier. To this end, I suppose all members need to be watching a snapshot dir and when a new snapshot appears, all try to 'enter' its barrier?
I believe that would be the case if we used curator. I don't think we can't use it -- and the factoring out of the *Comms/*Rpcs would potentially allow us to move that in a future rev.
bq. s it true that you do not want members to start 'snapshotting' until ALL participants have 'entered' the barrier? Does it matter if they start doing their work soon as they 'enter' the barrier (using curator/zk receipe terms). Reading on, it seems like its fine if members just go about their merry way....working on their part of snapshot. If not all members complete, the coordinator will clean up the incomplete.
At the end of the day, the full barrier is only required for the snapshot that completely blocks all writes to get a truly consistent snapshot. The weaker snapshots (either the timestamp based or log roll based) won't give those guarantees and doesn't actually need the full barrier. For the first cut however, I'm probably going to use it since it handles the error propagation and cross process cancellation.
bq. What do you think of the terms in the zk receipe: i.e. rather than 'reach' a barrier, 'enter' it?
I'm fine with it -- I'll change the terms acquire -> enter, reached -> leave in the next rev I post. (in the v3 version I still need to clean up the nomenclature in the tests).
I'll do another rev of the docs to make it consistent with the changes being made.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508128#comment-13508128 ]
Andrew Purtell edited comment on HBASE-7212 at 12/2/12 2:00 AM:
----------------------------------------------------------------
bq. The main questions I had when I was initially understanding the previous implementation was "Is this 2pc?" and "Do we need 2pc?". The answers are: what we have implemented here has two phases but is not true two-phase commit. 2pc, as defined in the literature (http://www.cs.berkeley.edu/~brewer/cs262/Aries.pdf), requires that once the coordinator says something is committed, any failures at a member or coordinator must be recover by failing forward and completing it. The key point here is that while we will need a global barrier for one of the snapshot flavors (global), it don't need full 2PC because 1) the we don't need to undo work (like a log roll or flush) if some sub part of the first phase (our acquire/2pc's prepare) fails, and because 2) we don't need to recover failing forward if anything fails in the second phase (our release/2pc's commit). In the latter case we just fail and delete .snapshot/.tmp reminants in the fs, and carry on with extra flushed/rolled hlogs.
+1
This makes a good case. I like the "keep it as simple as possible and only do as much as we actually need to" approach.
Edit: Moved unrelated comment to HBASE-7254
was (Author: apurtell):
bq. The main questions I had when I was initially understanding the previous implementation was "Is this 2pc?" and "Do we need 2pc?". The answers are: what we have implemented here has two phases but is not true two-phase commit. 2pc, as defined in the literature (http://www.cs.berkeley.edu/~brewer/cs262/Aries.pdf), requires that once the coordinator says something is committed, any failures at a member or coordinator must be recover by failing forward and completing it. The key point here is that while we will need a global barrier for one of the snapshot flavors (global), it don't need full 2PC because 1) the we don't need to undo work (like a log roll or flush) if some sub part of the first phase (our acquire/2pc's prepare) fails, and because 2) we don't need to recover failing forward if anything fails in the second phase (our release/2pc's commit). In the latter case we just fail and delete .snapshot/.tmp reminants in the fs, and carry on with extra flushed/rolled hlogs.
+1
This makes a good case. I like the "keep it as simple as possible and only do as much as we actually need to" approach.
I can see a use for this in security too. We could tighten up the permissions cache using a barrier for grant and revoke ops. In other words, replace the current ZK watcher based permissions cache "RPC via ZK" with the Procedure mechanism that provides much the same, but with the added benefit that we can fail the grant or revoke op if one or more RSes fail to ack the update.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505187#comment-13505187 ]
Jonathan Hsieh commented on HBASE-7212:
---------------------------------------
https://reviews.apache.org/r/8240/
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-7212:
----------------------------------
Attachment: 121127-global-barrier-proc.pdf
Attached a quick deck with a summary of the design and semantics of this mechanism.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-7212:
----------------------------------
Parent Issue: HBASE-7290 (was: HBASE-6055)
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-7212:
----------------------------------
Status: Patch Available (was: In Progress)
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505194#comment-13505194 ]
Hadoop QA commented on HBASE-7212:
----------------------------------
{color:red}-1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12555109/hbase-7212.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 13 new or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/3414//console
This message is automatically generated.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Hsieh updated HBASE-7212:
----------------------------------
Attachment: hbase-7212.patch
pre-hbase-7212.patch
Include a patch that includes all work on top of the jesse's snapshots branch, and then the actual patch under review.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Jonathan Hsieh (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Work on HBASE-7212 started by Jonathan Hsieh.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7212) Globally Barriered Procedure
mechanism
Posted by "Andrew Purtell (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-7212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13508128#comment-13508128 ]
Andrew Purtell commented on HBASE-7212:
---------------------------------------
bq. The main questions I had when I was initially understanding the previous implementation was "Is this 2pc?" and "Do we need 2pc?". The answers are: what we have implemented here has two phases but is not true two-phase commit. 2pc, as defined in the literature (http://www.cs.berkeley.edu/~brewer/cs262/Aries.pdf), requires that once the coordinator says something is committed, any failures at a member or coordinator must be recover by failing forward and completing it. The key point here is that while we will need a global barrier for one of the snapshot flavors (global), it don't need full 2PC because 1) the we don't need to undo work (like a log roll or flush) if some sub part of the first phase (our acquire/2pc's prepare) fails, and because 2) we don't need to recover failing forward if anything fails in the second phase (our release/2pc's commit). In the latter case we just fail and delete .snapshot/.tmp reminants in the fs, and carry on with extra flushed/rolled hlogs.
+1
This makes a good case. I like the "keep it as simple as possible and only do as much as we actually need to" approach.
I can see a use for this in security too. We could tighten up the permissions cache using a barrier for grant and revoke ops. In other words, replace the current ZK watcher based permissions cache "RPC via ZK" with the Procedure mechanism that provides much the same, but with the added benefit that we can fail the grant or revoke op if one or more RSes fail to ack the update.
> Globally Barriered Procedure mechanism
> --------------------------------------
>
> Key: HBASE-7212
> URL: https://issues.apache.org/jira/browse/HBASE-7212
> Project: HBase
> Issue Type: Sub-task
> Components: snapshots
> Affects Versions: hbase-6055
> Reporter: Jonathan Hsieh
> Assignee: Jonathan Hsieh
> Fix For: hbase-6055
>
> Attachments: 121127-global-barrier-proc.pdf, hbase-7212.patch, pre-hbase-7212.patch
>
>
> This is a simplified version of what was proposed in HBASE-6573. Instead of claiming to be a 2pc or 3pc implementation (which implies logging at each actor, and recovery operations) this is just provides a best effort global barrier mechanism called a Procedure.
> Users need only to implement a methods to acquireBarrier, to act when insideBarrier, and to releaseBarrier that use the ExternalException cooperative error checking mechanism.
> Globally consistent snapshots require the ability to quiesce writes to a set of region servers before a the snapshot operation is executed. Also if any node fails, it needs to be able to notify them so that they abort.
> The first cut of other online snapshots don't need the fully barrier but may still use this for its error propagation mechanisms.
> This version removes the extra layer incurred in the previous implementation due to the use of generics, separates the coordinator and members, and reduces the amount of inheritance used in favor of composition.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira