[jira] [Comment Edited] (HBASE-21856) Consider Causal Replication Ordering

["Bin Shi (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HBASE-21856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16924560#comment-16924560 ] 

Bin Shi edited comment on HBASE-21856 at 9/6/19 7:52 PM:
---------------------------------------------------------

[~apurtell], what you said about "source side ordering" makes sense to me. I want to know more about why the source needs to ship edits of a region to the same destination in the sink, as described by "Sink side ordering". 

Currently, the source ships the batches (a batch contains edits of multiple regions) with multiple Replication RPCs in parallel, and each RPC is a synchronous call (waiting for replication response before sending another batch). On the sink side, when a region server receives a Replication RPC, it splits the batch into smalls batches per region, calls table.batch() synchronously to apply the batch to the region, then sends replication response to the source. If my understanding is correct, what we really miss here is that the source should ship batches from the same region sequentially, instead of making sure these batches landing on the same destination. With the former, since we only have single source now, we can preserve the order for a region in the presence of cross cluster replication, no matter the batches from the same region lands on the same destination or on the different destinations. Without the former, even the batches from the same region landing on the same destination, we still can't preserve the order.

Please let me know if I miss anything.


was (Author: bin shi):
[~apurtell], what you said about "source side ordering" makes sense to me. I want to know more about why the source needs to ship edits of a region to the same destination in the sink, as described by "Sink side ordering". 

Currently, the source ships the batches (a batch contains edits of multiple regions) with multiple Replication RPC in parallel, and for each RPC, it is a synchronous call (waiting for replication response before sending another batch). On the sink side, when a region server receives a Replication RPC, it splits the batch into smalls batches per region, calls table.batch() synchronously to apply the batch to the region, then sends replication response to the source. If my understanding is correct, what we really miss here is that the source should ship batches from the same region sequentially, instead of making sure these batches landing on the same destination. With the former, since we only have single source now, we can preserve the order for a region in the presence of cross cluster replication, no matter the batches from the same region lands on the same destination or on the different destinations. Without the former, even the batches from the same region landing on the same destination, we still can't preserve the order.

Please let me know if I miss something.

> Consider Causal Replication Ordering
> ------------------------------------
>
>                 Key: HBASE-21856
>                 URL: https://issues.apache.org/jira/browse/HBASE-21856
>             Project: HBase
>          Issue Type: Brainstorming
>          Components: Replication
>            Reporter: Lars Hofhansl
>            Priority: Major
>              Labels: Replication
>
> We've had various efforts to improve the ordering guarantees for HBase replication, most notably Serial Replication.
> I think in many cases guaranteeing a Total Replication Order is not required, but a simpler Causal Replication Order is sufficient.
> Specifically we would guarantee causal ordering for a single Rowkey. Any changes to a Row - Puts, Deletes, etc - would be replicated in the exact order in which they occurred in the source system.
> Unlike total ordering this can be accomplished with only local region server control.
> I don't have a full design in mind, let's discuss here. It should be sufficient to to the following:
> # RegionServers only adopt the replication queues from other RegionServers for regions they (now) own. This requires log splitting for replication.
> # RegionServers ship all edits for queues adopted from other servers before any of their "own" edits are shipped.
> It's probably a bit more involved, but should be much cheaper that the total ordering provided by serial replication.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)