You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hbase.apache.org by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org> on 2010/06/29 19:40:51 UTC
[jira] Created: (HBASE-2804) [replication] Support ICVs in a
master-master setup
[replication] Support ICVs in a master-master setup
---------------------------------------------------
Key: HBASE-2804
URL: https://issues.apache.org/jira/browse/HBASE-2804
Project: HBase
Issue Type: New Feature
Reporter: Jean-Daniel Cryans
Fix For: 0.21.0
Currently an ICV ends up as a Put in the HLogs, which ReplicationSource ships to ReplicationSink that in turn only recreates the Put and not the ICV itself. This means that in a master-master replication setup where the same counters are implemented on both side, the Puts will actually overwrite each other.
We need to find a way to support this use case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2804) [replication] Support ICVs in a
master-master setup
Posted by "ryan rawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12883712#action_12883712 ]
ryan rawson commented on HBASE-2804:
------------------------------------
it would make sense to 'shard' ICV by datacenters, where each datacenter gets it's own ICV column then anyone wishing to know the total would just get all columns and sum. Different datacenters would not overwrite each other. The only problem is this is more of an application level thing, and isnt baked into the API anywhere setting people up for failure down the road.
The problem with doing something like shipping deltas is that it becomes difficult to bring up a new cluster, since the cluster will need a 'starting point' combined with a sequence of deltas that must mesh perfectly or else the replica cluster will be out of sync.
> [replication] Support ICVs in a master-master setup
> ---------------------------------------------------
>
> Key: HBASE-2804
> URL: https://issues.apache.org/jira/browse/HBASE-2804
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Currently an ICV ends up as a Put in the HLogs, which ReplicationSource ships to ReplicationSink that in turn only recreates the Put and not the ICV itself. This means that in a master-master replication setup where the same counters are implemented on both side, the Puts will actually overwrite each other.
> We need to find a way to support this use case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2804) [replication] Support ICVs in a
master-master setup
Posted by "stack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12889418#action_12889418 ]
stack commented on HBASE-2804:
------------------------------
This is going to be done in the 0.90.0 timeframe? Otherwise, lets move it out?
> [replication] Support ICVs in a master-master setup
> ---------------------------------------------------
>
> Key: HBASE-2804
> URL: https://issues.apache.org/jira/browse/HBASE-2804
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> Currently an ICV ends up as a Put in the HLogs, which ReplicationSource ships to ReplicationSink that in turn only recreates the Put and not the ICV itself. This means that in a master-master replication setup where the same counters are implemented on both side, the Puts will actually overwrite each other.
> We need to find a way to support this use case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2804) [replication] Support ICVs in a
master-master setup
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12918198#action_12918198 ]
Jonathan Gray commented on HBASE-2804:
--------------------------------------
+1 on punting to 0.92
> [replication] Support ICVs in a master-master setup
> ---------------------------------------------------
>
> Key: HBASE-2804
> URL: https://issues.apache.org/jira/browse/HBASE-2804
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Fix For: 0.90.0
>
>
> Currently an ICV ends up as a Put in the HLogs, which ReplicationSource ships to ReplicationSink that in turn only recreates the Put and not the ICV itself. This means that in a master-master replication setup where the same counters are implemented on both side, the Puts will actually overwrite each other.
> We need to find a way to support this use case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HBASE-2804) [replication] Support ICVs in a
master-master setup
Posted by "Jean-Daniel Cryans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jean-Daniel Cryans updated HBASE-2804:
--------------------------------------
Fix Version/s: (was: 0.90.0)
0.92.0
Punting to 0.92.0
> [replication] Support ICVs in a master-master setup
> ---------------------------------------------------
>
> Key: HBASE-2804
> URL: https://issues.apache.org/jira/browse/HBASE-2804
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Fix For: 0.92.0
>
>
> Currently an ICV ends up as a Put in the HLogs, which ReplicationSource ships to ReplicationSink that in turn only recreates the Put and not the ICV itself. This means that in a master-master replication setup where the same counters are implemented on both side, the Puts will actually overwrite each other.
> We need to find a way to support this use case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HBASE-2804) [replication] Support ICVs in a
master-master setup
Posted by "Jonathan Gray (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HBASE-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12884842#action_12884842 ]
Jonathan Gray commented on HBASE-2804:
--------------------------------------
you can pay the cost at write-time by executing double writes to each DC, or you can pay the cost at read-time by executing double reads from each DC.
i think it's worth at least trying to get some "eventually consistent" master-master counters via async replication. is the 'starting point' any more complex than normal replication? i guess if you are considering duplicate writes as idempotent (they kind of are but not completely) it's different but if not it doesn't seem any more difficult, the ordering of operations does not matter in this case only that the operations are strictly not idempotent.
> [replication] Support ICVs in a master-master setup
> ---------------------------------------------------
>
> Key: HBASE-2804
> URL: https://issues.apache.org/jira/browse/HBASE-2804
> Project: HBase
> Issue Type: New Feature
> Reporter: Jean-Daniel Cryans
> Fix For: 0.21.0
>
>
> Currently an ICV ends up as a Put in the HLogs, which ReplicationSource ships to ReplicationSink that in turn only recreates the Put and not the ICV itself. This means that in a master-master replication setup where the same counters are implemented on both side, the Puts will actually overwrite each other.
> We need to find a way to support this use case.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.