You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Karthik N <ka...@gmail.com> on 2012/06/26 16:52:53 UTC
Multi datacenter, WAN hiccups and replication
My Cassandra ring spans two DCs. I use local quorum with replication
factor=3. I do a write in DC1 with local quorum. Data gets written to
multiple nodes in DC1. For the same write to propagate to DC2 only one
copy is sent from the coordinator node in DC1 to a coordinator node in
DC2 for optimizing traffic over the WAN (from what I have read in the
Cassandra documentation)
Will a Wan hiccup result in a Hinted Handoff (HH) being created in
DC1's coordinator for DC2 to be delivered when the Wan link is up
again?
Re: Multi datacenter, WAN hiccups and replication
Posted by aaron morton <aa...@thelastpickle.com>.
> Therefore I was wondering if Cassandra already intelligently optimizes
> for HH-over-WAN (since this is common) or alternately if there's a way
> to enable HH for WAN replication?
When the coordinator is preparing to process the request, done nodes in a foreign DC are treated like down nodes in a local DC. So a hint is stored for each down node. So more than 1 cross DC message is used when the hints are replayed.
When the local DC coordinator sends a forwarded message to a foreign DC coordinator it sets up an expectation to hear back from each of the replicas in the foreign DC. If any of these fail to return, because say the foreign DC coordinator went down, a hint is stored. So again, more than 1 cross DC message will be used when the hints are replayed.
Replaying hints is triggered when a node notices that another is up again. There would be no sense in forwarding the messages as they are sent directly to the recipient.
Hope that helps.
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 27/06/2012, at 5:14 AM, Karthik N wrote:
> Let me attempt to articulate my question a little better.
>
> Say I choose LOCAL_QUORUM with a Replication Factor of 3. Cassandra
> stores three copies in my local datacenter. Therefore the cost
> associated with "losing" one node is not very high locally, and I
> usually HH, and use read repair/nodetool repair instead.
>
> However over the WAN network blips are quite normal and HH really
> helps. More so because for WAN replication Cassandra sends only one
> copy to a coordinator in the remote datacenter.
>
> Therefore I was wondering if Cassandra already intelligently optimizes
> for HH-over-WAN (since this is common) or alternately if there's a way
> to enable HH for WAN replication?
>
> Thank you.
>
> On Tue, Jun 26, 2012 at 9:22 AM, Mohit Anchlia <mo...@gmail.com> wrote:
>>
>>
>> On Tue, Jun 26, 2012 at 8:16 AM, Karthik N <ka...@gmail.com> wrote:
>>>
>>> Since Cassandra optimizes and sends only one copy over the WAN, can I opt
>>> in only for HH for WAN replication and avoid HH for the local quorum? (since
>>> I know I have more copies)
>>>
>>>
>>
>> I am not sure if I understand your question. In general I don't think you
>> can selectively decide on HH. Besides HH should only be used when the outage
>> is in mts, for longer outages using HH would only create memory pressure.
>>>
>>> On Tuesday, June 26, 2012, Mohit Anchlia wrote:
>>>>
>>>>
>>>> On Tue, Jun 26, 2012 at 7:52 AM, Karthik N <ka...@gmail.com> wrote:
>>>>>
>>>>> My Cassandra ring spans two DCs. I use local quorum with replication
>>>>> factor=3. I do a write in DC1 with local quorum. Data gets written to
>>>>> multiple nodes in DC1. For the same write to propagate to DC2 only one
>>>>> copy is sent from the coordinator node in DC1 to a coordinator node in
>>>>> DC2 for optimizing traffic over the WAN (from what I have read in the
>>>>> Cassandra documentation)
>>>>>
>>>>> Will a Wan hiccup result in a Hinted Handoff (HH) being created in
>>>>> DC1's coordinator for DC2 to be delivered when the Wan link is up
>>>>> again?
>>>>
>>>>
>>>> I have seen hinted handoff messages in the log files when the remote DC
>>>> is unreachable. But this mechanism is only used for a the time defined in
>>>> cassandra.yaml file.
>>>
>>>
>>>
>>> --
>>> Thanks, Karthik
>>
>>
Re: Multi datacenter, WAN hiccups and replication
Posted by Karthik N <ka...@gmail.com>.
I re-read my last post and didn't think I had done a good job articulating.
Sorry! I'll try again...
Say I choose LOCAL_QUORUM with a Replication Factor of 3. Cassandra
stores three copies in my local datacenter. Therefore the cost
associated with "losing" one node is not very high locally, and I
usually disable HH, and use read repair/nodetool repair instead.
However over the WAN, network blips are quite normal and HH really
helps. More so because for WAN replication Cassandra sends only one
copy to a coordinator in the remote datacenter, and it's rather vital
for that copy to make it over to keep the two datacenters in sync.
Therefore I was wondering if Cassandra already intelligently special cases
for HH-over-WAN (since this is common) even if HH is disabled or alternately
if there's a way to enable HH for WAN replication only while disabling it for
the LOCAL_QUORUM?
Thank you.
Thanks, Karthik
On Tue, Jun 26, 2012 at 10:14 AM, Karthik N <ka...@gmail.com> wrote:
> Let me attempt to articulate my question a little better.
>
> Say I choose LOCAL_QUORUM with a Replication Factor of 3. Cassandra
> stores three copies in my local datacenter. Therefore the cost
> associated with "losing" one node is not very high locally, and I
> usually HH, and use read repair/nodetool repair instead.
>
> However over the WAN network blips are quite normal and HH really
> helps. More so because for WAN replication Cassandra sends only one
> copy to a coordinator in the remote datacenter.
>
> Therefore I was wondering if Cassandra already intelligently optimizes
> for HH-over-WAN (since this is common) or alternately if there's a way
> to enable HH for WAN replication?
>
> Thank you.
>
> On Tue, Jun 26, 2012 at 9:22 AM, Mohit Anchlia <mo...@gmail.com> wrote:
>>
>>
>> On Tue, Jun 26, 2012 at 8:16 AM, Karthik N <ka...@gmail.com> wrote:
>>>
>>> Since Cassandra optimizes and sends only one copy over the WAN, can I opt
>>> in only for HH for WAN replication and avoid HH for the local quorum? (since
>>> I know I have more copies)
>>>
>>>
>>
>> I am not sure if I understand your question. In general I don't think you
>> can selectively decide on HH. Besides HH should only be used when the outage
>> is in mts, for longer outages using HH would only create memory pressure.
>>>
>>> On Tuesday, June 26, 2012, Mohit Anchlia wrote:
>>>>
>>>>
>>>> On Tue, Jun 26, 2012 at 7:52 AM, Karthik N <ka...@gmail.com> wrote:
>>>>>
>>>>> My Cassandra ring spans two DCs. I use local quorum with replication
>>>>> factor=3. I do a write in DC1 with local quorum. Data gets written to
>>>>> multiple nodes in DC1. For the same write to propagate to DC2 only one
>>>>> copy is sent from the coordinator node in DC1 to a coordinator node in
>>>>> DC2 for optimizing traffic over the WAN (from what I have read in the
>>>>> Cassandra documentation)
>>>>>
>>>>> Will a Wan hiccup result in a Hinted Handoff (HH) being created in
>>>>> DC1's coordinator for DC2 to be delivered when the Wan link is up
>>>>> again?
>>>>
>>>>
>>>> I have seen hinted handoff messages in the log files when the remote DC
>>>> is unreachable. But this mechanism is only used for a the time defined in
>>>> cassandra.yaml file.
>>>
>>>
>>>
>>> --
>>> Thanks, Karthik
>>
>>
Re: Multi datacenter, WAN hiccups and replication
Posted by Karthik N <ka...@gmail.com>.
Let me attempt to articulate my question a little better.
Say I choose LOCAL_QUORUM with a Replication Factor of 3. Cassandra
stores three copies in my local datacenter. Therefore the cost
associated with "losing" one node is not very high locally, and I
usually HH, and use read repair/nodetool repair instead.
However over the WAN network blips are quite normal and HH really
helps. More so because for WAN replication Cassandra sends only one
copy to a coordinator in the remote datacenter.
Therefore I was wondering if Cassandra already intelligently optimizes
for HH-over-WAN (since this is common) or alternately if there's a way
to enable HH for WAN replication?
Thank you.
On Tue, Jun 26, 2012 at 9:22 AM, Mohit Anchlia <mo...@gmail.com> wrote:
>
>
> On Tue, Jun 26, 2012 at 8:16 AM, Karthik N <ka...@gmail.com> wrote:
>>
>> Since Cassandra optimizes and sends only one copy over the WAN, can I opt
>> in only for HH for WAN replication and avoid HH for the local quorum? (since
>> I know I have more copies)
>>
>>
>
> I am not sure if I understand your question. In general I don't think you
> can selectively decide on HH. Besides HH should only be used when the outage
> is in mts, for longer outages using HH would only create memory pressure.
>>
>> On Tuesday, June 26, 2012, Mohit Anchlia wrote:
>>>
>>>
>>> On Tue, Jun 26, 2012 at 7:52 AM, Karthik N <ka...@gmail.com> wrote:
>>>>
>>>> My Cassandra ring spans two DCs. I use local quorum with replication
>>>> factor=3. I do a write in DC1 with local quorum. Data gets written to
>>>> multiple nodes in DC1. For the same write to propagate to DC2 only one
>>>> copy is sent from the coordinator node in DC1 to a coordinator node in
>>>> DC2 for optimizing traffic over the WAN (from what I have read in the
>>>> Cassandra documentation)
>>>>
>>>> Will a Wan hiccup result in a Hinted Handoff (HH) being created in
>>>> DC1's coordinator for DC2 to be delivered when the Wan link is up
>>>> again?
>>>
>>>
>>> I have seen hinted handoff messages in the log files when the remote DC
>>> is unreachable. But this mechanism is only used for a the time defined in
>>> cassandra.yaml file.
>>
>>
>>
>> --
>> Thanks, Karthik
>
>
Re: Multi datacenter, WAN hiccups and replication
Posted by Mohit Anchlia <mo...@gmail.com>.
On Tue, Jun 26, 2012 at 8:16 AM, Karthik N <ka...@gmail.com> wrote:
> Since Cassandra optimizes and sends only one copy over the WAN, can I opt
> in only for HH for WAN replication and avoid HH for the local quorum?
> (since I know I have more copies)
>
>
>
I am not sure if I understand your question. In general I don't think you
can selectively decide on HH. Besides HH should only be used when the
outage is in mts, for longer outages using HH would only create memory
pressure.
> On Tuesday, June 26, 2012, Mohit Anchlia wrote:
>
>>
>> On Tue, Jun 26, 2012 at 7:52 AM, Karthik N <ka...@gmail.com> wrote:
>>
>>> My Cassandra ring spans two DCs. I use local quorum with replication
>>> factor=3. I do a write in DC1 with local quorum. Data gets written to
>>> multiple nodes in DC1. For the same write to propagate to DC2 only one
>>> copy is sent from the coordinator node in DC1 to a coordinator node in
>>> DC2 for optimizing traffic over the WAN (from what I have read in the
>>> Cassandra documentation)
>>>
>>> Will a Wan hiccup result in a Hinted Handoff (HH) being created in
>>> DC1's coordinator for DC2 to be delivered when the Wan link is up
>>> again?
>>>
>>
>> I have seen hinted handoff messages in the log files when the remote DC
>> is unreachable. But this mechanism is only used for a the time defined in
>> cassandra.yaml file.
>
>
>
> --
> Thanks, Karthik
>
Re: Multi datacenter, WAN hiccups and replication
Posted by Karthik N <ka...@gmail.com>.
Since Cassandra optimizes and sends only one copy over the WAN, can I opt
in only for HH for WAN replication and avoid HH for the local quorum?
(since I know I have more copies)
On Tuesday, June 26, 2012, Mohit Anchlia wrote:
>
> On Tue, Jun 26, 2012 at 7:52 AM, Karthik N <karthik.nar@gmail.com<javascript:_e({}, 'cvml', 'karthik.nar@gmail.com');>
> > wrote:
>
>> My Cassandra ring spans two DCs. I use local quorum with replication
>> factor=3. I do a write in DC1 with local quorum. Data gets written to
>> multiple nodes in DC1. For the same write to propagate to DC2 only one
>> copy is sent from the coordinator node in DC1 to a coordinator node in
>> DC2 for optimizing traffic over the WAN (from what I have read in the
>> Cassandra documentation)
>>
>> Will a Wan hiccup result in a Hinted Handoff (HH) being created in
>> DC1's coordinator for DC2 to be delivered when the Wan link is up
>> again?
>>
>
> I have seen hinted handoff messages in the log files when the remote DC is
> unreachable. But this mechanism is only used for a the time defined in
> cassandra.yaml file.
--
Thanks, Karthik
Re: Multi datacenter, WAN hiccups and replication
Posted by Mohit Anchlia <mo...@gmail.com>.
On Tue, Jun 26, 2012 at 7:52 AM, Karthik N <ka...@gmail.com> wrote:
> My Cassandra ring spans two DCs. I use local quorum with replication
> factor=3. I do a write in DC1 with local quorum. Data gets written to
> multiple nodes in DC1. For the same write to propagate to DC2 only one
> copy is sent from the coordinator node in DC1 to a coordinator node in
> DC2 for optimizing traffic over the WAN (from what I have read in the
> Cassandra documentation)
>
> Will a Wan hiccup result in a Hinted Handoff (HH) being created in
> DC1's coordinator for DC2 to be delivered when the Wan link is up
> again?
>
I have seen hinted handoff messages in the log files when the remote DC is
unreachable. But this mechanism is only used for a the time defined in
cassandra.yaml file.