[jira] [Commented] (CASSANDRA-6732) Cross DC writes not compatible 1.2->1.1 during rolling upgrade

["Shawn Smith (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-6732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14103203#comment-14103203 ] 

Shawn Smith commented on CASSANDRA-6732:
----------------------------------------

This week I ran into this problem testing upgrade from 1.1.12 to 1.2.18.  I want to document for others that, as far as I can tell, the attached patch fixes one case but not all.  When the local coordinator wants to relay cross-DC writes through a remote relay, there are 4 cases to consider (based on my tests and reading the code):

# Local coordinator is Cassandra 1.1, remote relay is Cassandra 1.1: \\ Everything works. The old "FORWARD" header is used as expected.
# Local coordinator is Cassandra 1.2, remote relay is Cassandra 1.1: \\ Everything works due to this patch.  The Cassandra 1.2 coordinator will send individual messages to all the remote DC servers.
# Local coordinator is Cassandra 1.2, remote relay is Cassandra 1.2: \\ All Cassandra 1.1 servers in the remote DC receive the write but ack to the remote relay instead of the coordinator.  This is because the Cassandra 1.1 servers ignore the new "FWD_FRM" header and, starting with Cassandra 1.2, the relay sets "message.from" to itself on the forwarded messages.  The net effect: writes get applied, but EACH_QUORUM, ALL writes fail due to lost acks.
# Local coordinator is Cassandra 1.1, remote relay is Cassandra 1.2: \\ The remote relay ignores the "FORWARD" header and does not forward messages.  The net effect: only the remote relay applies the write, other servers in the remote DC drop the write.  EACH_QUORUM, ALL writes fail and data is left inconsistent, repair is required if hints are lost.

Apologies if I have any mistakes here.  It's hard to test each combination independently.

> Cross DC writes not compatible 1.2->1.1 during rolling upgrade
> --------------------------------------------------------------
>
>                 Key: CASSANDRA-6732
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-6732
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>            Reporter: Jeremiah Jordan
>            Assignee: Jeremiah Jordan
>             Fix For: 1.2.16
>
>         Attachments: 0001-Don-t-try-to-do-Cross-DC-forwarding-to-old-nodes.patch
>
>
> During a rolling upgrade from 1.1.12 to 1.2.15 one DC at a time only 1/3 of the writes to the first DC to be upgraded actually make it to the other DC, and LOTS of hints attempt to get made.
> Looks like the header for forwarded writes changed from 1.1->1.2 so the 1.1 nodes can't read it.



--
This message was sent by Atlassian JIRA
(v6.2#6252)