[jira] [Created] (CASSANDRA-12683) Batch statement fails with consistency ONE when only 1 node up in DC

["Andy Klages (JIRA)" <ji...@apache.org>]

Andy Klages created CASSANDRA-12683:
---------------------------------------

             Summary: Batch statement fails with consistency ONE when only 1 node up in DC
                 Key: CASSANDRA-12683
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-12683
             Project: Cassandra
          Issue Type: Bug
          Components: Coordination
         Environment: 3 Cassandra nodes (N1, N2, N3)
2 Data Centers (DC1, DC2)
N1 and N2 members of DC1
N3 a member of DC2
1 keyspace using SimpleReplicationStrategy with RF=3 (can also be 2):
CREATE KEYSPACE ks WITH REPLICATION={'class':'SimpleStrategy','replication_factor':3};
1 column family in the keyspace. For simplicity, a partition key of bigint and one other field of any type. No cluster key needed:
CREATE TABLE ks.test (
        id      bigint,
        flag    boolean,
        PRIMARY KEY(id)
);

            Reporter: Andy Klages
            Priority: Minor


If Cassandra node N2 is stopped, that only leaves one node (N1) in DC1 running. Output from "nodetool status" is:

Datacenter: DC1
=========================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns    Host ID                               Rack
DN  N2            ...
UN  N1            ...
Datacenter: DC2
======================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
--  Address        Load       Tokens  Owns    Host ID                               Rack
UN  N3            ...


The following batch statement will fail when executed on N1 with consistency level of ONE using cqlsh (also fails with Java using Datastax driver):

CONSISTENCY ONE
BEGIN BATCH
UPDATE ks.test SET flag=true where id=1;
UPDATE ks.test SET flag=true where id=2;
APPLY BATCH;

The failure is:

  File "apache-cassandra-2.1.15/bin/../lib/cassandra-driver-internal-only-2.7.2.zip/cassandra-driver-2.7.2/cassandra/cluster.py", line 3347, in result
    raise self._final_exception
Unavailable: code=1000 [Unavailable exception] message="Cannot achieve consistency level ONE" info={'required_replicas': 1, 'alive_replicas': 0, 'consistency': 'ONE'}

There are 2 replicas alive but for some reason, Cassandra thinks there are none.

If each statement is executed individually (i.e. no batch), each one succeeds.

This same batch query succeeds on N3, which is the other node in the cluster that is running. My analysis shows this happens when the query is executed on a node in a DC where all other nodes in that DC is down. The batch statement has 2 or more queries with different partition keys. If all of the partition keys are the same value, it succeeds. As the replication factor is set to the number of nodes in the cluster (full replication) and the consistency level is ONE, the batch statement should succeed.

2 workarounds for this are:

1. Set the consistency level to ANY.
2. Use an unlogged batch.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)