Consistency Problem with Quorum consistencyLevel configuration

[Jason Tang <ar...@gmail.com>]

Hi

    I met the consistency problem when we have Quorum for both read and
write.

    I use MultigetSubSliceQuery to query rows from super column limit size
100, and then read it, then delete it. And start another around.

    But I found, the row which should be delete by last query, it still
shown from next around query.

    And also form normal Column Family, I updated the value of one column
from status='FALSE' to status='TURE', and next time I query it, the status
still 'FALSE'.

    More detail:

   - It not happened not every time (1/10,000)
   - The time between two round query is around 500 ms (but we found two
   query which 2 seconds happened later then the first one, still have this
   consistency problem)
   - We use ntp as our cluster time synchronization solution.
   - We have 6 nodes, and replication factor is 3

    Some body say, Cassandra suppose to have such problem, because read may
not happen before write inside Cassandra. But for two seconds?! And if so,
it meaningless to have Quorum or other consistency level configuration.

   So first of all, is it the correct behavior of Cassandra, and if not,
what data we need to analyze for further investment.

BRs
Ares

Re: Consistency Problem with Quorum consistencyLevel configuration

[Jason Tang <ar...@gmail.com>]

Hi
  After enable Cassandra debug log, I got following log, it shows the
delete mutation send to other two nodes rather then local node.
  And then the read command come to the local nodes.
  And local one found the mismatch.
  But I don't know why local node return the local dirty data. It supposed
to repair the data, and return correct one?

192.168.0.6:
DEBUG [MutationStage:61] 2012-06-26 23:09:00,036
RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc',
key='33323130537570657254616e6730', modifications=[ColumnFamily(queue
-deleted at 1340723340044000- [])]) applied.  Sending response to 3555@/
192.168.0.5

192.168.0.4:
DEBUG [MutationStage:40] 2012-06-26 23:09:00,041
RowMutationVerbHandler.java (line 60) RowMutation(keyspace='drc',
key='33323130537570657254616e6730', modifications=[ColumnFamily(queue
-deleted at 1340723340044000- [])]) applied.  Sending response to 3556@/
192.168.0.5

192.168.0.5 (local one):
DEBUG [pool-2-thread-20] 2012-06-26 23:09:00,105 StorageProxy.java (line
705) Digest mismatch: org.apache.cassandra.service.DigestMismatchException:
Mismatch for key DecoratedKey(76499729728376587390748888639933581556,
33323130537570657254616e6730) (b20ac6ec0d29393d70e200027c094d13 vs
d41d8cd98f00b204e9800998ecf8427e)



2012/6/25 Jason Tang <ar...@gmail.com>

> Hi
>
>     I met the consistency problem when we have Quorum for both read and
> write.
>
>     I use MultigetSubSliceQuery to query rows from super column limit size
> 100, and then read it, then delete it. And start another around.
>
>     But I found, the row which should be delete by last query, it still
> shown from next around query.
>
>     And also form normal Column Family, I updated the value of one column
> from status='FALSE' to status='TURE', and next time I query it, the status
> still 'FALSE'.
>
>     More detail:
>
>    - It not happened not every time (1/10,000)
>    - The time between two round query is around 500 ms (but we found two
>    query which 2 seconds happened later then the first one, still have this
>    consistency problem)
>    - We use ntp as our cluster time synchronization solution.
>    - We have 6 nodes, and replication factor is 3
>
>     Some body say, Cassandra suppose to have such problem, because read
> may not happen before write inside Cassandra. But for two seconds?! And if
> so, it meaningless to have Quorum or other consistency level configuration.
>
>    So first of all, is it the correct behavior of Cassandra, and if not,
> what data we need to analyze for further investment.
>
> BRs
> Ares
>