You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Laxmikant Upadhyay <la...@gmail.com> on 2019/06/11 09:26:20 UTC
Is my range read query behaving strange ?
In a 3 node cassandra 2.1.16 cluster where, one node has old mutation and
two nodes have evict-able (crossed gc grace period) tombstone produced by
TTL. A read range query with local quorum return the old mutation as
result. However expected result should be empty. Next time running the same
query results no data as expected. Why this strange behaviour?
*Steps to Reproduce :*
Create a cassandra-2.1.16 3 node cluster. Disable hinted handoff for each
node.
#ccm node1 nodetool ring
Datacenter: datacenter1
==========
Address Rack Status State Load Owns
Token
3074457345618258602
127.0.0.1 rack1 Up Normal 175.12 KB 100.00%
-9223372036854775808
127.0.0.2 rack1 Up Normal 177.87 KB 100.00%
-3074457345618258603
127.0.0.3 rack1 Up Normal 175.13 KB 100.00%
3074457345618258602
#Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = { 'class' :
'NetworkTopologyStrategy', 'datacenter1' : 3 };
cqlsh> CREATE TABLE test.table1 (key text, col text, val text,PRIMARY KEY
((key), col));
cqlsh> ALTER TABLE test.table1 with GC_GRACE_SECONDS = 120;
cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
'abc','xyz');
#ccm flush
#ccm node3 stop
cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
'abc','xyz') USING TTL 60;
#ccm flush
#wait for 3 min so that the tombstone crosses its gc grace period.
#ccm node3 start
cqlsh> select * from test.table1 where token (key) > 3074457345618258602
and token (key) < -9223372036854775808 ;
key | col | val
------+-----+-----
key2 | abc | xyz
(1 rows)
#ccm flush
-> Here read repair triggers and the old mutation moves to the one of the
node where tombstone is present (not both the node)
cqlsh> select * from test.vouchers where token (key) > 3074457345618258602
and token (key) < -9223372036854775808 ;
key | col | val
-----+-----+-----
(0 rows)
--
regards,
Laxmikant Upadhyay
Re: Is my range read query behaving strange ?
Posted by Laxmikant Upadhyay <la...@gmail.com>.
Raised a ticket https://issues.apache.org/jira/browse/CASSANDRA-15159 for
the same.
On Thu, Jun 13, 2019 at 3:55 PM Laxmikant Upadhyay <la...@gmail.com>
wrote:
> This issue is reproducible on *3.11.4 and 2.1.21* as well. (not yet
> checked on 3.0)
>
> Range query could be : select * from test.table1; (In this case, Read
> repair actually sending old mutation to the node which has tombstone )
> I also ran normal read and it also returns a row in this case instead of
> empty response.
>
> cqlsh> select * from test.table1;
>
> key | col | val
> ------+-----+-----
> key2 | abc | xyz
>
> (1 rows)
> cqlsh> select * from test.table1 where key = 'key2';
>
> key | col | val
> ------+-----+-----
> key2 | abc | xyz
>
> Why is this behaviour ? Why read is ignoring purge-able tombstone ?
>
>
>
> On Thu, Jun 13, 2019 at 12:30 PM Laxmikant Upadhyay <
> laxmikant.hcl@gmail.com> wrote:
>
>> HI Michael,
>>
>> Thanks for your reply.
>> I don't think this issue is related to CASSANDRA-12765
>> <https://issues.apache.org/jira/browse/CASSANDRA-12765> as in my case
>> the sstable which has tombstone does not have maxLocalDeletionTime ==
>> Integer.MAX_VALUE . I am able to reproduce this issue on 2.1.17 as well.
>>
>> I am attaching the steps to reproduce on 2.1.17 (with minor change from
>> previous steps to make sure one request must go to the node which has old
>> mutation). I have also attached the trace of range read query.
>>
>> Should I raise a jira for the same ?
>>
>> On Wed, Jun 12, 2019 at 9:00 AM Michael Shuler <mi...@pbandjelly.org>
>> wrote:
>>
>>> (dropped dev@ x-post; user@ was correct)
>>>
>>> Possibly #12765, fixed in 2.1.17. Wouldn't hurt to update to latest
>>> 2.1.21.
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-12765
>>> https://github.com/apache/cassandra/blob/cassandra-2.1/CHANGES.txt#L1-L36
>>>
>>> Michael
>>>
>>> On 6/11/19 9:58 PM, Laxmikant Upadhyay wrote:
>>> > Does range query ignore purgable tombstone (which crossed grace
>>> period)
>>> > in some cases?
>>> >
>>> > On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay
>>> > <laxmikant.hcl@gmail.com <ma...@gmail.com>> wrote:
>>> >
>>> > In a 3 node cassandra 2.1.16 cluster where, one node has old
>>> > mutation and two nodes have evict-able (crossed gc grace period)
>>> > tombstone produced by TTL. A read range query with local quorum
>>> > return the old mutation as result. However expected result should
>>> be
>>> > empty. Next time running the same query results no data as
>>> expected.
>>> > Why this strange behaviour?
>>> >
>>> >
>>> > *Steps to Reproduce :*
>>> > Create a cassandra-2.1.16 3 node cluster. Disable hinted handoff
>>> > for each node.
>>> >
>>> > #ccm node1 nodetool ring
>>> > Datacenter: datacenter1
>>> > ==========
>>> > Address Rack Status State Load Owns
>>>
>>> > Token
>>> >
>>>
>>> > 3074457345618258602
>>> > 127.0.0.1 rack1 Up Normal 175.12 KB 100.00%
>>> > -9223372036854775808
>>> > 127.0.0.2 rack1 Up Normal 177.87 KB 100.00%
>>> > -3074457345618258603
>>> > 127.0.0.3 rack1 Up Normal 175.13 KB 100.00%
>>> > 3074457345618258602
>>> >
>>> > #Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
>>> >
>>> > cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = {
>>> > 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 3 };
>>> > cqlsh> CREATE TABLE test.table1 (key text, col text, val
>>> > text,PRIMARY KEY ((key), col));
>>> > cqlsh> ALTER TABLE test.table1 with GC_GRACE_SECONDS = 120;
>>> >
>>> > cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
>>> > 'abc','xyz');
>>> >
>>> > #ccm flush
>>> >
>>> > #ccm node3 stop
>>> >
>>> > cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
>>> > 'abc','xyz') USING TTL 60;
>>> >
>>> > #ccm flush
>>> >
>>> > #wait for 3 min so that the tombstone crosses its gc grace period.
>>> >
>>> > #ccm node3 start
>>> >
>>> > cqlsh> select * from test.table1 where token (key) >
>>> > 3074457345618258602 and token (key) < -9223372036854775808 ;
>>> >
>>> > key | col | val
>>> > ------+-----+-----
>>> > key2 | abc | xyz
>>> >
>>> > (1 rows)
>>> >
>>> > #ccm flush
>>> > -> Here read repair triggers and the old mutation moves to the one
>>> > of the node where tombstone is present (not both the node)
>>> >
>>> >
>>> > cqlsh> select * from test.vouchers where token (key) >
>>> > 3074457345618258602 and token (key) < -9223372036854775808 ;
>>> >
>>> > key | col | val
>>> > -----+-----+-----
>>> >
>>> > (0 rows)
>>> >
>>> >
>>> > --
>>> >
>>> > regards,
>>> > Laxmikant Upadhyay
>>> >
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>
>>>
>>
>> --
>>
>> regards,
>> Laxmikant Upadhyay
>>
>>
>
> --
>
> regards,
> Laxmikant Upadhyay
>
>
--
regards,
Laxmikant Upadhyay
Re: Is my range read query behaving strange ?
Posted by Laxmikant Upadhyay <la...@gmail.com>.
This issue is reproducible on *3.11.4 and 2.1.21* as well. (not yet
checked on 3.0)
Range query could be : select * from test.table1; (In this case, Read
repair actually sending old mutation to the node which has tombstone )
I also ran normal read and it also returns a row in this case instead of
empty response.
cqlsh> select * from test.table1;
key | col | val
------+-----+-----
key2 | abc | xyz
(1 rows)
cqlsh> select * from test.table1 where key = 'key2';
key | col | val
------+-----+-----
key2 | abc | xyz
Why is this behaviour ? Why read is ignoring purge-able tombstone ?
On Thu, Jun 13, 2019 at 12:30 PM Laxmikant Upadhyay <la...@gmail.com>
wrote:
> HI Michael,
>
> Thanks for your reply.
> I don't think this issue is related to CASSANDRA-12765
> <https://issues.apache.org/jira/browse/CASSANDRA-12765> as in my case the
> sstable which has tombstone does not have maxLocalDeletionTime ==
> Integer.MAX_VALUE . I am able to reproduce this issue on 2.1.17 as well.
>
> I am attaching the steps to reproduce on 2.1.17 (with minor change from
> previous steps to make sure one request must go to the node which has old
> mutation). I have also attached the trace of range read query.
>
> Should I raise a jira for the same ?
>
> On Wed, Jun 12, 2019 at 9:00 AM Michael Shuler <mi...@pbandjelly.org>
> wrote:
>
>> (dropped dev@ x-post; user@ was correct)
>>
>> Possibly #12765, fixed in 2.1.17. Wouldn't hurt to update to latest
>> 2.1.21.
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-12765
>> https://github.com/apache/cassandra/blob/cassandra-2.1/CHANGES.txt#L1-L36
>>
>> Michael
>>
>> On 6/11/19 9:58 PM, Laxmikant Upadhyay wrote:
>> > Does range query ignore purgable tombstone (which crossed grace period)
>> > in some cases?
>> >
>> > On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay
>> > <laxmikant.hcl@gmail.com <ma...@gmail.com>> wrote:
>> >
>> > In a 3 node cassandra 2.1.16 cluster where, one node has old
>> > mutation and two nodes have evict-able (crossed gc grace period)
>> > tombstone produced by TTL. A read range query with local quorum
>> > return the old mutation as result. However expected result should be
>> > empty. Next time running the same query results no data as expected.
>> > Why this strange behaviour?
>> >
>> >
>> > *Steps to Reproduce :*
>> > Create a cassandra-2.1.16 3 node cluster. Disable hinted handoff
>> > for each node.
>> >
>> > #ccm node1 nodetool ring
>> > Datacenter: datacenter1
>> > ==========
>> > Address Rack Status State Load Owns
>> > Token
>> >
>> > 3074457345618258602
>> > 127.0.0.1 rack1 Up Normal 175.12 KB 100.00%
>> > -9223372036854775808
>> > 127.0.0.2 rack1 Up Normal 177.87 KB 100.00%
>> > -3074457345618258603
>> > 127.0.0.3 rack1 Up Normal 175.13 KB 100.00%
>> > 3074457345618258602
>> >
>> > #Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
>> >
>> > cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = {
>> > 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 3 };
>> > cqlsh> CREATE TABLE test.table1 (key text, col text, val
>> > text,PRIMARY KEY ((key), col));
>> > cqlsh> ALTER TABLE test.table1 with GC_GRACE_SECONDS = 120;
>> >
>> > cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
>> > 'abc','xyz');
>> >
>> > #ccm flush
>> >
>> > #ccm node3 stop
>> >
>> > cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
>> > 'abc','xyz') USING TTL 60;
>> >
>> > #ccm flush
>> >
>> > #wait for 3 min so that the tombstone crosses its gc grace period.
>> >
>> > #ccm node3 start
>> >
>> > cqlsh> select * from test.table1 where token (key) >
>> > 3074457345618258602 and token (key) < -9223372036854775808 ;
>> >
>> > key | col | val
>> > ------+-----+-----
>> > key2 | abc | xyz
>> >
>> > (1 rows)
>> >
>> > #ccm flush
>> > -> Here read repair triggers and the old mutation moves to the one
>> > of the node where tombstone is present (not both the node)
>> >
>> >
>> > cqlsh> select * from test.vouchers where token (key) >
>> > 3074457345618258602 and token (key) < -9223372036854775808 ;
>> >
>> > key | col | val
>> > -----+-----+-----
>> >
>> > (0 rows)
>> >
>> >
>> > --
>> >
>> > regards,
>> > Laxmikant Upadhyay
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>
> --
>
> regards,
> Laxmikant Upadhyay
>
>
--
regards,
Laxmikant Upadhyay
Re: Is my range read query behaving strange ?
Posted by Laxmikant Upadhyay <la...@gmail.com>.
HI Michael,
Thanks for your reply.
I don't think this issue is related to CASSANDRA-12765
<https://issues.apache.org/jira/browse/CASSANDRA-12765> as in my case the
sstable which has tombstone does not have maxLocalDeletionTime ==
Integer.MAX_VALUE . I am able to reproduce this issue on 2.1.17 as well.
I am attaching the steps to reproduce on 2.1.17 (with minor change from
previous steps to make sure one request must go to the node which has old
mutation). I have also attached the trace of range read query.
Should I raise a jira for the same ?
On Wed, Jun 12, 2019 at 9:00 AM Michael Shuler <mi...@pbandjelly.org>
wrote:
> (dropped dev@ x-post; user@ was correct)
>
> Possibly #12765, fixed in 2.1.17. Wouldn't hurt to update to latest 2.1.21.
>
> https://issues.apache.org/jira/browse/CASSANDRA-12765
> https://github.com/apache/cassandra/blob/cassandra-2.1/CHANGES.txt#L1-L36
>
> Michael
>
> On 6/11/19 9:58 PM, Laxmikant Upadhyay wrote:
> > Does range query ignore purgable tombstone (which crossed grace period)
> > in some cases?
> >
> > On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay
> > <laxmikant.hcl@gmail.com <ma...@gmail.com>> wrote:
> >
> > In a 3 node cassandra 2.1.16 cluster where, one node has old
> > mutation and two nodes have evict-able (crossed gc grace period)
> > tombstone produced by TTL. A read range query with local quorum
> > return the old mutation as result. However expected result should be
> > empty. Next time running the same query results no data as expected.
> > Why this strange behaviour?
> >
> >
> > *Steps to Reproduce :*
> > Create a cassandra-2.1.16 3 node cluster. Disable hinted handoff
> > for each node.
> >
> > #ccm node1 nodetool ring
> > Datacenter: datacenter1
> > ==========
> > Address Rack Status State Load Owns
> > Token
> >
> > 3074457345618258602
> > 127.0.0.1 rack1 Up Normal 175.12 KB 100.00%
> > -9223372036854775808
> > 127.0.0.2 rack1 Up Normal 177.87 KB 100.00%
> > -3074457345618258603
> > 127.0.0.3 rack1 Up Normal 175.13 KB 100.00%
> > 3074457345618258602
> >
> > #Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
> >
> > cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = {
> > 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 3 };
> > cqlsh> CREATE TABLE test.table1 (key text, col text, val
> > text,PRIMARY KEY ((key), col));
> > cqlsh> ALTER TABLE test.table1 with GC_GRACE_SECONDS = 120;
> >
> > cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
> > 'abc','xyz');
> >
> > #ccm flush
> >
> > #ccm node3 stop
> >
> > cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
> > 'abc','xyz') USING TTL 60;
> >
> > #ccm flush
> >
> > #wait for 3 min so that the tombstone crosses its gc grace period.
> >
> > #ccm node3 start
> >
> > cqlsh> select * from test.table1 where token (key) >
> > 3074457345618258602 and token (key) < -9223372036854775808 ;
> >
> > key | col | val
> > ------+-----+-----
> > key2 | abc | xyz
> >
> > (1 rows)
> >
> > #ccm flush
> > -> Here read repair triggers and the old mutation moves to the one
> > of the node where tombstone is present (not both the node)
> >
> >
> > cqlsh> select * from test.vouchers where token (key) >
> > 3074457345618258602 and token (key) < -9223372036854775808 ;
> >
> > key | col | val
> > -----+-----+-----
> >
> > (0 rows)
> >
> >
> > --
> >
> > regards,
> > Laxmikant Upadhyay
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
--
regards,
Laxmikant Upadhyay
Re: Is my range read query behaving strange ?
Posted by Michael Shuler <mi...@pbandjelly.org>.
(dropped dev@ x-post; user@ was correct)
Possibly #12765, fixed in 2.1.17. Wouldn't hurt to update to latest 2.1.21.
https://issues.apache.org/jira/browse/CASSANDRA-12765
https://github.com/apache/cassandra/blob/cassandra-2.1/CHANGES.txt#L1-L36
Michael
On 6/11/19 9:58 PM, Laxmikant Upadhyay wrote:
> Does range query ignore purgable tombstone (which crossed grace period)
> in some cases?
>
> On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay
> <laxmikant.hcl@gmail.com <ma...@gmail.com>> wrote:
>
> In a 3 node cassandra 2.1.16 cluster where, one node has old
> mutation and two nodes have evict-able (crossed gc grace period)
> tombstone produced by TTL. A read range query with local quorum
> return the old mutation as result. However expected result should be
> empty. Next time running the same query results no data as expected.
> Why this strange behaviour?
>
>
> *Steps to Reproduce :*
> Create a cassandra-2.1.16 3 node cluster. Disable hinted handoff
> for each node.
>
> #ccm node1 nodetool ring
> Datacenter: datacenter1
> ==========
> Address Rack Status State Load Owns
> Token
>
> 3074457345618258602
> 127.0.0.1 rack1 Up Normal 175.12 KB 100.00%
> -9223372036854775808
> 127.0.0.2 rack1 Up Normal 177.87 KB 100.00%
> -3074457345618258603
> 127.0.0.3 rack1 Up Normal 175.13 KB 100.00%
> 3074457345618258602
>
> #Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
>
> cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = {
> 'class' : 'NetworkTopologyStrategy', 'datacenter1' : 3 };
> cqlsh> CREATE TABLE test.table1 (key text, col text, val
> text,PRIMARY KEY ((key), col));
> cqlsh> ALTER TABLE test.table1 with GC_GRACE_SECONDS = 120;
>
> cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
> 'abc','xyz');
>
> #ccm flush
>
> #ccm node3 stop
>
> cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
> 'abc','xyz') USING TTL 60;
>
> #ccm flush
>
> #wait for 3 min so that the tombstone crosses its gc grace period.
>
> #ccm node3 start
>
> cqlsh> select * from test.table1 where token (key) >
> 3074457345618258602 and token (key) < -9223372036854775808 ;
>
> key | col | val
> ------+-----+-----
> key2 | abc | xyz
>
> (1 rows)
>
> #ccm flush
> -> Here read repair triggers and the old mutation moves to the one
> of the node where tombstone is present (not both the node)
>
>
> cqlsh> select * from test.vouchers where token (key) >
> 3074457345618258602 and token (key) < -9223372036854775808 ;
>
> key | col | val
> -----+-----+-----
>
> (0 rows)
>
>
> --
>
> regards,
> Laxmikant Upadhyay
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Is my range read query behaving strange ?
Posted by Laxmikant Upadhyay <la...@gmail.com>.
Does range query ignore purgable tombstone (which crossed grace period) in
some cases?
On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay <la...@gmail.com>
wrote:
> In a 3 node cassandra 2.1.16 cluster where, one node has old mutation and
> two nodes have evict-able (crossed gc grace period) tombstone produced by
> TTL. A read range query with local quorum return the old mutation as
> result. However expected result should be empty. Next time running the same
> query results no data as expected. Why this strange behaviour?
>
>
> *Steps to Reproduce :*
> Create a cassandra-2.1.16 3 node cluster. Disable hinted handoff for each
> node.
>
> #ccm node1 nodetool ring
> Datacenter: datacenter1
> ==========
> Address Rack Status State Load Owns
> Token
>
> 3074457345618258602
> 127.0.0.1 rack1 Up Normal 175.12 KB 100.00%
> -9223372036854775808
> 127.0.0.2 rack1 Up Normal 177.87 KB 100.00%
> -3074457345618258603
> 127.0.0.3 rack1 Up Normal 175.13 KB 100.00%
> 3074457345618258602
>
> #Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
>
> cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = { 'class' :
> 'NetworkTopologyStrategy', 'datacenter1' : 3 };
> cqlsh> CREATE TABLE test.table1 (key text, col text, val text,PRIMARY KEY
> ((key), col));
> cqlsh> ALTER TABLE test.table1 with GC_GRACE_SECONDS = 120;
>
> cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
> 'abc','xyz');
>
> #ccm flush
>
> #ccm node3 stop
>
> cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
> 'abc','xyz') USING TTL 60;
>
> #ccm flush
>
> #wait for 3 min so that the tombstone crosses its gc grace period.
>
> #ccm node3 start
>
> cqlsh> select * from test.table1 where token (key) > 3074457345618258602
> and token (key) < -9223372036854775808 ;
>
> key | col | val
> ------+-----+-----
> key2 | abc | xyz
>
> (1 rows)
>
> #ccm flush
> -> Here read repair triggers and the old mutation moves to the one of the
> node where tombstone is present (not both the node)
>
>
> cqlsh> select * from test.vouchers where token (key) > 3074457345618258602
> and token (key) < -9223372036854775808 ;
>
> key | col | val
> -----+-----+-----
>
> (0 rows)
>
>
> --
>
> regards,
> Laxmikant Upadhyay
>
>
Re: Is my range read query behaving strange ?
Posted by Laxmikant Upadhyay <la...@gmail.com>.
Does range query ignore purgable tombstone (which crossed grace period) in
some cases?
On Tue, Jun 11, 2019, 2:56 PM Laxmikant Upadhyay <la...@gmail.com>
wrote:
> In a 3 node cassandra 2.1.16 cluster where, one node has old mutation and
> two nodes have evict-able (crossed gc grace period) tombstone produced by
> TTL. A read range query with local quorum return the old mutation as
> result. However expected result should be empty. Next time running the same
> query results no data as expected. Why this strange behaviour?
>
>
> *Steps to Reproduce :*
> Create a cassandra-2.1.16 3 node cluster. Disable hinted handoff for each
> node.
>
> #ccm node1 nodetool ring
> Datacenter: datacenter1
> ==========
> Address Rack Status State Load Owns
> Token
>
> 3074457345618258602
> 127.0.0.1 rack1 Up Normal 175.12 KB 100.00%
> -9223372036854775808
> 127.0.0.2 rack1 Up Normal 177.87 KB 100.00%
> -3074457345618258603
> 127.0.0.3 rack1 Up Normal 175.13 KB 100.00%
> 3074457345618258602
>
> #Connect to cqlsh and set CONISISTENCY LOCAL_QUORUM;
>
> cqlsh> CREATE KEYSPACE IF NOT EXISTS test WITH REPLICATION = { 'class' :
> 'NetworkTopologyStrategy', 'datacenter1' : 3 };
> cqlsh> CREATE TABLE test.table1 (key text, col text, val text,PRIMARY KEY
> ((key), col));
> cqlsh> ALTER TABLE test.table1 with GC_GRACE_SECONDS = 120;
>
> cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
> 'abc','xyz');
>
> #ccm flush
>
> #ccm node3 stop
>
> cqlsh> INSERT INTO test.table1 (key, col, val) VALUES ('key2',
> 'abc','xyz') USING TTL 60;
>
> #ccm flush
>
> #wait for 3 min so that the tombstone crosses its gc grace period.
>
> #ccm node3 start
>
> cqlsh> select * from test.table1 where token (key) > 3074457345618258602
> and token (key) < -9223372036854775808 ;
>
> key | col | val
> ------+-----+-----
> key2 | abc | xyz
>
> (1 rows)
>
> #ccm flush
> -> Here read repair triggers and the old mutation moves to the one of the
> node where tombstone is present (not both the node)
>
>
> cqlsh> select * from test.vouchers where token (key) > 3074457345618258602
> and token (key) < -9223372036854775808 ;
>
> key | col | val
> -----+-----+-----
>
> (0 rows)
>
>
> --
>
> regards,
> Laxmikant Upadhyay
>
>