You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jared Rodriguez <jr...@kitedesk.com> on 2015/05/12 11:50:55 UTC
Consistency Issues
I have a specific update and query that I need to ensure has strong
consistency. To that end, when I do the write, I set the consistency level
to ALL. Shortly afterwards, I do a query for that record with a
consistency of ONE and somehow get back stale data.
This is a 4 node cluster running Cassandra 2.0.6 on Ubuntu on AWS. The
keyspace has a replication factor of 2. I am using the DataStax Java
driver version 2.1.4.
Updates to the table happen with prepared statements.
Any help would be greatly appreciated.
--
Jared Rodriguez
Re: Consistency Issues
Posted by Jared Rodriguez <jr...@kitedesk.com>.
It looks like NTP was the problem. Thanks for the solution!!!
On Wed, May 13, 2015 at 9:20 AM, Robert Wille <rw...@fold3.com> wrote:
> Timestamps have millisecond granularity. If you make multiple writes
> within the same millisecond, then the outcome is not deterministic.
>
> Also, make sure you are running ntp. Clock skew will manifest itself
> similarly.
>
> On May 13, 2015, at 3:47 AM, Jared Rodriguez <jr...@kitedesk.com>
> wrote:
>
> Thanks for the feedback. We have dug in deeper and upgraded to
> Cassandra 2.0.14 and are seeing the same issue. What appears to be
> happening is that if a record is initially written, then the first read is
> fine. But if we immediately update that record with a second write, that
> then the second read is problematic.
>
> We have a 4 node cluster and a replication factor of 2. What seems to
> be happening on the initial write the record is sent to nodes A and B. If
> a secondary write (update) of the record occurs while the record is in the
> memtable and not yet written to the sstable of A or B, that the next read
> returns nothing.
>
> We are continuing to dig in and get as much detail as possible before
> opening this as a JIRA.
>
> On Tue, May 12, 2015 at 6:51 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Tue, May 12, 2015 at 12:35 PM, Michael Shuler <michael@pbandjelly.org
>> > wrote:
>>
>>> This is a 4 node cluster running Cassandra 2.0.6
>>>>
>>>
>>> Can you reproduce the same issue on 2.0.14? (or better yet, the
>>> cassandra-2.0 branch HEAD, which will soon ship 2.0.15) If you get the same
>>> results, please, open a JIRA with the reproduction steps.
>>
>>
>> And if you do file such a JIRA, please let the list know the JIRA URL,
>> to close the loop!
>>
>> =Rob
>>
>>
>
>
>
> --
> Jared Rodriguez
>
>
>
--
Jared Rodriguez
Re: Consistency Issues
Posted by Robert Wille <rw...@fold3.com>.
Timestamps have millisecond granularity. If you make multiple writes within the same millisecond, then the outcome is not deterministic.
Also, make sure you are running ntp. Clock skew will manifest itself similarly.
On May 13, 2015, at 3:47 AM, Jared Rodriguez <jr...@kitedesk.com>> wrote:
Thanks for the feedback. We have dug in deeper and upgraded to Cassandra 2.0.14 and are seeing the same issue. What appears to be happening is that if a record is initially written, then the first read is fine. But if we immediately update that record with a second write, that then the second read is problematic.
We have a 4 node cluster and a replication factor of 2. What seems to be happening on the initial write the record is sent to nodes A and B. If a secondary write (update) of the record occurs while the record is in the memtable and not yet written to the sstable of A or B, that the next read returns nothing.
We are continuing to dig in and get as much detail as possible before opening this as a JIRA.
On Tue, May 12, 2015 at 6:51 PM, Robert Coli <rc...@eventbrite.com>> wrote:
On Tue, May 12, 2015 at 12:35 PM, Michael Shuler <mi...@pbandjelly.org>> wrote:
This is a 4 node cluster running Cassandra 2.0.6
Can you reproduce the same issue on 2.0.14? (or better yet, the cassandra-2.0 branch HEAD, which will soon ship 2.0.15) If you get the same results, please, open a JIRA with the reproduction steps.
And if you do file such a JIRA, please let the list know the JIRA URL, to close the loop!
=Rob
--
Jared Rodriguez
Re: Consistency Issues
Posted by Jared Rodriguez <jr...@kitedesk.com>.
Thanks for the feedback. We have dug in deeper and upgraded to Cassandra
2.0.14 and are seeing the same issue. What appears to be happening is that
if a record is initially written, then the first read is fine. But if we
immediately update that record with a second write, that then the second
read is problematic.
We have a 4 node cluster and a replication factor of 2. What seems to be
happening on the initial write the record is sent to nodes A and B. If a
secondary write (update) of the record occurs while the record is in the
memtable and not yet written to the sstable of A or B, that the next read
returns nothing.
We are continuing to dig in and get as much detail as possible before
opening this as a JIRA.
On Tue, May 12, 2015 at 6:51 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Tue, May 12, 2015 at 12:35 PM, Michael Shuler <mi...@pbandjelly.org>
> wrote:
>
>> This is a 4 node cluster running Cassandra 2.0.6
>>>
>>
>> Can you reproduce the same issue on 2.0.14? (or better yet, the
>> cassandra-2.0 branch HEAD, which will soon ship 2.0.15) If you get the same
>> results, please, open a JIRA with the reproduction steps.
>
>
> And if you do file such a JIRA, please let the list know the JIRA URL, to
> close the loop!
>
> =Rob
>
>
--
Jared Rodriguez
Re: Consistency Issues
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, May 12, 2015 at 12:35 PM, Michael Shuler <mi...@pbandjelly.org>
wrote:
> This is a 4 node cluster running Cassandra 2.0.6
>>
>
> Can you reproduce the same issue on 2.0.14? (or better yet, the
> cassandra-2.0 branch HEAD, which will soon ship 2.0.15) If you get the same
> results, please, open a JIRA with the reproduction steps.
And if you do file such a JIRA, please let the list know the JIRA URL, to
close the loop!
=Rob
Re: Consistency Issues
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 05/12/2015 04:50 AM, Jared Rodriguez wrote:
> I have a specific update and query that I need to ensure has strong
> consistency. To that end, when I do the write, I set the consistency
> level to ALL. Shortly afterwards, I do a query for that record with a
> consistency of ONE and somehow get back stale data.
>
> This is a 4 node cluster running Cassandra 2.0.6
Can you reproduce the same issue on 2.0.14? (or better yet, the
cassandra-2.0 branch HEAD, which will soon ship 2.0.15) If you get the
same results, please, open a JIRA with the reproduction steps.
--
Kind regards,
Michael