Cassandra version 1.0.10 Data Loss upon restart

[Ankit Patel <an...@gmail.com>]

We are experiencing data loss with Cassandra 1.0.10 when we had restarted
the without flushing. We see in the cassandra logs that the commitlogs were
read back without any problems. Until the restart the data was correct.
However, after the node restarted we retrieved older version of the data
(row caching is turned off). We are reading/writing to a single cassandra
node that is replicated to a single node setup at another data center. The
times are synchronized across our machines. Has anyone experienced this
type of behavior?


Thanks,

Ankit Patel

Re: Cassandra version 1.0.10 Data Loss upon restart

[Robert Coli <rc...@eventbrite.com>]

On Tue, Nov 25, 2014 at 6:40 AM, Ankit Patel <an...@gmail.com> wrote:

> The JIRA https://issues.apache.org/jira/browse/CASSANDRA-4446 refers to
> the problem after we've invoked drain. However, we did not invoke drain or
> flush. We are running one node cassandra within one data center and it is
> being replicated with another node in another data center. We are using the
> thrift API in java application to retrieve and modify the data from the
> primary node only. We are not using the other node in the other data center
> for any operations. The data loss well exceeds the commit log sync period.
>

Right, what I meant by "the era of CASSANDRA-4446" is that around that time
there were reports of weird behavior around the commit log. Not your
particular weird behavior, but similar enough that I am not *overly*
surprised to hear it. Unfortunately no one is likely to want to fix your by
now quite old version, especially as a re-write of the commit log occurred
in (IIRC) 1.1.x series.

I recommend upgrading at least to the HEAD of 1.1.x ASAP.

=Rob
PS - I understand and sympathize, the above is not a terribly satisfying
answer in the face of data loss. :/

Re: Cassandra version 1.0.10 Data Loss upon restart

[Ankit Patel <an...@gmail.com>]

Rob,

The JIRA https://issues.apache.org/jira/browse/CASSANDRA-4446 refers to the
problem after we've invoked drain. However, we did not invoke drain or
flush. We are running one node cassandra within one data center and it is
being replicated with another node in another data center. We are using the
thrift API in java application to retrieve and modify the data from the
primary node only. We are not using the other node in the other data center
for any operations. The data loss well exceeds the commit log sync period.

Thanks,
Ankit

On Mon, Nov 24, 2014 at 8:53 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Mon, Nov 24, 2014 at 5:51 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> What is your replication factor? What CL are you using to read?
>>
>
> Ah, I see from OP that RF is 1.
>
> As a general statement, RF=1 is an edge case which very, very few people
> have ever operated in production. It is relatively likely that there are
> some undiscovered edge cases which relate to it.
>
> That said, this would be a particularly glaring one, which I would expect
> to be discovered in other contexts.
>
> =Rob
>
>

Re: Cassandra version 1.0.10 Data Loss upon restart

[Robert Coli <rc...@eventbrite.com>]

On Mon, Nov 24, 2014 at 5:51 PM, Robert Coli <rc...@eventbrite.com> wrote:

> What is your replication factor? What CL are you using to read?
>

Ah, I see from OP that RF is 1.

As a general statement, RF=1 is an edge case which very, very few people
have ever operated in production. It is relatively likely that there are
some undiscovered edge cases which relate to it.

That said, this would be a particularly glaring one, which I would expect
to be discovered in other contexts.

=Rob

Re: Cassandra version 1.0.10 Data Loss upon restart

[Robert Coli <rc...@eventbrite.com>]

On Mon, Nov 24, 2014 at 3:19 PM, Ankit Patel <an...@gmail.com> wrote:

> We are experiencing data loss with Cassandra 1.0.10 when we had restarted
> the without flushing. We see in the cassandra logs that the commitlogs were
> read back without any problems. Until the restart the data was correct.
> However, after the node restarted we retrieved older version of the data
> (row caching is turned off). We are reading/writing to a single cassandra
> node that is replicated to a single node setup at another data center. The
> times are synchronized across our machines. Has anyone experienced this
> type of behavior?
>

I'm not sure I've heard of a particular issue where data is not correctly
replayed after restart.

But 1.0.10 is the era of :

https://issues.apache.org/jira/browse/CASSANDRA-4446

"nodetool drain sometimes doesn't mark commitlog fully flushed"

... and there have been other bugs relating to commitlog replay since...

What is your replication factor? What CL are you using to read?

If you look in SStables on the affected node, which versions of the row
exist?

=Rob