You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Jianing Hu <ji...@gmail.com> on 2010/07/29 02:47:16 UTC
cassandra 0.6.1 read returns wrong data?
We recently migrated part of our MySQL database to a 3-node Cassandra
cluster with a replication factor of 3. Couple of days ago we noticed
that Cassandra sometimes returns the wrong data. Not corrupted data,
but data for a different key than the one being asked for. This error
appears to be random and intermittent, and happens for maybe every 10K
reads. I'm working on a test suite that can reproduce this. Meanwhile
just wanted to ask if anyone has seen this problem before? I've tried
consistency levels of both QUORUM and ONE, and see the issue with
both.
Thanks,
- Jianing
Re: cassandra 0.6.1 read returns wrong data?
Posted by Jianing Hu <ji...@gmail.com>.
The error I'm seeing seems to be random: if I try to get the data
again I usually get the correct data. Although maybe compaction
happened between when the error occurred and when I checked again and
the bad key was fixed? I'll try upgrading to 0.6.4 anyway and see if
it helps. Thanks for the help.
On Mon, Aug 2, 2010 at 12:29 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> Yes, it is deterministic (but compaction could change which precise
> keys are affected)
>
> On Mon, Aug 2, 2010 at 1:15 PM, Jianing Hu <ji...@gmail.com> wrote:
>> Does that bug cause *random* data read errors? Looks like it may fail
>> in a deterministic way, but I'm not familiar with the code base so
>> please correct me if I'm wrong.
>>
>> On Fri, Jul 30, 2010 at 8:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>>> This is probably a bug fixed in 0.6.2:
>>>
>>> * fix size of row in spanned index entries (CASSANDRA-1056)
>>>
>>> You should upgrade to 0.6.4 (due out this weekend).
>>>
>>> On Wed, Jul 28, 2010 at 7:47 PM, Jianing Hu <ji...@gmail.com> wrote:
>>>> We recently migrated part of our MySQL database to a 3-node Cassandra
>>>> cluster with a replication factor of 3. Couple of days ago we noticed
>>>> that Cassandra sometimes returns the wrong data. Not corrupted data,
>>>> but data for a different key than the one being asked for. This error
>>>> appears to be random and intermittent, and happens for maybe every 10K
>>>> reads. I'm working on a test suite that can reproduce this. Meanwhile
>>>> just wanted to ask if anyone has seen this problem before? I've tried
>>>> consistency levels of both QUORUM and ONE, and see the issue with
>>>> both.
>>>>
>>>> Thanks,
>>>> - Jianing
>>>>
>>>
>>>
>>>
>>> --
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder of Riptano, the source for professional Cassandra support
>>> http://riptano.com
>>>
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: cassandra 0.6.1 read returns wrong data?
Posted by Jonathan Ellis <jb...@gmail.com>.
Yes, it is deterministic (but compaction could change which precise
keys are affected)
On Mon, Aug 2, 2010 at 1:15 PM, Jianing Hu <ji...@gmail.com> wrote:
> Does that bug cause *random* data read errors? Looks like it may fail
> in a deterministic way, but I'm not familiar with the code base so
> please correct me if I'm wrong.
>
> On Fri, Jul 30, 2010 at 8:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
>> This is probably a bug fixed in 0.6.2:
>>
>> * fix size of row in spanned index entries (CASSANDRA-1056)
>>
>> You should upgrade to 0.6.4 (due out this weekend).
>>
>> On Wed, Jul 28, 2010 at 7:47 PM, Jianing Hu <ji...@gmail.com> wrote:
>>> We recently migrated part of our MySQL database to a 3-node Cassandra
>>> cluster with a replication factor of 3. Couple of days ago we noticed
>>> that Cassandra sometimes returns the wrong data. Not corrupted data,
>>> but data for a different key than the one being asked for. This error
>>> appears to be random and intermittent, and happens for maybe every 10K
>>> reads. I'm working on a test suite that can reproduce this. Meanwhile
>>> just wanted to ask if anyone has seen this problem before? I've tried
>>> consistency levels of both QUORUM and ONE, and see the issue with
>>> both.
>>>
>>> Thanks,
>>> - Jianing
>>>
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: cassandra 0.6.1 read returns wrong data?
Posted by Jianing Hu <ji...@gmail.com>.
Does that bug cause *random* data read errors? Looks like it may fail
in a deterministic way, but I'm not familiar with the code base so
please correct me if I'm wrong.
On Fri, Jul 30, 2010 at 8:49 PM, Jonathan Ellis <jb...@gmail.com> wrote:
> This is probably a bug fixed in 0.6.2:
>
> * fix size of row in spanned index entries (CASSANDRA-1056)
>
> You should upgrade to 0.6.4 (due out this weekend).
>
> On Wed, Jul 28, 2010 at 7:47 PM, Jianing Hu <ji...@gmail.com> wrote:
>> We recently migrated part of our MySQL database to a 3-node Cassandra
>> cluster with a replication factor of 3. Couple of days ago we noticed
>> that Cassandra sometimes returns the wrong data. Not corrupted data,
>> but data for a different key than the one being asked for. This error
>> appears to be random and intermittent, and happens for maybe every 10K
>> reads. I'm working on a test suite that can reproduce this. Meanwhile
>> just wanted to ask if anyone has seen this problem before? I've tried
>> consistency levels of both QUORUM and ONE, and see the issue with
>> both.
>>
>> Thanks,
>> - Jianing
>>
>
>
>
> --
> Jonathan Ellis
> Project Chair, Apache Cassandra
> co-founder of Riptano, the source for professional Cassandra support
> http://riptano.com
>
Re: cassandra 0.6.1 read returns wrong data?
Posted by Jonathan Ellis <jb...@gmail.com>.
This is probably a bug fixed in 0.6.2:
* fix size of row in spanned index entries (CASSANDRA-1056)
You should upgrade to 0.6.4 (due out this weekend).
On Wed, Jul 28, 2010 at 7:47 PM, Jianing Hu <ji...@gmail.com> wrote:
> We recently migrated part of our MySQL database to a 3-node Cassandra
> cluster with a replication factor of 3. Couple of days ago we noticed
> that Cassandra sometimes returns the wrong data. Not corrupted data,
> but data for a different key than the one being asked for. This error
> appears to be random and intermittent, and happens for maybe every 10K
> reads. I'm working on a test suite that can reproduce this. Meanwhile
> just wanted to ask if anyone has seen this problem before? I've tried
> consistency levels of both QUORUM and ONE, and see the issue with
> both.
>
> Thanks,
> - Jianing
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com
Re: cassandra 0.6.1 read returns wrong data?
Posted by Jianing Hu <ji...@gmail.com>.
That's an interesting thought. My code runs in FCGI and although the
cassandra connection is used to serve multiple requests, those
requests are supposedly processed sequentially, in a while
($request->Accept() >= 0) loop. However, we do call FCGI::finish to
close the request (so the HTTP request would finish) while continue
writing to cassandra. I wonder if that may cause the next request to
be accepted and trump the current one? I don't think it should but
it's worth testing.
Thanks,
- Jianing
On Thu, Jul 29, 2010 at 6:23 PM, Aaron Morton <aa...@thelastpickle.com> wrote:
> I was accidentally sharing connections between threads, and getting strange
> results. Is your client multi threaded?
>
> Can you provide some more information, such as the client library, how the
> data is written and how you're deciding that the returned results are the
> wrong ones.
>
> Is the read inconsistency against data that is frequently changing? Could it
> be a problem with the way the data is being stored?
>
> Aaron
>
>
>
>
> On 30 Jul, 2010,at 11:43 AM, Jianing Hu <ji...@gmail.com> wrote:
>
> Hi Aaron,
>
> Thanks for the reply. Can you explain what you mean by "sharing
> connections around"?
>
> I'm just calling a simple "get", and the data returned is for a
> completely different key. It's intermittent and hard to produce in my
> test environment, but can be observed in our production environment
> couple hundred times a day.
>
> Thanks,
> - Jianing
>
> On Thu, Jul 29, 2010 at 2:58 PM, Aaron Morton <aa...@thelastpickle.com>
> wrote:
>> I noticed this once when accidentally sharing connections around. Could
>> that
>> be the case ?
>>
>> What sort of commands are you running ? Could you be seeing this problem ?
>> http://www.mail-archive.com/user@cassandra.apache.org/msg04831.html
>>
>> Aaron
>>
>>
>> On 29 Jul, 2010,at 12:47 PM, Jianing Hu <ji...@gmail.com> wrote:
>>
>> We recently migrated part of our MySQL database to a 3-node Cassandra
>> cluster with a replication factor of 3. Couple of days ago we noticed
>> that Cassandra sometimes returns the wrong data. Not corrupted data,
>> but data for a different key than the one being asked for. This error
>> appears to be random and intermittent, and happens for maybe every 10K
>> reads. I'm working on a test suite that can reproduce this. Meanwhile
>> just wanted to ask if anyone has seen this problem before? I've tried
>> consistency levels of both QUORUM and ONE, and see the issue with
>> both.
>>
>> Thanks,
>> - Jianing
>>
>
Re: cassandra 0.6.1 read returns wrong data?
Posted by Aaron Morton <aa...@thelastpickle.com>.
I was accidentally sharing connections between threads, and getting strange results. Is your client multi threaded?
Can you provide some more information, such as the client library, how the data is written and how you're deciding that the returned results are the wrong ones.
Is the read inconsistency against data that is frequently changing? Could it be a problem with the way the data is being stored?
Aaron
On 30 Jul, 2010,at 11:43 AM, Jianing Hu <ji...@gmail.com> wrote:
> Hi Aaron,
>
> Thanks for the reply. Can you explain what you mean by "sharing
> connections around"?
>
> I'm just calling a simple "get", and the data returned is for a
> completely different key. It's intermittent and hard to produce in my
> test environment, but can be observed in our production environment
> couple hundred times a day.
>
> Thanks,
> - Jianing
>
> On Thu, Jul 29, 2010 at 2:58 PM, Aaron Morton <aa...@thelastpickle.com> wrote:
> > I noticed this once when accidentally sharing connections around. Could that
> > be the case ?
> >
> > What sort of commands are you running ? Could you be seeing this problem ?
> > http://www.mail-archive.com/user@cassandra.apache.org/msg04831.html
> >
> > Aaron
> >
> >
> > On 29 Jul, 2010,at 12:47 PM, Jianing Hu <ji...@gmail.com> wrote:
> >
> > We recently migrated part of our MySQL database to a 3-node Cassandra
> > cluster with a replication factor of 3. Couple of days ago we noticed
> > that Cassandra sometimes returns the wrong data. Not corrupted data,
> > but data for a different key than the one being asked for. This error
> > appears to be random and intermittent, and happens for maybe every 10K
> > reads. I'm working on a test suite that can reproduce this. Meanwhile
> > just wanted to ask if anyone has seen this problem before? I've tried
> > consistency levels of both QUORUM and ONE, and see the issue with
> > both.
> >
> > Thanks,
> > - Jianing
> >
Re: cassandra 0.6.1 read returns wrong data?
Posted by Jianing Hu <ji...@gmail.com>.
Hi Aaron,
Thanks for the reply. Can you explain what you mean by "sharing
connections around"?
I'm just calling a simple "get", and the data returned is for a
completely different key. It's intermittent and hard to produce in my
test environment, but can be observed in our production environment
couple hundred times a day.
Thanks,
- Jianing
On Thu, Jul 29, 2010 at 2:58 PM, Aaron Morton <aa...@thelastpickle.com> wrote:
> I noticed this once when accidentally sharing connections around. Could that
> be the case ?
>
> What sort of commands are you running ? Could you be seeing this problem ?
> http://www.mail-archive.com/user@cassandra.apache.org/msg04831.html
>
> Aaron
>
>
> On 29 Jul, 2010,at 12:47 PM, Jianing Hu <ji...@gmail.com> wrote:
>
> We recently migrated part of our MySQL database to a 3-node Cassandra
> cluster with a replication factor of 3. Couple of days ago we noticed
> that Cassandra sometimes returns the wrong data. Not corrupted data,
> but data for a different key than the one being asked for. This error
> appears to be random and intermittent, and happens for maybe every 10K
> reads. I'm working on a test suite that can reproduce this. Meanwhile
> just wanted to ask if anyone has seen this problem before? I've tried
> consistency levels of both QUORUM and ONE, and see the issue with
> both.
>
> Thanks,
> - Jianing
>
Re: cassandra 0.6.1 read returns wrong data?
Posted by Aaron Morton <aa...@thelastpickle.com>.
I noticed this once when accidentally sharing connections around. Could that be the case ?
What sort of commands are you running ? Could you be seeing this problem ?
http://www.mail-archive.com/user@cassandra.apacheorg/msg04831.html
Aaron
On 29 Jul, 2010,at 12:47 PM, Jianing Hu <ji...@gmail.com> wrote:
> We recently migrated part of our MySQL database to a 3-node Cassandra
> cluster with a replication factor of 3. Couple of days ago we noticed
> that Cassandra sometimes returns the wrong data. Not corrupted data,
> but data for a different key than the one being asked for. This error
> appears to be random and intermittent, and happens for maybe every 10K
> reads. I'm working on a test suite that can reproduce this. Meanwhile
> just wanted to ask if anyone has seen this problem before? I've tried
> consistency levels of both QUORUM and ONE, and see the issue with
> both.
>
> Thanks,
> - Jianing