columns disappearing intermittently

[Blake Eggleston <bl...@grapheffect.com>]

Hi All,

We're having a problem with our cassandra cluster and are at a loss as to the cause.

We have what appear to be columns that disappear for a little while, then reappear. The rest of the row is returned normally during this time. This is, of course, very disturbing, and is wreaking havoc with our application.

A bit more info about what's happening:

We are repeatedly executing the same query against our cluster. Every so often, one of the columns will disappear from the row and will remain gone for some time. Then after continually executing the same query, the column will come back. The queries are being executed against a 3 node cluster, with a replication factor of 3, and all reads and writes are done with a quorum consistency level.

We upgraded from cassandra 1.1.12 to 1.2.6 last week, but only started seeing issues this morning.

Has anyone had a problem like this before, or have any idea what might be causing it?

Re: columns disappearing intermittently

[sankalp kohli <ko...@gmail.com>]

Since you are doing QUORAM, looks like the column is not found in 2
machines. Did you switch to vnodes? Try to see which machines are
responding when you get the column and when not. Might be able to identity
the problem


On Wed, Jul 3, 2013 at 4:56 AM, Eric Stevens <mi...@gmail.com> wrote:

> I wonder if one particular node is having trouble; when you notice the
> missing column, what happens if you execute the read manually from cqlsh or
> cassandra-cli independently directly on each node?
>
>
> On Wed, Jul 3, 2013 at 2:00 AM, Blake Eggleston <bl...@grapheffect.com>wrote:
>
>> Hi All,
>>
>> We're having a problem with our cassandra cluster and are at a loss as to
>> the cause.
>>
>> We have what appear to be columns that disappear for a little while, then
>> reappear. The rest of the row is returned normally during this time. This
>> is, of course, very disturbing, and is wreaking havoc with our application.
>>
>> A bit more info about what's happening:
>>
>> We are repeatedly executing the same query against our cluster. Every so
>> often, one of the columns will disappear from the row and will remain gone
>> for some time. Then after continually executing the same query, the column
>> will come back. The queries are being executed against a 3 node cluster,
>> with a replication factor of 3, and all reads and writes are done with a
>> quorum consistency level.
>>
>> We upgraded from cassandra 1.1.12 to 1.2.6 last week, but only started
>> seeing issues this morning.
>>
>> Has anyone had a problem like this before, or have any idea what might be
>> causing it?
>
>
>

Re: columns disappearing intermittently

[Eric Stevens <mi...@gmail.com>]

I wonder if one particular node is having trouble; when you notice the
missing column, what happens if you execute the read manually from cqlsh or
cassandra-cli independently directly on each node?


On Wed, Jul 3, 2013 at 2:00 AM, Blake Eggleston <bl...@grapheffect.com>wrote:

> Hi All,
>
> We're having a problem with our cassandra cluster and are at a loss as to
> the cause.
>
> We have what appear to be columns that disappear for a little while, then
> reappear. The rest of the row is returned normally during this time. This
> is, of course, very disturbing, and is wreaking havoc with our application.
>
> A bit more info about what's happening:
>
> We are repeatedly executing the same query against our cluster. Every so
> often, one of the columns will disappear from the row and will remain gone
> for some time. Then after continually executing the same query, the column
> will come back. The queries are being executed against a 3 node cluster,
> with a replication factor of 3, and all reads and writes are done with a
> quorum consistency level.
>
> We upgraded from cassandra 1.1.12 to 1.2.6 last week, but only started
> seeing issues this morning.
>
> Has anyone had a problem like this before, or have any idea what might be
> causing it?