Problem with counter columns

[Yulian Oifa <oi...@gmail.com>]

Hello to all
i am using counter columns in cassandra cluster with 3 nodes.
all 3 nodes are up and synchronized with ntp timeserver , same with client.
I am using libthrift java client.

Current problem i am having is that part of writes to counter columns
simply disappears ( most of time different values are added so i am able to
follow what missed and what not ).Approximately 40-50% of add requests
disappears.

So for example if initial value is 1 , and i want to add 1 , 2 , 3 i would
expect final result on 7 , but it will return me the value 5 since add(2)
will be lost ( no exception is thrown ).

I did not disabled replica on write ( it is set to true ):

ColumnFamily: subscriptionsCounters
      Key Validation Class: org.apache.cassandra.db.marshal.TimeUUIDType
      Default column value validator:
org.apache.cassandra.db.marshal.CounterColumnType
      Columns sorted by: org.apache.cassandra.db.marshal.UTF8Type
      Row cache size / save period in seconds: 0.0/0
      Row Cache Provider:
org.apache.cassandra.cache.ConcurrentLinkedHashCacheProvider
      Key cache size / save period in seconds: 200000.0/14400
      Memtable thresholds: 1.1484375/1440/245 (millions of ops/minutes/MB)
      GC grace seconds: 864000
      Compaction min/max thresholds: 4/32
      Read repair chance: 1.0
      Replicate on write: true
      Built indexes: []


Also i dont see anything in log in any not , neither warning nor error.
I am writing data with Local QUORUM Consistency Level since it stopped
working with one.
same with read operations.


Current cassandra version is 0.8.10.

How can i debug , find the problem????

Thanks and best regards
Yulian Oifa

Re: Problem with counter columns

[Yulian Oifa <oi...@gmail.com>]

Hello
I have resolved the issue by recreating tables and recalculating them ( i
use counter columns for summary , so live data allows me to recalculate it
).

As for upgrading i walked over issues list and found that issues either
where resolved in 0.8.3 , 0.8.4 or opened till now ( my version is 0.8.10 )

Especially the issues :
https://issues.apache.org/jira/browse/CASSANDRA-4775

And previous one that was moved into 4775 :
https://issues.apache.org/jira/browse/CASSANDRA-2495

So my goal is to identify the problem first , prior to upgrading and
opening duplicate case , etc.
I am almost sure that what caused the error was Out of memory problems
which where caused by big super column tables.after that i had to restart a
servers one after another.
Counters got out of sync and looks like when update went to correct server
everything was working , otherwise update was discarded ( servers was not
able to resync on counter columns ).
Looks like also nodetool repair does nothing with counter columns , but
only regular columns.

Unfortunately i did not found any hint about what was wrong in counter
columns and did not saw anything in log files , which makes me only hope
that next time the issue will not be reproduced....
For now i will have to stay with current version till i will see that
described issues are resolved in cassandra.

Best regards
Yulian Oifa



On Thu, Sep 19, 2013 at 7:06 PM, Robert Coli <rc...@eventbrite.com> wrote:

> On Wed, Sep 18, 2013 at 11:07 AM, Yulian Oifa <oi...@gmail.com>wrote:
>
>> i am using counter columns in cassandra cluster with 3 nodes.
>>
>
>>
> Current cassandra version is 0.8.10.
>>
>> How can i debug , find the problem????
>>
>
> The problem is using Counters in Cassandra 0.8.
>
> But seriously, I don't know whether the particular issue you describe is
> fixed upstream. But if it isn't, no one will fix it in 0.8, so you should
> probably...
>
> 1) upgrade to Cassandra 1.2.9 (note that you likely need to pass through
> 1.0/1.1)
> 2) attempt to reproduce
> 3) if you can, file a JIRA and update this thread with a link to it
>
> =Rob
>
>

Re: Problem with counter columns

[Robert Coli <rc...@eventbrite.com>]

On Wed, Sep 18, 2013 at 11:07 AM, Yulian Oifa <oi...@gmail.com> wrote:

> i am using counter columns in cassandra cluster with 3 nodes.
>

>
Current cassandra version is 0.8.10.
>
> How can i debug , find the problem????
>

The problem is using Counters in Cassandra 0.8.

But seriously, I don't know whether the particular issue you describe is
fixed upstream. But if it isn't, no one will fix it in 0.8, so you should
probably...

1) upgrade to Cassandra 1.2.9 (note that you likely need to pass through
1.0/1.1)
2) attempt to reproduce
3) if you can, file a JIRA and update this thread with a link to it

=Rob