You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Alain RODRIGUEZ (Created) (JIRA)" <ji...@apache.org> on 2011/11/07 19:52:51 UTC
[jira] [Created] (CASSANDRA-3465) Wrong cunters values when RF > 1
Wrong cunters values when RF > 1
--------------------------------
Key: CASSANDRA-3465
URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.0.0
Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
Reporter: Alain RODRIGUEZ
Priority: Critical
I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
/*-------------------------------------------------- CODE -------------------------------------------------------*/
$servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
"ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
"ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
"ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
"ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
$pool = new ConnectionPool("mykeyspace", $servers);
$stats_test = new ColumnFamily($pool, 'stats_test',
$read_consistency_level=cassandra_ConsistencyLevel::ONE,
$write_consistency_level=cassandra_ConsistencyLevel::ONE);
$time = date( 'YmdH', time());
for($i=0; $i<10; $i++){
for($c=1; $c<=3; $c++){
$stats_test->add($c, $time.':test');
}
$counts = $stats_test->multiget(array(1,2,3));
echo('Counter1: '.$counts[1][$time.':test']."\n");
echo('Counter2: '.$counts[2][$time.':test']."\n");
echo('Counter3: '.$counts[3][$time.':test']."\n\n");
}
/*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
/*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
Counter1: 1
Counter2: 1
Counter3: 1
Counter1: 2
Counter2: 2
Counter3: 2
Counter1: 3
Counter2: 3
Counter3: 3
Counter1: 3
Counter2: 4
Counter3: 4
Counter1: 4
Counter2: 5
Counter3: 3
Counter1: 5
Counter2: 6
Counter3: 3
Counter1: 6
Counter2: 7
Counter3: 4
Counter1: 4
Counter2: 8
Counter3: 7
Counter1: 5
Counter2: 9
Counter3: 8
Counter1: 8
Counter2: 4
Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3465) Wrong counters values when RF >
1
Posted by "Alain RODRIGUEZ (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146331#comment-13146331 ]
Alain RODRIGUEZ commented on CASSANDRA-3465:
--------------------------------------------
Sorry about the unclearness of my description, don't hesitate to ask me more thing about this issue, I want to fix this asap, so I will help as much as I can.
It's the same when trying with CL.ONE, CL.QUORUM or CL.ALL. I think the issue doesn't come from the CL but from the RF, or more accurately from the way that counters replicas are managed.
This test code : (3 nodes cluster, 1 CF described below, 5 rows with identical column)
/*------------------------------ CODE ---------------------------------------*/
$servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
"ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
"ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com");
$pool = new ConnectionPool("mykeyspace", $servers);
$stats_test = new ColumnFamily($pool, 'test_counter',
$read_consistency_level=cassandra_ConsistencyLevel::QUORUM,
$write_consistency_level=cassandra_ConsistencyLevel::QUORUM);
$time = date( 'YmdH', time());
for($i=0; $i<10; $i++){
for($c=1; $c<=5; $c++){
$stats_test->add($c, $time.':test');
}
$counts = $stats_test->multiget(array(1,2,3,4,5));
echo('Counter1: '.$counts[1][$time.':test']."\n");
echo('Counter2: '.$counts[2][$time.':test']."\n");
echo('Counter3: '.$counts[3][$time.':test']."\n");
echo('Counter4: '.$counts[4][$time.':test']."\n");
echo('Counter5: '.$counts[5][$time.':test']."\n\n");
}
/*-----------------------END OF CODE ---------------------------------------*/
Gives these outputs:
With RF = 1 :
Counter1: 1
Counter2: 1
Counter3: 1
Counter4: 1
Counter5: 1
Counter1: 2
Counter2: 2
Counter3: 2
Counter4: 2
Counter5: 2
Counter1: 3
Counter2: 3
Counter3: 3
Counter4: 3
Counter5: 3
Counter1: 4
Counter2: 4
Counter3: 4
Counter4: 4
Counter5: 4
Counter1: 5
Counter2: 5
Counter3: 5
Counter4: 5
Counter5: 5
Counter1: 6
Counter2: 6
Counter3: 6
Counter4: 6
Counter5: 6
Counter1: 7
Counter2: 7
Counter3: 7
Counter4: 7
Counter5: 7
Counter1: 8
Counter2: 8
Counter3: 8
Counter4: 8
Counter5: 8
Counter1: 9
Counter2: 9
Counter3: 9
Counter4: 9
Counter5: 9
Counter1: 10
Counter2: 10
Counter3: 10
Counter4: 10
Counter5: 10
This is the expected behaviour.
With RF = 2 :
Counter1: 1
Counter2: 1
Counter3: 1
Counter4: 1
Counter5: 1
Counter1: 2
Counter2: 2
Counter3: 2
Counter4: 2
Counter5: 1
Counter1: 3
Counter2: 3
Counter3: 3
Counter4: 3
Counter5: 1
Counter1: 4
Counter2: 4
Counter3: 3
Counter4: 4
Counter5: 2
Counter1: 5
Counter2: 5
Counter3: 3
Counter4: 5
Counter5: 3
Counter1: 6
Counter2: 6
Counter3: 3
Counter4: 6
Counter5: 4
Counter1: 7
Counter2: 7
Counter3: 3
Counter4: 7
Counter5: 5
Counter1: 8
Counter2: 8
Counter3: 4
Counter4: 8
Counter5: 5
Counter1: 9
Counter2: 9
Counter3: 4
Counter4: 9
Counter5: 6
Counter1: 10
Counter2: 10
Counter3: 4
Counter4: 10
Counter5: 6
And if I continue requesting for these counters I will always have 10 on counters 1,2 and 4, but on counter 3 I will have either 4 or 6 alternatively (it is the same with counter 5)
With RF = 3 :
Counter1: 1
Counter2: 1
Counter3: 1
Counter4: 1
Counter5: 1
Counter1: 2
Counter2: 1
Counter3: 2
Counter4: 2
Counter5: 2
Counter1: 2
Counter2: 2
Counter3: 3
Counter4: 2
Counter5: 3
Counter1: 2
Counter2: 3
Counter3: 4
Counter4: 3
Counter5: 4
Counter1: 3
Counter2: 3
Counter3: 5
Counter4: 3
Counter5: 4
Counter1: 4
Counter2: 3
Counter3: 6
Counter4: 3
Counter5: 5
Counter1: 5
Counter2: 3
Counter3: 7
Counter4: 4
Counter5: 5
Counter1: 6
Counter2: 4
Counter3: 6
Counter4: 5
Counter5: 6
Counter1: 7
Counter2: 5
Counter3: 7
Counter4: 5
Counter5: 7
Counter1: 7
Counter2: 6
Counter3: 8
Counter4: 6
Counter5: 7
Requesting for these rows more times gives me some differents values :
Counter1: 7
Counter2: 6
Counter3: 8
Counter4: 6
Counter5: 5
Counter1: 7
Counter2: 9
Counter3: 8
Counter4: 6
Counter5: 5
Here is the "show schema" of my keyspace :
create keyspace mykeyspace
with placement_strategy = 'SimpleStrategy'
and strategy_options = {replication_factor : 2} <-- I changed this value for tests to 1, 2, 3
and durable_writes = true;
use mykeyspace;
create column family test_counter
with column_type = 'Standard'
and comparator = 'BytesType'
and default_validation_class = 'CounterColumnType'
and key_validation_class = 'BytesType'
and rows_cached = 0.0
and row_cache_save_period = 0
and row_cache_keys_to_save = 2147483647
and keys_cached = 200000.0
and key_cache_save_period = 14400
and read_repair_chance = 1.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and row_cache_provider = 'ConcurrentLinkedHashCacheProvider'
and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy';
I think that my cluster is well configured :
Address DC Rack Status State Load Owns Token
x datacenter1 rack1 Up Normal 59.33 MB 33.33% 0
y datacenter1 rack1 Up Normal 64.45 MB 33.33% 56713727820156407428984779325531226112
z datacenter1 rack1 Up Normal 76.69 MB 33.33% 113427455640312814857969558651062452224
I hope the problem is better explained now. If some point is still unclear, please let me know.
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3465) Wrong counters values when RF > 1
Posted by "Alain RODRIGUEZ (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alain RODRIGUEZ updated CASSANDRA-3465:
---------------------------------------
Attachment: logServer2_cl_all.log
logServer1_cl_all.log
logServer0_cl_all.log
I just attached the logs of the 3 servers with debug activated while running the test script with CL.ALL
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
> Attachments: logServer0.log, logServer0_cl_all.log, logServer1.log, logServer1_cl_all.log, logServer2.log, logServer2_cl_all.log
>
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3465) Wrong counters values when RF >
1
Posted by "Alain RODRIGUEZ (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146933#comment-13146933 ]
Alain RODRIGUEZ commented on CASSANDRA-3465:
--------------------------------------------
Hi I attached logs from my 3 servers, with CL.QUORUM, running the test only once.
I enabled debug :
# Application logging options
log4j.logger.org.apache.cassandra=DEBUG
log4j.logger.org.apache.cassandra.db=DEBUG
log4j.logger.org.apache.cassandra.service.StorageProxy=DEBUG
The logs were cleared before running the test and stopped a bit after. I have OpsCenter installed on the server 0.
I hope it will help.
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
> Attachments: logServer0.log, logServer1.log, logServer2.log
>
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3465) Wrong counters values when RF > 1
Posted by "Sylvain Lebresne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3465:
----------------------------------------
Attachment: 0001-add-debug-infos.patch
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
> Attachments: 0001-add-debug-infos.patch, logServer0.log, logServer0_cl_all.log, logServer1.log, logServer1_cl_all.log, logServer2.log, logServer2_cl_all.log
>
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3465) Wrong counters values when RF >
1
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146402#comment-13146402 ]
Sylvain Lebresne commented on CASSANDRA-3465:
---------------------------------------------
Haven't been able to reproduce yet, even with more counters and more iterations. I didn't phpcassa though (but I tried mimicking the test as best as possible), so it could be worth making sure this doesn't come from that.
Did you look at the logs of the nodes after the test to see if there was anything wrong in there. Otherwise, if you can try enabling debug logs on each of the nodes, running your test above (with CL.QUORUM or CL.ALL, *not* CL.ONE) and attach the resulting logs.
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3465) Wrong counters values when RF >
1
Posted by "Sylvain Lebresne (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne resolved CASSANDRA-3465.
-----------------------------------------
Resolution: Not A Problem
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
> Attachments: 0001-add-debug-infos.patch, logServer0.log, logServer0_cl_all.log, logServer1.log, logServer1_cl_all.log, logServer2.log, logServer2_cl_all.log
>
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3465) Wrong counters values when RF > 1
Posted by "Brandon Williams (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Williams updated CASSANDRA-3465:
----------------------------------------
Summary: Wrong counters values when RF > 1 (was: Wrong cunters values when RF > 1)
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3465) Wrong counters values when RF >
1
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147795#comment-13147795 ]
Sylvain Lebresne commented on CASSANDRA-3465:
---------------------------------------------
{quote}
But I don't know why you think it comes from the CL, because I thought the CL shoudn't change the way that data are written but just the number of node we need to request before returning the information. After performing a repair on all nodes, shouldn't I have the same results with CL.ONE, CL.QUORUM or CL.ALL ?
"And getting the results you get at CL.ONE is not surprising." --> Why not ?
{quote}
The code you've pasted is not guaranteed to show a value of 10 for each counter at the end of the test with CL.ONE. That's the way CL.ONE works. The last reads at CL.ONE will only hit one node X before returning, but if the writes were at CL.ONE, they also wait for only one node Y and so there is no guarantee that X == Y every time. So in a test like the one you've pasted where you do inserts and reads in rapid succession, you have a very good change of not getting 10 for each counters at the end. Yes, eventually all node should get 10, and yes, if you run some repair you should be guaranteed to see 10 everywhere, but I guess what I'm saying is that QUORUM (or ALL) gives the guarantee that at the end of the test all counters should be 10, so let's not bother with CL.ONE and all the 'but' that comes with it if we don't have to.
Now the thing is, I tried your exact code with phpcassa and was still not able to reproduce. Which makes it kind of hard to debug. And I looked at the logs, but there is not enough info to really help. So the best I can offer is the attached patch with added debug messages. If you are willing to apply this patch (you will likely need to use the current 1.0 branch, from svn or https://github.com/apache/cassandra), and redo the test at QUORUM or ALL with debug enabled and send the logs. There should be enough info to help to at least get a fair idea of what is wrong. But if you could stop opscenter when doing so, that would be awesome, because it generates lots of garbage in the log that makes it harder to look at.
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
> Attachments: 0001-add-debug-infos.patch, logServer0.log, logServer0_cl_all.log, logServer1.log, logServer1_cl_all.log, logServer2.log, logServer2_cl_all.log
>
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3465) Wrong counters values when RF > 1
Posted by "Alain RODRIGUEZ (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alain RODRIGUEZ updated CASSANDRA-3465:
---------------------------------------
Attachment: logServer2.log
logServer1.log
logServer0.log
Logs from each server after running the discribed test.
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
> Attachments: logServer0.log, logServer1.log, logServer2.log
>
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3465) Wrong counters values when RF >
1
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13146259#comment-13146259 ]
Sylvain Lebresne commented on CASSANDRA-3465:
---------------------------------------------
Wait, did you try with CL.QUORUM instead of CL.ONE for both inserts and reads ? You're description is unclear.
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3465) Wrong counters values when RF >
1
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148628#comment-13148628 ]
Sylvain Lebresne commented on CASSANDRA-3465:
---------------------------------------------
Glad you fixed your problem. Having 2 nodes sharing the same files in the system keyspace would likely be what caused the problem (the NodeIdInfo files in particular), don't do that.
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
> Attachments: 0001-add-debug-infos.patch, logServer0.log, logServer0_cl_all.log, logServer1.log, logServer1_cl_all.log, logServer2.log, logServer2_cl_all.log
>
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3465) Wrong counters values when RF >
1
Posted by "Alain RODRIGUEZ (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147134#comment-13147134 ]
Alain RODRIGUEZ commented on CASSANDRA-3465:
--------------------------------------------
I've made a mistake doing this little test script.
$stats_test = new ColumnFamily($pool, 'test_counter',
$read_consistency_level=cassandra_ConsistencyLevel::QUORUM,
$write_consistency_level=cassandra_ConsistencyLevel::QUORUM);
should have been:
$stats_test = new ColumnFamily($pool, 'test_counter',
true, true,cassandra_ConsistencyLevel::QUORUM, cassandra_ConsistencyLevel::QUORUM);
I'm sorry about it.
Now I see in the logs that modifications are using CL.QUORUM. I'm sure to read with CL.QUORUM too.
DEBUG [MutationStage:13] 2011-11-09 15:30:39,483 CounterMutationVerbHandler.java (line 53) Applying forwarded CounterMutation(RowMutation(keyspace='mykeyspace', key='31', modifications=[ColumnFamily(test_counter [323031313131303931353a74657374:false:8@1320852636516,])]), QUORUM)
DEBUG [MutationStage:13] 2011-11-09 15:30:39,483 StorageProxy.java (line 135) insert writing local & replicate CounterMutation(RowMutation(keyspace='mykeyspace', key='31', modifications=[test_counter]), QUORUM)
It still doesn't work properly, the behaviour reamins exactly the same as with a CL.ONE.
But I don't know why you think it comes from the CL, because I thought the CL shoudn't change the way that data are written but just the number of node we need to request before returning the information. After performing a repair on all nodes, shouldn't I have the same results with CL.ONE, CL.QUORUM or CL.ALL ?
"And getting the results you get at CL.ONE is not surprising." --> Why not ?
By the way, I have try with a CL.ALL and it doesn't work better.
DEBUG [MutationStage:8] 2011-11-09 16:02:37,180 CounterMutationVerbHandler.java (line 53) Applying forwarded CounterMutation(RowMutation(keyspace='mykeyspace', key='34', modifications=[ColumnFamily(test_counter [323031313131303931363a74657374:false:8@1320854552469,])]), ALL)
DEBUG [MutationStage:8] 2011-11-09 16:02:37,180 StorageProxy.java (line 135) insert writing local & replicate CounterMutation(RowMutation(keyspace='mykeyspace', key='34', modifications=[test_counter]), ALL)
This looks very weird to me, especially since you can not reproduce this behaviour.
If I can do more, just let me know.
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
> Attachments: logServer0.log, logServer1.log, logServer2.log
>
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3465) Wrong counters values when RF >
1
Posted by "Alain RODRIGUEZ (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148569#comment-13148569 ]
Alain RODRIGUEZ commented on CASSANDRA-3465:
--------------------------------------------
Hi, I was not able to reproduce this bug with the source built from github, so I have no logs for you. I decided to make a new cluster with the new version of Cassandra (1.0.2). I had exactly the same problem. So I thought it definitely comes from my configuration. I turned off my entire cluster, removed all the keyspaces and all the files inside the data/system directory to completly reset the cluster.
Now it works like a charm... I don't exactly know where this problem came from. If you're going to continue investigating, you should know I configured my cluster by installing & configuring one EC2 instance that I cloned twice. Before launching Cassandra on the 2 new nodes I changed there listen_address and tokens in cassandra.yaml and rm .../data/system/Location* .../commitlog/*. Maybe shouldn't I have used clones or should I have removed all from the system directory directly, I don't know.
If something in my configuration was wrong I may have been noticed of it.
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
> Attachments: 0001-add-debug-infos.patch, logServer0.log, logServer0_cl_all.log, logServer1.log, logServer1_cl_all.log, logServer2.log, logServer2_cl_all.log
>
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3465) Wrong cunters values when RF > 1
Posted by "Jonathan Ellis (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-3465:
-----------------------------------------
Assignee: Sylvain Lebresne
> Wrong cunters values when RF > 1
> --------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3465) Wrong counters values when RF >
1
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3465?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13147079#comment-13147079 ]
Sylvain Lebresne commented on CASSANDRA-3465:
---------------------------------------------
Are you sure you were using CL.QUORUM with those tests?
Because if you do, that's likely a phpcassa bug that doesn't respect the consistency level you're asking for, because there is no trace of any query at CL.QUORUM in those logs, only queries at CL.ONE. And getting the results you get at CL.ONE is not surprising.
> Wrong counters values when RF > 1
> ---------------------------------
>
> Key: CASSANDRA-3465
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3465
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.0.0
> Environment: Amazon EC2 (cluster of 5 t1.micro), phpCassa 0.8.a.2
> Reporter: Alain RODRIGUEZ
> Assignee: Sylvain Lebresne
> Priority: Critical
> Attachments: logServer0.log, logServer1.log, logServer2.log
>
>
> I have got a CF that contains many counters of some events. When I'm at RF = 1 and simulate 10 events, they are well counted.
> However, when I switch to a RF = 3, my counter show a wrong value that sometimes change when requested twice (it can return 7, then 5 instead of 10 all the time).
> I first thought that it was a problem of CL because I seem to remember that I read once that I had to use CL.One for reads and writes with counters. So I tried with CL.One, without success...
> /*-------------------------------------------------- CODE -------------------------------------------------------*/
> $servers = array("ec2-xxx-xxx-xxx-xxx.eu-west-1.compute.amazonaws.com",
> "ec2-yyy-yyy-yyy-yyy.eu-west-1.compute.amazonaws.com",
> "ec2-zzz-zzz-zzz-zzz.eu-west-1.compute.amazonaws.com",
> "ec2-aaa-aaa-aaa-aaa.eu-west-1.compute.amazonaws.com",
> "ec2-bbb-bbb-bbb-bbb.eu-west-1.compute.amazonaws.com");
> $pool = new ConnectionPool("mykeyspace", $servers);
> $stats_test = new ColumnFamily($pool, 'stats_test',
> $read_consistency_level=cassandra_ConsistencyLevel::ONE,
> $write_consistency_level=cassandra_ConsistencyLevel::ONE);
>
> $time = date( 'YmdH', time());
>
> for($i=0; $i<10; $i++){
> for($c=1; $c<=3; $c++){
> $stats_test->add($c, $time.':test');
> }
> $counts = $stats_test->multiget(array(1,2,3));
> echo('Counter1: '.$counts[1][$time.':test']."\n");
> echo('Counter2: '.$counts[2][$time.':test']."\n");
> echo('Counter3: '.$counts[3][$time.':test']."\n\n");
> }
> /*-------------------------------- END OF CODE -------------------------------------------------------------------------*/
> /*-------------------------------------------------- OUTPUT ------------------------------------------------------------*/
> Counter1: 1
> Counter2: 1
> Counter3: 1
> Counter1: 2
> Counter2: 2
> Counter3: 2
> Counter1: 3
> Counter2: 3
> Counter3: 3
> Counter1: 3
> Counter2: 4
> Counter3: 4
> Counter1: 4
> Counter2: 5
> Counter3: 3
> Counter1: 5
> Counter2: 6
> Counter3: 3
> Counter1: 6
> Counter2: 7
> Counter3: 4
> Counter1: 4
> Counter2: 8
> Counter3: 7
> Counter1: 5
> Counter2: 9
> Counter3: 8
> Counter1: 8
> Counter2: 4
> Counter3: 9
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira