You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Boris Yen (JIRA)" <ji...@apache.org> on 2011/08/09 11:42:28 UTC
[jira] [Created] (CASSANDRA-3006) Enormous counter
Enormous counter
-----------------
Key: CASSANDRA-3006
URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8.3
Environment: ubuntu 10.04
Reporter: Boris Yen
I have two-node cluster with the following keyspace and column family settings.
Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
Keyspace: test:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [datacenter1:2]
Column Families:
ColumnFamily: testCounter (Super)
"APP status information."
Key Validation Class: org.apache.cassandra.db.marshal.BytesType
Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
Row cache size / save period in seconds: 0.0/0
Key cache size / save period in seconds: 200000.0/14400
Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
GC grace seconds: 864000
Compaction min/max thresholds: 4/32
Read repair chance: 1.0
Replicate on write: true
Built indexes: []
Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
I use 0.8.3 the reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3006) Enormous counter
Posted by "Boris Yen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082091#comment-13082091 ]
Boris Yen commented on CASSANDRA-3006:
--------------------------------------
Here is the test program I am using now. the hector version is 0.8.0-2.
Hope this will be helpful.
------------------------------------------------
import java.util.Arrays;
import me.prettyprint.cassandra.model.AllOneConsistencyLevelPolicy;
import me.prettyprint.cassandra.serializers.StringSerializer;
import me.prettyprint.cassandra.service.CassandraHostConfigurator;
import me.prettyprint.cassandra.service.ThriftCluster;
import me.prettyprint.hector.api.Keyspace;
import me.prettyprint.hector.api.beans.HCounterColumn;
import me.prettyprint.hector.api.factory.HFactory;
import me.prettyprint.hector.api.mutation.Mutator;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
public class CounterTest {
private Logger logger = LoggerFactory.getLogger(CounterTest.class) ;
private static final Integer COUNTER_NUM = 1000 ;
private static final StringSerializer ss = StringSerializer.get();
private static final String HOST = "172.17.19.151:9160" ;
private ThriftCluster cluster ;
/**
* @param args
*/
public static void main(String[] args) {
CounterTest tc = new CounterTest() ;
try {
tc.testAlarmCounter() ;
} catch (InterruptedException e) {
}
}
public CounterTest(){
CassandraHostConfigurator chc = new CassandraHostConfigurator(HOST) ;
chc.setMaxActive(100) ;
chc.setMaxIdle(10) ;
chc.setCassandraThriftSocketTimeout(60000) ;
cluster = new ThriftCluster("Test Cluster", chc) ;
}
public void testAlarmCounter() throws InterruptedException{
int successCounter = 0 ;
int cl = 0;
for(int i=0; i<COUNTER_NUM; i++){
try{
logger.info("count: "+i) ;
Mutator<String> mutator = HFactory.createMutator(getKeyspace(cl), StringSerializer.get());
HCounterColumn<String> column = HFactory.createCounterColumn("testSC", 1L) ;
mutator.addCounter("sc", "testCounter", HFactory.createCounterSuperColumn("testC", Arrays.asList(column), ss, ss));
mutator.execute() ;
successCounter++ ;
} catch(Exception e){
logger.info("Error! Change consistency level to 1.", e) ;
cl=1 ;
}
Thread.sleep(50) ;
}
logger.info("\nsuccess counter: "+successCounter) ;
}
private Keyspace getKeyspace(int cl){
if(cl == 1)
return HFactory.createKeyspace("test", cluster, new AllOneConsistencyLevelPolicy()) ;
else
return HFactory.createKeyspace("test", cluster) ; // default consistency level is Quorum
}
}
> Enormous counter
> -----------------
>
> Key: CASSANDRA-3006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.3
> Environment: ubuntu 10.04
> Reporter: Boris Yen
> Assignee: Sylvain Lebresne
>
> I have two-node cluster with the following keyspace and column family settings.
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
> Keyspace: test:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [datacenter1:2]
> Column Families:
> ColumnFamily: testCounter (Super)
> "APP status information."
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: []
> Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
> After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
> I use 0.8.3 to reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3006) Enormous counter
Posted by "Boris Yen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081545#comment-13081545 ]
Boris Yen commented on CASSANDRA-3006:
--------------------------------------
I forgot the mention that the counter is out of sync between these two nodes, one shows 481387 and the other one shows 20706.
> Enormous counter
> -----------------
>
> Key: CASSANDRA-3006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.3
> Environment: ubuntu 10.04
> Reporter: Boris Yen
>
> I have two-node cluster with the following keyspace and column family settings.
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
> Keyspace: test:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [datacenter1:2]
> Column Families:
> ColumnFamily: testCounter (Super)
> "APP status information."
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: []
> Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
> After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
> I use 0.8.3 the reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3006) Enormous counter
Posted by "Boris Yen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082122#comment-13082122 ]
Boris Yen commented on CASSANDRA-3006:
--------------------------------------
In order to make it easier to reproduce this issue, I document how I recreate this issue step by step.
1. clean any thing that is inside /var/lib/cassandra on node 172.17.19.151
2. start cassandra on node 172.17.19.151.
3. clean any thing that is inside /var/lib/cassnadra on node 172.17.19.152
4. modify the cassandra.yaml of 172.17.19.152 and add 172.17.19.151 as a seed.
5. start cassandra on node 172.17.19.152, I could see two node has formed a cluster, I also double check that using nodetool.
6. on node 172.17.19.151, I use cassandra-cli: to connect 172.17.19.151/9160, and execute commands ->
create keyspace test
with placement_strategy = 'org.apache.cassandra.locator.NetworkTopologyStrategy'
and strategy_options = [{datacenter1:2}];
create column family testCounter
with column_type = Super
and default_validation_class = CounterColumnType
and replicate_on_write = true
and comparator = BytesType
and subcomparator = BytesType
and comment = 'APP status information.';
7. use the test program to add the counter 1000 times. between each adding action the program will pause 50 millisecond.
8. in the middle of the adding process, shut down the cassandra on node 172.17.19.152, (let's say I shut down node 172.17.19.152 when count is 200.). Because the test program changes the consistency level to One when it encounters an exception (timeout exception to be exact), the following adding actions will still be success.
9. wait for the overall adding process to complete. I saw "success counter: 999" due to one exception.
10. use the cassandra-cli to connect to 172.17.19.151 and 172.17.19.152 and check the counter value, the value is 1001 on both nodes. It shows 1001 because hector will retry when it encounters the timeout exception.
11. shutdown the cassandra on 172.17.19.151, wait for a few seconds, I saw "InetAddress /172.17.19.151 is now dead" on node 172.17.19.152.
12. after seeing "InetAddress /172.17.19.151 is now dead", restart the cassandra on node 172.17.19.151.
13. check the counter again with cassandra-cli on both nodes, this time the counter should no longer be 1001, it should be other weird number.
Hope someone else could recreate it by these steps.
> Enormous counter
> -----------------
>
> Key: CASSANDRA-3006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.3
> Environment: ubuntu 10.04
> Reporter: Boris Yen
> Assignee: Sylvain Lebresne
>
> I have two-node cluster with the following keyspace and column family settings.
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
> Keyspace: test:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [datacenter1:2]
> Column Families:
> ColumnFamily: testCounter (Super)
> "APP status information."
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: []
> Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
> After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
> I use 0.8.3 to reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3006) Enormous counter
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13081660#comment-13081660 ]
Sylvain Lebresne commented on CASSANDRA-3006:
---------------------------------------------
I've haven't had luck with reproducing so far. I've tried to stick with the description above but did not used hector (not saying it is hector fault though, maybe it is the way it does retry that I don't emulate well). If you are able to share a minimal hector script with which you reproduce this easily, that would be very helpful.
> Enormous counter
> -----------------
>
> Key: CASSANDRA-3006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.3
> Environment: ubuntu 10.04
> Reporter: Boris Yen
> Assignee: Sylvain Lebresne
>
> I have two-node cluster with the following keyspace and column family settings.
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
> Keyspace: test:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [datacenter1:2]
> Column Families:
> ColumnFamily: testCounter (Super)
> "APP status information."
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: []
> Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
> After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
> I use 0.8.3 to reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3006) Enormous counter
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082439#comment-13082439 ]
Hudson commented on CASSANDRA-3006:
-----------------------------------
Integrated in Cassandra-0.8 #267 (See [https://builds.apache.org/job/Cassandra-0.8/267/])
Force deserialization of RowMutation for counters
patch by slebresne; reviewed by jbellis for CASSANDRA-3006
slebresne : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1156221
Files :
* /cassandra/branches/cassandra-0.8/src/java/org/apache/cassandra/db/RowMutation.java
* /cassandra/branches/cassandra-0.8/CHANGES.txt
> Enormous counter
> -----------------
>
> Key: CASSANDRA-3006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.3
> Environment: ubuntu 10.04
> Reporter: Boris Yen
> Assignee: Sylvain Lebresne
> Fix For: 0.8.4
>
> Attachments: 3006.patch
>
>
> I have two-node cluster with the following keyspace and column family settings.
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
> Keyspace: test:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [datacenter1:2]
> Column Families:
> ColumnFamily: testCounter (Super)
> "APP status information."
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: []
> Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
> After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
> I use 0.8.3 to reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3006) Enormous counter
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3006:
----------------------------------------
Attachment: 3006.patch
Thanks, that helps a lot.
The problem is due to the RowMutation optimization that keeps the serialized data received on the wire to use in the commit log. This is wrong for counters because we use the deserialization of the RowMutation to clean the delta on the counter columns.
At least in the script to reproduce, this was a hint problem. The first node was creating a hint for the second one and was storing it himself (and the value of this hints was not cleared because of the bug above).
Patch attached to fix this. This basically disable the optimization for RowMutation that contains counter. I don't pretend this is particularly clean but I don't see any other simple solution.
> Enormous counter
> -----------------
>
> Key: CASSANDRA-3006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.3
> Environment: ubuntu 10.04
> Reporter: Boris Yen
> Assignee: Sylvain Lebresne
> Attachments: 3006.patch
>
>
> I have two-node cluster with the following keyspace and column family settings.
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
> Keyspace: test:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [datacenter1:2]
> Column Families:
> ColumnFamily: testCounter (Super)
> "APP status information."
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: []
> Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
> After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
> I use 0.8.3 to reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3006) Enormous counter
Posted by "Boris Yen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Boris Yen updated CASSANDRA-3006:
---------------------------------
Description:
I have two-node cluster with the following keyspace and column family settings.
Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
Keyspace: test:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [datacenter1:2]
Column Families:
ColumnFamily: testCounter (Super)
"APP status information."
Key Validation Class: org.apache.cassandra.db.marshal.BytesType
Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
Row cache size / save period in seconds: 0.0/0
Key cache size / save period in seconds: 200000.0/14400
Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
GC grace seconds: 864000
Compaction min/max thresholds: 4/32
Read repair chance: 1.0
Replicate on write: true
Built indexes: []
Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
I use 0.8.3 to reproduce this bug, but I think this also happens on 0.8.2 or before also.
was:
I have two-node cluster with the following keyspace and column family settings.
Cluster Information:
Snitch: org.apache.cassandra.locator.SimpleSnitch
Partitioner: org.apache.cassandra.dht.RandomPartitioner
Schema versions:
63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
Keyspace: test:
Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
Durable Writes: true
Options: [datacenter1:2]
Column Families:
ColumnFamily: testCounter (Super)
"APP status information."
Key Validation Class: org.apache.cassandra.db.marshal.BytesType
Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
Row cache size / save period in seconds: 0.0/0
Key cache size / save period in seconds: 200000.0/14400
Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
GC grace seconds: 864000
Compaction min/max thresholds: 4/32
Read repair chance: 1.0
Replicate on write: true
Built indexes: []
Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
I use 0.8.3 the reproduce this bug, but I think this also happens on 0.8.2 or before also.
> Enormous counter
> -----------------
>
> Key: CASSANDRA-3006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.3
> Environment: ubuntu 10.04
> Reporter: Boris Yen
>
> I have two-node cluster with the following keyspace and column family settings.
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
> Keyspace: test:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [datacenter1:2]
> Column Families:
> ColumnFamily: testCounter (Super)
> "APP status information."
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: []
> Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
> After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
> I use 0.8.3 to reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3006) Enormous counter
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082385#comment-13082385 ]
Jonathan Ellis commented on CASSANDRA-3006:
-------------------------------------------
+1
(nit: we can break from the loop once we find a counter)
> Enormous counter
> -----------------
>
> Key: CASSANDRA-3006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.3
> Environment: ubuntu 10.04
> Reporter: Boris Yen
> Assignee: Sylvain Lebresne
> Fix For: 0.8.4
>
> Attachments: 3006.patch
>
>
> I have two-node cluster with the following keyspace and column family settings.
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
> Keyspace: test:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [datacenter1:2]
> Column Families:
> ColumnFamily: testCounter (Super)
> "APP status information."
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: []
> Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
> After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
> I use 0.8.3 to reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3006) Enormous counter
Posted by "Boris Yen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13083978#comment-13083978 ]
Boris Yen commented on CASSANDRA-3006:
--------------------------------------
Just tried 0.8.4. It seems much better now, but I still got value that is not what I expected.
Follow the same steps above, after step 11, I check the counter on .152, the counter values changes from 1001 to 200. And then I follow the steps 12 and 13. The counter value of .151 is still 1001, but the counter value of .152 is still 200. After changing the consistencylevel to quorum and do the read again, this time the counter value of .152 became 1001.
I am kind of confused, because at step 10, I can get 1001 on both nodes, how come the value of .152 changes to 200 when .151 is down. Is this the right behavior of current design?
> Enormous counter
> -----------------
>
> Key: CASSANDRA-3006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.3
> Environment: ubuntu 10.04
> Reporter: Boris Yen
> Assignee: Sylvain Lebresne
> Fix For: 0.8.4
>
> Attachments: 3006.patch
>
>
> I have two-node cluster with the following keyspace and column family settings.
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
> Keyspace: test:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [datacenter1:2]
> Column Families:
> ColumnFamily: testCounter (Super)
> "APP status information."
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: []
> Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
> After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
> I use 0.8.3 to reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3006) Enormous counter
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-3006:
-----------------------------------------
Assignee: Sylvain Lebresne
> Enormous counter
> -----------------
>
> Key: CASSANDRA-3006
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3006
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.3
> Environment: ubuntu 10.04
> Reporter: Boris Yen
> Assignee: Sylvain Lebresne
>
> I have two-node cluster with the following keyspace and column family settings.
> Cluster Information:
> Snitch: org.apache.cassandra.locator.SimpleSnitch
> Partitioner: org.apache.cassandra.dht.RandomPartitioner
> Schema versions:
> 63fda700-c243-11e0-0000-2d03dcafebdf: [172.17.19.151, 172.17.19.152]
> Keyspace: test:
> Replication Strategy: org.apache.cassandra.locator.NetworkTopologyStrategy
> Durable Writes: true
> Options: [datacenter1:2]
> Column Families:
> ColumnFamily: testCounter (Super)
> "APP status information."
> Key Validation Class: org.apache.cassandra.db.marshal.BytesType
> Default column value validator: org.apache.cassandra.db.marshal.CounterColumnType
> Columns sorted by: org.apache.cassandra.db.marshal.BytesType/org.apache.cassandra.db.marshal.BytesType
> Row cache size / save period in seconds: 0.0/0
> Key cache size / save period in seconds: 200000.0/14400
> Memtable thresholds: 1.1578125/1440/247 (millions of ops/MB/minutes)
> GC grace seconds: 864000
> Compaction min/max thresholds: 4/32
> Read repair chance: 1.0
> Replicate on write: true
> Built indexes: []
> Then, I use a test program based on hector to add a counter column (testCounter[sc][column]) 1000 times. In the middle the adding process, I intentional shut down the node 172.17.19.152. In addition to that, the test program is smart enough to switch the consistency level from Quorum to One, so that the following adding actions would not fail.
> After all the adding actions are done, I start the cassandra on 172.17.19.152, and I use cassandra-cli to check if the counter is correct on both nodes, and I got a result 1001 which should be reasonable because hector will retry once. However, when I shut down 172.17.19.151 and after 172.17.19.152 is aware of 172.17.19.151 is down, I try to start the cassandra on 172.17.19.151 again. Then, I check the counter again, this time I got a result 481387 which is so wrong.
> I use 0.8.3 to reproduce this bug, but I think this also happens on 0.8.2 or before also.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira