You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "nk11 (JIRA)" <ji...@apache.org> on 2009/05/13 14:50:45 UTC
[jira] Created: (CASSANDRA-168) thread problem?
thread problem?
---------------
Key: CASSANDRA-168
URL: https://issues.apache.org/jira/browse/CASSANDRA-168
Project: Cassandra
Issue Type: Bug
Reporter: nk11
With the original code CASSANDRA-153 works fine, even for large values of max.
I change a little bit the test.
int max = 10000;
Random rnd = new Random();
for (int a = 0; a < max; a++) {
System.out.println(a);
client.insert("Table1", "k1:" + rnd.nextInt(), "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
}
client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
Now the log says just:
ERROR [pool-1-thread-1] 2009-05-13 15:41:01,908 Cassandra.java (line 1187) Internal error processing get_key_range
java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:527)
at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.TimeoutException: Operation timed out.
at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
... 7 more
But also some of:
DEBUG [ROW-MUTATION-STAGE:1] 2009-05-13 15:40:56,970 RowMutationVerbHandler.java (line 70) Applying RowMutation(key='k1:-1907502342', modifications=[ColumnFamily(Super1 [SuperColumn(x [x:false:1@0])])])
ERROR [ROW-READ-STAGE:1] 2009-05-13 15:40:56,955 DebuggableThreadPoolExecutor.java (line 125) Error in ThreadPoolExecutor
java.lang.RuntimeException: java.util.ConcurrentModificationException
at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:27)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.ConcurrentModificationException
at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
at java.util.HashMap$KeyIterator.next(Unknown Source)
at java.util.AbstractQueue.addAll(Unknown Source)
at org.apache.cassandra.db.Memtable.sortedKeyIterator(Memtable.java:424)
at org.apache.cassandra.db.Table.getKeyRangeUnsafe(Table.java:912)
at org.apache.cassandra.db.Table.getKeyRange(Table.java:883)
at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
... 4 more
If a put a Thread.sleep() after the inserts or run only the get_key_range call all is ok.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-168) thread problem?
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-168.
--------------------------------------
Resolution: Duplicate
fixed in CASSANDRA-161. you need to update to the latest trunk.
> thread problem?
> ---------------
>
> Key: CASSANDRA-168
> URL: https://issues.apache.org/jira/browse/CASSANDRA-168
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
>
> With the original code CASSANDRA-153 works fine, even for large values of max.
> I change a little bit the test.
> int max = 10000;
> Random rnd = new Random();
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + rnd.nextInt(), "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Now the log says just:
> ERROR [pool-1-thread-1] 2009-05-13 15:41:01,908 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:527)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
> But also some of:
> DEBUG [ROW-MUTATION-STAGE:1] 2009-05-13 15:40:56,970 RowMutationVerbHandler.java (line 70) Applying RowMutation(key='k1:-1907502342', modifications=[ColumnFamily(Super1 [SuperColumn(x [x:false:1@0])])])
> ERROR [ROW-READ-STAGE:1] 2009-05-13 15:40:56,955 DebuggableThreadPoolExecutor.java (line 125) Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.util.ConcurrentModificationException
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:27)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
> at java.util.HashMap$KeyIterator.next(Unknown Source)
> at java.util.AbstractQueue.addAll(Unknown Source)
> at org.apache.cassandra.db.Memtable.sortedKeyIterator(Memtable.java:424)
> at org.apache.cassandra.db.Table.getKeyRangeUnsafe(Table.java:912)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:883)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> ... 4 more
> If a put a Thread.sleep() after the inserts or run only the get_key_range call all is ok.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-168) thread problem?
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12709187#action_12709187 ]
Jonathan Ellis commented on CASSANDRA-168:
------------------------------------------
fixed better (patch posted to CASSANDRA-161)
> thread problem?
> ---------------
>
> Key: CASSANDRA-168
> URL: https://issues.apache.org/jira/browse/CASSANDRA-168
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
>
> With the original code CASSANDRA-153 works fine, even for large values of max.
> I change a little bit the test.
> int max = 10000;
> Random rnd = new Random();
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + rnd.nextInt(), "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Now the log says just:
> ERROR [pool-1-thread-1] 2009-05-13 15:41:01,908 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:527)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
> But also some of:
> DEBUG [ROW-MUTATION-STAGE:1] 2009-05-13 15:40:56,970 RowMutationVerbHandler.java (line 70) Applying RowMutation(key='k1:-1907502342', modifications=[ColumnFamily(Super1 [SuperColumn(x [x:false:1@0])])])
> ERROR [ROW-READ-STAGE:1] 2009-05-13 15:40:56,955 DebuggableThreadPoolExecutor.java (line 125) Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.util.ConcurrentModificationException
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:27)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
> at java.util.HashMap$KeyIterator.next(Unknown Source)
> at java.util.AbstractQueue.addAll(Unknown Source)
> at org.apache.cassandra.db.Memtable.sortedKeyIterator(Memtable.java:424)
> at org.apache.cassandra.db.Table.getKeyRangeUnsafe(Table.java:912)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:883)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> ... 4 more
> If a put a Thread.sleep() after the inserts or run only the get_key_range call all is ok.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-168) thread problem?
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708985#action_12708985 ]
Jonathan Ellis commented on CASSANDRA-168:
------------------------------------------
you are right, there is still a bug here.
> thread problem?
> ---------------
>
> Key: CASSANDRA-168
> URL: https://issues.apache.org/jira/browse/CASSANDRA-168
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
>
> With the original code CASSANDRA-153 works fine, even for large values of max.
> I change a little bit the test.
> int max = 10000;
> Random rnd = new Random();
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + rnd.nextInt(), "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Now the log says just:
> ERROR [pool-1-thread-1] 2009-05-13 15:41:01,908 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:527)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
> But also some of:
> DEBUG [ROW-MUTATION-STAGE:1] 2009-05-13 15:40:56,970 RowMutationVerbHandler.java (line 70) Applying RowMutation(key='k1:-1907502342', modifications=[ColumnFamily(Super1 [SuperColumn(x [x:false:1@0])])])
> ERROR [ROW-READ-STAGE:1] 2009-05-13 15:40:56,955 DebuggableThreadPoolExecutor.java (line 125) Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.util.ConcurrentModificationException
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:27)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
> at java.util.HashMap$KeyIterator.next(Unknown Source)
> at java.util.AbstractQueue.addAll(Unknown Source)
> at org.apache.cassandra.db.Memtable.sortedKeyIterator(Memtable.java:424)
> at org.apache.cassandra.db.Table.getKeyRangeUnsafe(Table.java:912)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:883)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> ... 4 more
> If a put a Thread.sleep() after the inserts or run only the get_key_range call all is ok.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-168) thread problem?
Posted by "nk11 (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708983#action_12708983 ]
nk11 commented on CASSANDRA-168:
--------------------------------
I always check out before submitting a bug.
And I forgot to specify, I tested with only one node.
But, I've tested on a slower machine back home and it did not reproduce... On another faster machine it reproduced each time!
> thread problem?
> ---------------
>
> Key: CASSANDRA-168
> URL: https://issues.apache.org/jira/browse/CASSANDRA-168
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
>
> With the original code CASSANDRA-153 works fine, even for large values of max.
> I change a little bit the test.
> int max = 10000;
> Random rnd = new Random();
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + rnd.nextInt(), "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Now the log says just:
> ERROR [pool-1-thread-1] 2009-05-13 15:41:01,908 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:527)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
> But also some of:
> DEBUG [ROW-MUTATION-STAGE:1] 2009-05-13 15:40:56,970 RowMutationVerbHandler.java (line 70) Applying RowMutation(key='k1:-1907502342', modifications=[ColumnFamily(Super1 [SuperColumn(x [x:false:1@0])])])
> ERROR [ROW-READ-STAGE:1] 2009-05-13 15:40:56,955 DebuggableThreadPoolExecutor.java (line 125) Error in ThreadPoolExecutor
> java.lang.RuntimeException: java.util.ConcurrentModificationException
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:27)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextEntry(Unknown Source)
> at java.util.HashMap$KeyIterator.next(Unknown Source)
> at java.util.AbstractQueue.addAll(Unknown Source)
> at org.apache.cassandra.db.Memtable.sortedKeyIterator(Memtable.java:424)
> at org.apache.cassandra.db.Table.getKeyRangeUnsafe(Table.java:912)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:883)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> ... 4 more
> If a put a Thread.sleep() after the inserts or run only the get_key_range call all is ok.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.