You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "nk11 (JIRA)" <ji...@apache.org> on 2009/05/07 22:20:45 UTC
[jira] Created: (CASSANDRA-153) get_key_range timeout and exception
get_key_range timeout and exception
-----------------------------------
Key: CASSANDRA-153
URL: https://issues.apache.org/jira/browse/CASSANDRA-153
Project: Cassandra
Issue Type: Bug
Reporter: nk11
My test code:
int max = 5000;
for (int a = 0; a < max; a++) {
System.out.println(a);
client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
}
client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
Produces in the logs:
ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
java.lang.RuntimeException: corrupt sstable
at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
at java.io.DataInputStream.readUTF(Unknown Source)
at java.io.RandomAccessFile.readUTF(Unknown Source)
at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
... 6 more
ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.TimeoutException: Operation timed out.
at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708856#action_12708856 ]
Hudson commented on CASSANDRA-153:
----------------------------------
Integrated in Cassandra #73 (See [http://hudson.zones.apache.org/hudson/job/Cassandra/73/])
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Assignee: Jonathan Ellis
> Attachments: 0001-CASSANDRA-153-unit-test-to-expose-bug-system-test-is-r.txt, 0002-cannonicalize-all-accesses-to-indexMeatdataMap.txt, 0003-check-for-at-end-of-data-in-iterator-init.txt, 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708228#action_12708228 ]
Jonathan Ellis commented on CASSANDRA-153:
------------------------------------------
no, because then we will always skip the current key, which in the Range context is always going to be a key we are interested in (since seekTo skips all the keys we do not want, stopping when it comes to the first interesting one).
saved = key
in the constructor is what allows next() to return that current key.
but, if seekTo actually skipped _everything_ (and current key is block index) then we don't want to include that. that's what 0003 fixes.
(I checked in case my intuition was wrong and the test_range_collation system test fails with your alternative, so they are indeed not equivalent.)
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 0001-CASSANDRA-153-unit-test-to-expose-bug-system-test-is-r.txt, 0002-cannonicalize-all-accesses-to-indexMeatdataMap.txt, 0003-check-for-at-end-of-data-in-iterator-init.txt, 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "nk11 (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707160#action_12707160 ]
nk11 commented on CASSANDRA-153:
--------------------------------
patched, rebuilded, removed var/cassandra, still failing with the same exception
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708052#action_12708052 ]
Jonathan Ellis commented on CASSANDRA-153:
------------------------------------------
fixed two more bugs. the first is what you were running into; SequenceFile and FileStruct both assume when seeking that the block index exists at the end of the Coordinate computed by SSTable. But we weren't always consistent in specifying the index filename and instead of raising an error Coordinate would just return something bogus. these are both fixed in 02.
03 fixes a bug in FS iteration when no keys in the range specified exist in a given SSTable.
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 0001-CASSANDRA-153-unit-test-to-expose-bug-system-test-is-r.txt, 0002-cannonicalize-all-accesses-to-indexMeatdataMap.txt, 0003-check-for-at-end-of-data-in-iterator-init.txt, 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-153) get_key_range timeout and exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-153:
-------------------------------------
Attachment: 153.patch
try it with this patch. if that doesn't work, rm -r /var/cassandra/* to reset things and try again; it's possible that you have corrupt data from a bug that is fixed now.
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "nk11 (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707528#action_12707528 ]
nk11 commented on CASSANDRA-153:
--------------------------------
I spoke to soon. For 20000 keys it reproduces again...
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "nk11 (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707082#action_12707082 ]
nk11 commented on CASSANDRA-153:
--------------------------------
1 node
Default config except:
<MemtableSizeInMB>1</MemtableSizeInMB>
<MemtableObjectCountInMillions>0.001</MemtableObjectCountInMillions>
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-153) get_key_range timeout and exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-153:
-------------------------------------
Attachment: 0003-check-for-at-end-of-data-in-iterator-init.txt
0002-cannonicalize-all-accesses-to-indexMeatdataMap.txt
0001-CASSANDRA-153-unit-test-to-expose-bug-system-test-is-r.txt
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 0001-CASSANDRA-153-unit-test-to-expose-bug-system-test-is-r.txt, 0002-cannonicalize-all-accesses-to-indexMeatdataMap.txt, 0003-check-for-at-end-of-data-in-iterator-init.txt, 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707595#action_12707595 ]
Jonathan Ellis commented on CASSANDRA-153:
------------------------------------------
committed second patch. still working on reading-past-EOF bug. I think it's compaction-related.
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "nk11 (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707429#action_12707429 ]
nk11 commented on CASSANDRA-153:
--------------------------------
it works
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "nk11 (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707254#action_12707254 ]
nk11 commented on CASSANDRA-153:
--------------------------------
If it helps:
The exception is thrown in SequenceFile.java at line 562 in getPositionFromBlockIndex() method.
String blockIndexKey = file_.readUTF();
The key is "k1:0" and blockIndexPosition=102016
If I do a file_.read() there I got -1 so the end of the file is reached.
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-153) get_key_range timeout and exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-153:
-------------------------------------
Attachment: 153-02.patch
fix for CME during getKeyRange
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708252#action_12708252 ]
Jun Rao commented on CASSANDRA-153:
-----------------------------------
Ok, I understood this now. You are right and the patch looks fine.
What confused me is that the iterator over FileStruct relies on a seekTo call first, which is not how a typical iterator works.
I got a CME execption on testCompactions. I guess that's related to another issue you just opened.
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 0001-CASSANDRA-153-unit-test-to-expose-bug-system-test-is-r.txt, 0002-cannonicalize-all-accesses-to-indexMeatdataMap.txt, 0003-check-for-at-end-of-data-in-iterator-init.txt, 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-153.
--------------------------------------
Resolution: Fixed
Assignee: Jonathan Ellis
committed
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Assignee: Jonathan Ellis
> Attachments: 0001-CASSANDRA-153-unit-test-to-expose-bug-system-test-is-r.txt, 0002-cannonicalize-all-accesses-to-indexMeatdataMap.txt, 0003-check-for-at-end-of-data-in-iterator-init.txt, 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708215#action_12708215 ]
Jun Rao commented on CASSANDRA-153:
-----------------------------------
The fix in patch 0003 seems redundant. The same code is already called in advance(). The correct fix seems to be getting rid of line
saved = key;
in FileStructIterator().
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 0001-CASSANDRA-153-unit-test-to-expose-bug-system-test-is-r.txt, 0002-cannonicalize-all-accesses-to-indexMeatdataMap.txt, 0003-check-for-at-end-of-data-in-iterator-init.txt, 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-153) get_key_range timeout and
exception
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12707398#action_12707398 ]
Jonathan Ellis commented on CASSANDRA-153:
------------------------------------------
committed first patch.
i can reproduce the exception now using a different test after one more patch to fix another bug first (attached).
> get_key_range timeout and exception
> -----------------------------------
>
> Key: CASSANDRA-153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-153
> Project: Cassandra
> Issue Type: Bug
> Reporter: nk11
> Attachments: 153-02.patch, 153.patch
>
>
> My test code:
> int max = 5000;
> for (int a = 0; a < max; a++) {
> System.out.println(a);
> client.insert("Table1", "k1:" + a, "Super1:x:x", new byte[] { (byte) 1 }, 0, false);
> }
> client.get_key_range("Table1", "k1:0", "k1:1000", 1000);
> Produces in the logs:
> ERROR [ROW-READ-STAGE:9] 2009-05-07 23:04:56,609 CassandraDaemon.java (line 61) Fatal exception in thread Thread[ROW-READ-STAGE:9,5,main]
> java.lang.RuntimeException: corrupt sstable
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:107)
> at org.apache.cassandra.db.Table.getKeyRange(Table.java:905)
> at org.apache.cassandra.service.RangeVerbHandler.doVerb(RangeVerbHandler.java:23)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:46)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.io.EOFException
> at java.io.RandomAccessFile.readUnsignedShort(Unknown Source)
> at java.io.DataInputStream.readUTF(Unknown Source)
> at java.io.RandomAccessFile.readUTF(Unknown Source)
> at org.apache.cassandra.io.SequenceFile$AbstractReader.getPositionFromBlockIndex(SequenceFile.java:562)
> at org.apache.cassandra.db.FileStruct.seekTo(FileStruct.java:86)
> ... 6 more
> ERROR [pool-1-thread-2] 2009-05-07 23:05:01,593 Cassandra.java (line 1187) Internal error processing get_key_range
> java.lang.RuntimeException: error reading keyrange RangeCommand(table='Table1', startWith='k1:0', stopAt='k1:1000', maxResults=1000)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:682)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:511)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:1183)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:805)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:677)
> ... 7 more
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.