You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "daishi (JIRA)" <ji...@apache.org> on 2009/05/06 00:27:30 UTC
[jira] Created: (CASSANDRA-136) Internal error processing
get_columns_since
Internal error processing get_columns_since
-------------------------------------------
Key: CASSANDRA-136
URL: https://issues.apache.org/jira/browse/CASSANDRA-136
Project: Cassandra
Issue Type: Bug
Affects Versions: trunk
Environment: Hudson build #59, accessing Cassandra via thrift/python
Reporter: daishi
In a cluster configuration, I understand that data consistency/availability is on an "as things propagate" basis.
However, I did not expect to see an actual exception thrown on a normal query using 'get_columns_since'.
I think the correct behavior when a node is unable to obtain the requested data is to simply return less data
(in this case maybe an empty list?).
Client-side I saw:
---
...i/org/apache/cassandra/Cassandra.py in recv_get_columns_since(self)
533 x.read(self._iprot)
534 self._iprot.readMessageEnd()
--> 535 raise x
536 result = get_columns_since_result()
537 result.read(self._iprot)
TApplicationException: Internal error processing get_columns_since
---
and server-side I saw:
---
ERROR [pool-1-thread-1] 2009-05-05 18:04:21,781 Cassandra.java (line 1106) Internal error processing get_columns_since
java.lang.RuntimeException: java.util.concurrent.TimeoutException: Operation timed out.
at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:262)
at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:374)
at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:117)
at org.apache.cassandra.service.CassandraServer.get_columns_since(CassandraServer.java:159)
at org.apache.cassandra.service.Cassandra$Processor$get_columns_since.process(Cassandra.java:1100)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:860)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:619)
Caused by: java.util.concurrent.TimeoutException: Operation timed out.
at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:258)
... 9 more
---
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-136) Internal error processing
get_columns_since
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708738#action_12708738 ]
Jonathan Ellis commented on CASSANDRA-136:
------------------------------------------
You should not see timeout errors during normal operation (where normal = sane load etc). I have never seen a timeout exception that wasn't caused by the remote node throwing an exception while processing the request and hence not replying at all.
> Internal error processing get_columns_since
> -------------------------------------------
>
> Key: CASSANDRA-136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-136
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: Hudson build #59, accessing Cassandra via thrift/python
> Reporter: daishi
>
> In a cluster configuration, I understand that data consistency/availability is on an "as things propagate" basis.
> However, I did not expect to see an actual exception thrown on a normal query using 'get_columns_since'.
> I think the correct behavior when a node is unable to obtain the requested data is to simply return less data
> (in this case maybe an empty list?).
> Client-side I saw:
> ---
> ...i/org/apache/cassandra/Cassandra.py in recv_get_columns_since(self)
> 533 x.read(self._iprot)
> 534 self._iprot.readMessageEnd()
> --> 535 raise x
> 536 result = get_columns_since_result()
> 537 result.read(self._iprot)
>
> TApplicationException: Internal error processing get_columns_since
> ---
> and server-side I saw:
> ---
> ERROR [pool-1-thread-1] 2009-05-05 18:04:21,781 Cassandra.java (line 1106) Internal error processing get_columns_since
> java.lang.RuntimeException: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:262)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:374)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:117)
> at org.apache.cassandra.service.CassandraServer.get_columns_since(CassandraServer.java:159)
> at org.apache.cassandra.service.Cassandra$Processor$get_columns_since.process(Cassandra.java:1100)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:860)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:258)
> ... 9 more
> ---
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-136) Internal error processing
get_columns_since
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12711762#action_12711762 ]
Jonathan Ellis commented on CASSANDRA-136:
------------------------------------------
are you still experiencing problems w/ the RC?
> Internal error processing get_columns_since
> -------------------------------------------
>
> Key: CASSANDRA-136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-136
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: Hudson build #59, accessing Cassandra via thrift/python
> Reporter: daishi
>
> In a cluster configuration, I understand that data consistency/availability is on an "as things propagate" basis.
> However, I did not expect to see an actual exception thrown on a normal query using 'get_columns_since'.
> I think the correct behavior when a node is unable to obtain the requested data is to simply return less data
> (in this case maybe an empty list?).
> Client-side I saw:
> ---
> ...i/org/apache/cassandra/Cassandra.py in recv_get_columns_since(self)
> 533 x.read(self._iprot)
> 534 self._iprot.readMessageEnd()
> --> 535 raise x
> 536 result = get_columns_since_result()
> 537 result.read(self._iprot)
>
> TApplicationException: Internal error processing get_columns_since
> ---
> and server-side I saw:
> ---
> ERROR [pool-1-thread-1] 2009-05-05 18:04:21,781 Cassandra.java (line 1106) Internal error processing get_columns_since
> java.lang.RuntimeException: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:262)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:374)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:117)
> at org.apache.cassandra.service.CassandraServer.get_columns_since(CassandraServer.java:159)
> at org.apache.cassandra.service.Cassandra$Processor$get_columns_since.process(Cassandra.java:1100)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:860)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:258)
> ... 9 more
> ---
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-136) Internal error processing
get_columns_since
Posted by "daishi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
daishi resolved CASSANDRA-136.
------------------------------
Resolution: Cannot Reproduce
Fix Version/s: 0.3
I am no longer able to reproduce this with a fresh checkout of cassandra/branches/cassandra-0.3,
so I'm going to close this.
> Internal error processing get_columns_since
> -------------------------------------------
>
> Key: CASSANDRA-136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-136
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: Hudson build #59, accessing Cassandra via thrift/python
> Reporter: daishi
> Fix For: 0.3
>
>
> In a cluster configuration, I understand that data consistency/availability is on an "as things propagate" basis.
> However, I did not expect to see an actual exception thrown on a normal query using 'get_columns_since'.
> I think the correct behavior when a node is unable to obtain the requested data is to simply return less data
> (in this case maybe an empty list?).
> Client-side I saw:
> ---
> ...i/org/apache/cassandra/Cassandra.py in recv_get_columns_since(self)
> 533 x.read(self._iprot)
> 534 self._iprot.readMessageEnd()
> --> 535 raise x
> 536 result = get_columns_since_result()
> 537 result.read(self._iprot)
>
> TApplicationException: Internal error processing get_columns_since
> ---
> and server-side I saw:
> ---
> ERROR [pool-1-thread-1] 2009-05-05 18:04:21,781 Cassandra.java (line 1106) Internal error processing get_columns_since
> java.lang.RuntimeException: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:262)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:374)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:117)
> at org.apache.cassandra.service.CassandraServer.get_columns_since(CassandraServer.java:159)
> at org.apache.cassandra.service.Cassandra$Processor$get_columns_since.process(Cassandra.java:1100)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:860)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:258)
> ... 9 more
> ---
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-136) Internal error
processing get_columns_since
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708738#action_12708738 ]
Jonathan Ellis edited comment on CASSANDRA-136 at 5/12/09 8:21 PM:
-------------------------------------------------------------------
You should not see timeout errors during normal operation (where normal = sane load etc). I have never seen a timeout exception that wasn't caused by the remote node throwing a different exception while processing the request and hence not replying at all.
>From the debug messages you can tell what node it was trying to read from, e.g.,
cassandra-1: DEBUG weakreadremote reading GetColumnReadMessage(...) from 2714@72.3.232.84:7000 [pool-1-thread-2] 2009-05-12 22:19:23,831 StorageProxy.java:246
2714 is the message id that you can track this requests's progress with. 72.3.232.84 is the host that it's going to read from.
then check the logs on 72.3.232.84 for what happened to that message id.
was (Author: jbellis):
You should not see timeout errors during normal operation (where normal = sane load etc). I have never seen a timeout exception that wasn't caused by the remote node throwing an exception while processing the request and hence not replying at all.
> Internal error processing get_columns_since
> -------------------------------------------
>
> Key: CASSANDRA-136
> URL: https://issues.apache.org/jira/browse/CASSANDRA-136
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: trunk
> Environment: Hudson build #59, accessing Cassandra via thrift/python
> Reporter: daishi
>
> In a cluster configuration, I understand that data consistency/availability is on an "as things propagate" basis.
> However, I did not expect to see an actual exception thrown on a normal query using 'get_columns_since'.
> I think the correct behavior when a node is unable to obtain the requested data is to simply return less data
> (in this case maybe an empty list?).
> Client-side I saw:
> ---
> ...i/org/apache/cassandra/Cassandra.py in recv_get_columns_since(self)
> 533 x.read(self._iprot)
> 534 self._iprot.readMessageEnd()
> --> 535 raise x
> 536 result = get_columns_since_result()
> 537 result.read(self._iprot)
>
> TApplicationException: Internal error processing get_columns_since
> ---
> and server-side I saw:
> ---
> ERROR [pool-1-thread-1] 2009-05-05 18:04:21,781 Cassandra.java (line 1106) Internal error processing get_columns_since
> java.lang.RuntimeException: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:262)
> at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:374)
> at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:117)
> at org.apache.cassandra.service.CassandraServer.get_columns_since(CassandraServer.java:159)
> at org.apache.cassandra.service.Cassandra$Processor$get_columns_since.process(Cassandra.java:1100)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:860)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:619)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:95)
> at org.apache.cassandra.service.StorageProxy.weakReadRemote(StorageProxy.java:258)
> ... 9 more
> ---
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.