You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Simon Smith (JIRA)" <ji...@apache.org> on 2009/09/11 15:38:57 UTC
[jira] Created: (CASSANDRA-440) get_key_range problems when a node
is down
get_key_range problems when a node is down
------------------------------------------
Key: CASSANDRA-440
URL: https://issues.apache.org/jira/browse/CASSANDRA-440
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.4, 0.5
Environment: 64-bit 4GB Rackspace-cloud boxes running FC11 (saw problem on 32-bit platform as well)
Reporter: Simon Smith
I'm running Cassandra on 5 nodes using the
OrderPreservingPartitioner, and have populated Cassandra with 78
records, and I can use get_key_range via Thrift just fine. Then, if I
manually kill one of the nodes (if I kill off node #5), the node (node
#1) which I've been using to call get_key_range will timeout and the
error:
Thrift: Internal error processing get_key_range
The Cassandra output traceback:
ERROR - Encountered IOException on connection:
java.nio.channels.SocketChannel[closed]
java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
at org.apache.cassandra.net.TcpConnection.connect(TcpConnection.java:349)
at org.apache.cassandra.net.SelectorManager.doProcess(SelectorManager.java:131)
at org.apache.cassandra.net.SelectorManager.run(SelectorManager.java:98)
WARN - Closing down connection java.nio.channels.SocketChannel[closed]
ERROR - Internal error processing get_key_range
java.lang.RuntimeException: java.util.concurrent.TimeoutException:
Operation timed out.
at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:573)
at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:595)
at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:853)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:606)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:675)
Caused by: java.util.concurrent.TimeoutException: Operation timed out.
at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:97)
at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:569)
... 7 more
The error starts as soon as the downed node #5 goes down and lasts
until I restart the downed node #5.
bin/nodeprobe cluster is accurate (it knows quickly when #5 is down,
and when it is up again)
Since I set the replication set to 3, I'm confused as to why (after
the first few seconds or so) there is an error just because one host
is down temporarily.
(Jonathan Ellis and I discussed this on the mailing list, let me know if more information is needed.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (CASSANDRA-440) get_key_range problems when a node
is down
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-440.
--------------------------------------
Resolution: Fixed
Fix Version/s: 0.5
0.4
committed to 0.4 and 0.5 branches
> get_key_range problems when a node is down
> ------------------------------------------
>
> Key: CASSANDRA-440
> URL: https://issues.apache.org/jira/browse/CASSANDRA-440
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4, 0.5
> Environment: 64-bit 4GB Rackspace-cloud boxes running FC11 (saw problem on 32-bit platform as well)
> Reporter: Simon Smith
> Assignee: Jonathan Ellis
> Fix For: 0.4, 0.5
>
> Attachments: 440.patch
>
>
> I'm running Cassandra on 5 nodes using the
> OrderPreservingPartitioner, and have populated Cassandra with 78
> records, and I can use get_key_range via Thrift just fine. Then, if I
> manually kill one of the nodes (if I kill off node #5), the node (node
> #1) which I've been using to call get_key_range will timeout and the
> error:
> Thrift: Internal error processing get_key_range
> The Cassandra output traceback:
> ERROR - Encountered IOException on connection:
> java.nio.channels.SocketChannel[closed]
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.apache.cassandra.net.TcpConnection.connect(TcpConnection.java:349)
> at org.apache.cassandra.net.SelectorManager.doProcess(SelectorManager.java:131)
> at org.apache.cassandra.net.SelectorManager.run(SelectorManager.java:98)
> WARN - Closing down connection java.nio.channels.SocketChannel[closed]
> ERROR - Internal error processing get_key_range
> java.lang.RuntimeException: java.util.concurrent.TimeoutException:
> Operation timed out.
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:573)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:595)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:853)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:606)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:675)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:97)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:569)
> ... 7 more
> The error starts as soon as the downed node #5 goes down and lasts
> until I restart the downed node #5.
> bin/nodeprobe cluster is accurate (it knows quickly when #5 is down,
> and when it is up again)
> Since I set the replication set to 3, I'm confused as to why (after
> the first few seconds or so) there is an error just because one host
> is down temporarily.
> (Jonathan Ellis and I discussed this on the mailing list, let me know if more information is needed.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-440) get_key_range problems when a node
is down
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-440:
-------------------------------------
Attachment: 440.patch
refactor findSuitableEndpoint to throw UnavailableException (instead of letting callers error out with NPE) when no replica is alive. also check for endpoint live-ness in getNextEndpoint.
patch is against trunk but should also apply to 0.4.
> get_key_range problems when a node is down
> ------------------------------------------
>
> Key: CASSANDRA-440
> URL: https://issues.apache.org/jira/browse/CASSANDRA-440
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4, 0.5
> Environment: 64-bit 4GB Rackspace-cloud boxes running FC11 (saw problem on 32-bit platform as well)
> Reporter: Simon Smith
> Attachments: 440.patch
>
>
> I'm running Cassandra on 5 nodes using the
> OrderPreservingPartitioner, and have populated Cassandra with 78
> records, and I can use get_key_range via Thrift just fine. Then, if I
> manually kill one of the nodes (if I kill off node #5), the node (node
> #1) which I've been using to call get_key_range will timeout and the
> error:
> Thrift: Internal error processing get_key_range
> The Cassandra output traceback:
> ERROR - Encountered IOException on connection:
> java.nio.channels.SocketChannel[closed]
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.apache.cassandra.net.TcpConnection.connect(TcpConnection.java:349)
> at org.apache.cassandra.net.SelectorManager.doProcess(SelectorManager.java:131)
> at org.apache.cassandra.net.SelectorManager.run(SelectorManager.java:98)
> WARN - Closing down connection java.nio.channels.SocketChannel[closed]
> ERROR - Internal error processing get_key_range
> java.lang.RuntimeException: java.util.concurrent.TimeoutException:
> Operation timed out.
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:573)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:595)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:853)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:606)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:675)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:97)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:569)
> ... 7 more
> The error starts as soon as the downed node #5 goes down and lasts
> until I restart the downed node #5.
> bin/nodeprobe cluster is accurate (it knows quickly when #5 is down,
> and when it is up again)
> Since I set the replication set to 3, I'm confused as to why (after
> the first few seconds or so) there is an error just because one host
> is down temporarily.
> (Jonathan Ellis and I discussed this on the mailing list, let me know if more information is needed.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (CASSANDRA-440) get_key_range problems when a node
is down
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis reassigned CASSANDRA-440:
----------------------------------------
Assignee: Jonathan Ellis
> get_key_range problems when a node is down
> ------------------------------------------
>
> Key: CASSANDRA-440
> URL: https://issues.apache.org/jira/browse/CASSANDRA-440
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4, 0.5
> Environment: 64-bit 4GB Rackspace-cloud boxes running FC11 (saw problem on 32-bit platform as well)
> Reporter: Simon Smith
> Assignee: Jonathan Ellis
> Attachments: 440.patch
>
>
> I'm running Cassandra on 5 nodes using the
> OrderPreservingPartitioner, and have populated Cassandra with 78
> records, and I can use get_key_range via Thrift just fine. Then, if I
> manually kill one of the nodes (if I kill off node #5), the node (node
> #1) which I've been using to call get_key_range will timeout and the
> error:
> Thrift: Internal error processing get_key_range
> The Cassandra output traceback:
> ERROR - Encountered IOException on connection:
> java.nio.channels.SocketChannel[closed]
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.apache.cassandra.net.TcpConnection.connect(TcpConnection.java:349)
> at org.apache.cassandra.net.SelectorManager.doProcess(SelectorManager.java:131)
> at org.apache.cassandra.net.SelectorManager.run(SelectorManager.java:98)
> WARN - Closing down connection java.nio.channels.SocketChannel[closed]
> ERROR - Internal error processing get_key_range
> java.lang.RuntimeException: java.util.concurrent.TimeoutException:
> Operation timed out.
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:573)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:595)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:853)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:606)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:675)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:97)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:569)
> ... 7 more
> The error starts as soon as the downed node #5 goes down and lasts
> until I restart the downed node #5.
> bin/nodeprobe cluster is accurate (it knows quickly when #5 is down,
> and when it is up again)
> Since I set the replication set to 3, I'm confused as to why (after
> the first few seconds or so) there is an error just because one host
> is down temporarily.
> (Jonathan Ellis and I discussed this on the mailing list, let me know if more information is needed.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-440) get_key_range problems when a
node is down
Posted by "Simon Smith (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12755230#action_12755230 ]
Simon Smith commented on CASSANDRA-440:
---------------------------------------
Jonathan:
I tried out the patch you attached above, I applied it to 0.4, and it works for me. Now, as soon as I take a node down, there may be one or two seconds of the thrift-internal error, the timeout (which I totally expect, and this is obviously OK) but as soon as the host doing the querying can see the node is down, the error stops, and valid output is given by the get_key_range query again. And there isn't any disruption when the node comes back up.
Thanks!
Simon Smith
> get_key_range problems when a node is down
> ------------------------------------------
>
> Key: CASSANDRA-440
> URL: https://issues.apache.org/jira/browse/CASSANDRA-440
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4, 0.5
> Environment: 64-bit 4GB Rackspace-cloud boxes running FC11 (saw problem on 32-bit platform as well)
> Reporter: Simon Smith
> Assignee: Jonathan Ellis
> Attachments: 440.patch
>
>
> I'm running Cassandra on 5 nodes using the
> OrderPreservingPartitioner, and have populated Cassandra with 78
> records, and I can use get_key_range via Thrift just fine. Then, if I
> manually kill one of the nodes (if I kill off node #5), the node (node
> #1) which I've been using to call get_key_range will timeout and the
> error:
> Thrift: Internal error processing get_key_range
> The Cassandra output traceback:
> ERROR - Encountered IOException on connection:
> java.nio.channels.SocketChannel[closed]
> java.net.ConnectException: Connection refused
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:592)
> at org.apache.cassandra.net.TcpConnection.connect(TcpConnection.java:349)
> at org.apache.cassandra.net.SelectorManager.doProcess(SelectorManager.java:131)
> at org.apache.cassandra.net.SelectorManager.run(SelectorManager.java:98)
> WARN - Closing down connection java.nio.channels.SocketChannel[closed]
> ERROR - Internal error processing get_key_range
> java.lang.RuntimeException: java.util.concurrent.TimeoutException:
> Operation timed out.
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:573)
> at org.apache.cassandra.service.CassandraServer.get_key_range(CassandraServer.java:595)
> at org.apache.cassandra.service.Cassandra$Processor$get_key_range.process(Cassandra.java:853)
> at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:606)
> at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:253)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:675)
> Caused by: java.util.concurrent.TimeoutException: Operation timed out.
> at org.apache.cassandra.net.AsyncResult.get(AsyncResult.java:97)
> at org.apache.cassandra.service.StorageProxy.getKeyRange(StorageProxy.java:569)
> ... 7 more
> The error starts as soon as the downed node #5 goes down and lasts
> until I restart the downed node #5.
> bin/nodeprobe cluster is accurate (it knows quickly when #5 is down,
> and when it is up again)
> Since I set the replication set to 3, I'm confused as to why (after
> the first few seconds or so) there is an error just because one host
> is down temporarily.
> (Jonathan Ellis and I discussed this on the mailing list, let me know if more information is needed.)
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.