You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Aaron Morton (JIRA)" <ji...@apache.org> on 2011/02/07 13:50:32 UTC
[jira] Reopened: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Morton reopened CASSANDRA-2081:
-------------------------------------
StorageProxy.scan() is also reusing Message objects, with the same result of requests at QUORUM timing out for get_indexed_slice()
Log messages below show a get_indexed_slice() in a 3 node cluster @ QUORUM. The new log message shows the callback has already been removed.
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,893 CassandraServer.java (line 509) scan
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,894 StorageProxy.java (line 703) restricted ranges for query [-1,-1] are [[-1,85070591730234615865843651857942052864], (85070591730234615865843651857942052864,95070591730234615865843651857942052864], (95070591730234615865843651857942052864,108074891939685041992920030907211891412], (108074891939685041992920030907211891412,-1]]
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,919 StorageProxy.java (line 796) scan ranges are [-1,85070591730234615865843651857942052864],(85070591730234615865843651857942052864,95070591730234615865843651857942052864],(95070591730234615865843651857942052864,108074891939685041992920030907211891412],(108074891939685041992920030907211891412,-1]
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,919 ReadCallback.java (line 58) ReadCallback blocking for 2 responses
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,923 StorageProxy.java (line 819) reading org.apache.cassandra.db.IndexScanCommand@26a1b248 from 6289@/127.0.0.3
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,923 StorageProxy.java (line 819) reading org.apache.cassandra.db.IndexScanCommand@26a1b248 from 6289@/127.0.0.2
DEBUG [RequestResponseStage:1] 2011-02-08 01:46:32,926 ResponseVerbHandler.java (line 51) Processing response on a callback from 6289@/127.0.0.3
DEBUG [RequestResponseStage:2] 2011-02-08 01:46:32,928 ResponseVerbHandler.java (line 41) Callback already removed for 6289
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Assignee: Aaron Morton
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira