You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Thibaut (JIRA)" <ji...@apache.org> on 2011/01/31 20:40:28 UTC
[jira] Created: (CASSANDRA-2081) Consistency QUORUM does not work
anymore
Consistency QUORUM does not work anymore
----------------------------------------
Key: CASSANDRA-2081
URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: linux, hector + cassandra
Reporter: Thibaut
Priority: Blocker
Fix For: 0.7.1
I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
I have tried this with one dead node in my cluster.
If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
intr1n11 is marked as up however and I can also access the node through the cassandra cli.
192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
192.168.0.2 Up Normal 7.96 GB 5.00% 199
192.168.0.3 Up Normal 8.24 GB 5.00% 266
192.168.0.4 Up Normal 4.94 GB 5.00% 333
192.168.0.5 Up Normal 5.02 GB 5.00% 400
192.168.0.6 Up Normal 5 GB 5.00% 4cc
192.168.0.7 Up Normal 5.1 GB 5.00% 599
192.168.0.8 Up Normal 5.07 GB 5.00% 666
192.168.0.9 Up Normal 4.78 GB 5.00% 733
192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
192.168.0.12 Up Normal 5.31 GB 5.00% 999
192.168.0.13 Up Normal 5.56 GB 5.00% a66
192.168.0.14 Up Normal 5.82 GB 5.00% b33
192.168.0.15 Up Normal 5.57 GB 5.00% c00
192.168.0.16 Up Normal 5.03 GB 5.00% ccc
192.168.0.17 Up Normal 4.77 GB 5.00% d99
192.168.0.18 Down Normal ? 5.00% e66
192.168.0.19 Up Normal 4.78 GB 5.00% f33
192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Morton updated CASSANDRA-2081:
------------------------------------
Attachment: 2081-2.txt
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Assignee: Aaron Morton
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-2.txt, 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2081) Consistency QUORUM
does not work anymore
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988867#comment-12988867 ]
Thibaut edited comment on CASSANDRA-2081 at 1/31/11 8:02 PM:
-------------------------------------------------------------
My application hangs/blocks forever as I catch all the Hector exceptions and retry when there was an error.
Above log file messages will repeat itself again and again.
There are also no error messages in the cassandra log file.
was (Author: tbritz):
My application hangs/blocks forever as I catch all the Hector exceptions and retry when there was an error.
Above log file messages will repeat itself again and again.
> Consistency QUORUM does not work anymore
> ----------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990078#comment-12990078 ]
Thibaut commented on CASSANDRA-2081:
------------------------------------
As Michel noted, this fixes the problem. Thanks :-)
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989828#comment-12989828 ]
Aaron Morton commented on CASSANDRA-2081:
-----------------------------------------
Looking at the right code now.
In StorageProxy.fetchRows() line 395 the digestMessage is shared for all non local digest requests.
And I just realised the log messages above are coming from StorageProxy.getRangeSlice() where the message object is shared for all endpoints. The log messages from Thibaut show the same message ID on nodes 0.1 0.2 and 0.3.
My reading of the MessagingService and ResponseVerbHandler is that once a message with ID X has been received, if others are received with the same ID they will be ignored. Is that correct? If so this looks like a bug in the two methods above.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
> Attachments: 2081-logging.patch
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988865#comment-12988865 ]
Jonathan Ellis commented on CASSANDRA-2081:
-------------------------------------------
What kind of "doesn't work" are you seeing?
> Consistency QUORUM does not work anymore
> ----------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2081) Consistency QUORUM
does not work anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989032#comment-12989032 ]
Aaron Morton edited comment on CASSANDRA-2081 at 2/1/11 4:46 AM:
-----------------------------------------------------------------
I've sort of stumbled onto something similar with an 0.7 install. I need to go home now so cannot dig any deeper and rule out human error, but this is what I have.
5 node 0.7.0 install
1) Load data in using
python stress.py -d jb-cass1,jb-cass2,jb-cass3,jb-cass4,jb-cass5 -o insert -n 1000000 -e QUORUM -t 10 -i 1 -l 3
(use all 5 nodes, insert 1,000,000 rows with RF 3 and QUORUM and 10 threads, report progress every second)
2) Read back using
python stress.py -d jb-cass2,jb-cass3,jb-cass4,jb-cass5 -o read -n 1000000 -e QUORUM -t 10 -i 1
(note that jb-cass1 is removed from the list)
3) make big bang
Once the read has run a few seconds I ran "reboot -f" on node 1. I expect the read operations to complete, output was
11270,1315,1315,0.00839671943578,9
11631,361,361,0.00746133188792,11
11631,0,0,NaN,12
11631,0,0,NaN,13
11631,0,0,NaN,14
11631,0,0,NaN,15
11631,0,0,NaN,16
11631,0,0,NaN,17
11631,0,0,NaN,18
11631,0,0,NaN,19
Process Reader-10:
Traceback (most recent call last):
File "/vol/apps/python-2.6.4_64/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "stress.py", line 279, in run
r = self.cclient.get_slice(key, parent, p, consistency)
File "/local1/frameworks/cassandra/apache-cassandra-0.7.0-src/contrib/py_stress/cassandra/Cassandra.py", line 432, in get_slice
return self.recv_get_slice()
File "/local1/frameworks/cassandra/apache-cassandra-0.7.0-src/contrib/py_stress/cassandra/Cassandra.py", line 462, in recv_get_slice
raise result.te
All clients died. stress.py is not setting a timeout on the thrift socket, so am guessing this is server side.
I was running DEBUG on all the nodes (but had turned off the line numbers), this is from one. the 114.63 machine is obviously the one I killed.
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,186 StorageService.java (line org.apache.cassandra.service.StorageService) Sorted endpoints are /192.168.114.63,jb08.wetafx.co.nz/192.168.114.67,/192.168.114.64
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,186 QuorumResponseHandler.java (line org.apache.cassandra.service.QuorumResponseHandler) QuorumResponseHandler blocking for 2 responses
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,186 StorageProxy.java (line org.apache.cassandra.service.StorageProxy) strongread reading digest for SliceFromReadCommand(table='Keyspace1', key='30323334343534', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) from 6624@jb08.wetafx.co.nz/192.168.114.67
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,187 StorageProxy.java (line org.apache.cassandra.service.StorageProxy) strongread reading data for SliceFromReadCommand(table='Keyspace1', key='30323334343534', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) from 6623@/192.168.114.63
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,187 StorageProxy.java (line org.apache.cassandra.service.StorageProxy) strongread reading digest for SliceFromReadCommand(table='Keyspace1', key='30323334343534', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) from 6624@/192.168.114.64
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 0 of 5: 4330:false:34@1296532428248604
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 1 of 5: 4331:false:34@1296532428248637
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 2 of 5: 4332:false:34@1296532428248640
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 3 of 5: 4333:false:34@1296532428248642
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 4 of 5: 4334:false:34@1296532428248656
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 ReadVerbHandler.java (line org.apache.cassandra.db.ReadVerbHandler) digest is 220b82e28c2bb4be869c168243d75f01
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 ReadVerbHandler.java (line org.apache.cassandra.db.ReadVerbHandler) Read key 30323334343534; sending response to 7D8FA1FD-A2FE-6A54-7BB0-3B129206D1E1@jb08.wetafx.co.nz/192.168.114.67
DEBUG [RequestResponseStage:13] 2011-02-01 17:14:08,188 ResponseVerbHandler.java (line org.apache.cassandra.net.ResponseVerbHandler) Processing response on a callback from 7D8FA1FD-A2FE-6A54-7BB0-3B129206D1E1@jb08.wetafx.co.nz/192.168.114.67
DEBUG [RequestResponseStage:13] 2011-02-01 17:14:08,188 ReadResponseResolver.java (line org.apache.cassandra.service.ReadResponseResolver) Preprocessed digest response
DEBUG [RequestResponseStage:16] 2011-02-01 17:14:08,188 ResponseVerbHandler.java (line org.apache.cassandra.net.ResponseVerbHandler) Processing response on a callback from 7D8FA1FD-A2FE-6A54-7BB0-3B129206D1E1@/192.168.114.64
DEBUG [RequestResponseStage:16] 2011-02-01 17:14:08,188 ReadResponseResolver.java (line org.apache.cassandra.service.ReadResponseResolver) Preprocessed digest response
INFO [ScheduledTasks:1] 2011-02-01 17:14:15,438 Gossiper.java (line org.apache.cassandra.gms.Gossiper) InetAddress /192.168.114.63 is now dead.
DEBUG [ScheduledTasks:1] 2011-02-01 17:14:15,440 MessagingService.java (line org.apache.cassandra.net.MessagingService) Resetting pool for /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:14:17,442 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [pool-1-thread-1] 2011-02-01 17:14:18,183 CassandraServer.java (line org.apache.cassandra.thrift.CassandraServer) ... timed out
DEBUG [pool-1-thread-2] 2011-02-01 17:14:18,189 CassandraServer.java (line org.apache.cassandra.thrift.CassandraServer) ... timed out
DEBUG [pool-1-thread-1] 2011-02-01 17:14:18,232 ClientState.java (line org.apache.cassandra.service.ClientState) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-2] 2011-02-01 17:14:18,232 ClientState.java (line org.apache.cassandra.service.ClientState) logged out: #<User allow_all groups=[]>
DEBUG [ScheduledTasks:1] 2011-02-01 17:14:29,811 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [ScheduledTasks:1] 2011-02-01 17:15:29,814 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:15:53,637 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:08,667 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:23,697 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [ScheduledTasks:1] 2011-02-01 17:16:29,816 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:36,723 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:50,751 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:03,775 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:19,807 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [ScheduledTasks:1] 2011-02-01 17:17:29,818 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:32,834 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:42,852 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:56,880 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
need to leave work now so may not be able to get further into this until tomorrow.
was (Author: amorton):
I've sort of stumbled onto something similar with an 0.7 install. I need to go home now so cannot dig any deeper and rule out human error, but this is what I have.
5 node 0.7.0 install
1) Load data in using
python stress.py -d jb-cass1,jb-cass2,jb-cass3,jb-cass4,jb-cass5 -o insert -n 1000000 -e QUORUM -t 10 -i 1 -l 3
(use all 5 users, insert 1,000,000 rows with RF 3 and QUORUM and 10 threads, report progress every second)
2) Read back using
python stress.py -d jb-cass2,jb-cass3,jb-cass4,jb-cass5 -o read -n 1000000 -e QUORUM -t 10 -i 1
(note that jb-cass1 is removed from the list)
3) make big bang
Once the read has run a few seconds I ran "reboot -f" on node 1. I expect the read operations to complete, output was
11270,1315,1315,0.00839671943578,9
11631,361,361,0.00746133188792,11
11631,0,0,NaN,12
11631,0,0,NaN,13
11631,0,0,NaN,14
11631,0,0,NaN,15
11631,0,0,NaN,16
11631,0,0,NaN,17
11631,0,0,NaN,18
11631,0,0,NaN,19
Process Reader-10:
Traceback (most recent call last):
File "/vol/apps/python-2.6.4_64/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "stress.py", line 279, in run
r = self.cclient.get_slice(key, parent, p, consistency)
File "/local1/frameworks/cassandra/apache-cassandra-0.7.0-src/contrib/py_stress/cassandra/Cassandra.py", line 432, in get_slice
return self.recv_get_slice()
File "/local1/frameworks/cassandra/apache-cassandra-0.7.0-src/contrib/py_stress/cassandra/Cassandra.py", line 462, in recv_get_slice
raise result.te
All clients died. stress.py is not setting a timeout on the thrift socket, so am guessing this is server side.
I was running DEBUG on all the nodes (but had turned off the line numbers), this is from one. the 114.63 machine is obviously the one I killed.
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,186 StorageService.java (line org.apache.cassandra.service.StorageService) Sorted endpoints are /192.168.114.63,jb08.wetafx.co.nz/192.168.114.67,/192.168.114.64
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,186 QuorumResponseHandler.java (line org.apache.cassandra.service.QuorumResponseHandler) QuorumResponseHandler blocking for 2 responses
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,186 StorageProxy.java (line org.apache.cassandra.service.StorageProxy) strongread reading digest for SliceFromReadCommand(table='Keyspace1', key='30323334343534', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) from 6624@jb08.wetafx.co.nz/192.168.114.67
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,187 StorageProxy.java (line org.apache.cassandra.service.StorageProxy) strongread reading data for SliceFromReadCommand(table='Keyspace1', key='30323334343534', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) from 6623@/192.168.114.63
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,187 StorageProxy.java (line org.apache.cassandra.service.StorageProxy) strongread reading digest for SliceFromReadCommand(table='Keyspace1', key='30323334343534', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) from 6624@/192.168.114.64
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 0 of 5: 4330:false:34@1296532428248604
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 1 of 5: 4331:false:34@1296532428248637
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 2 of 5: 4332:false:34@1296532428248640
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 3 of 5: 4333:false:34@1296532428248642
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 4 of 5: 4334:false:34@1296532428248656
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 ReadVerbHandler.java (line org.apache.cassandra.db.ReadVerbHandler) digest is 220b82e28c2bb4be869c168243d75f01
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 ReadVerbHandler.java (line org.apache.cassandra.db.ReadVerbHandler) Read key 30323334343534; sending response to 7D8FA1FD-A2FE-6A54-7BB0-3B129206D1E1@jb08.wetafx.co.nz/192.168.114.67
DEBUG [RequestResponseStage:13] 2011-02-01 17:14:08,188 ResponseVerbHandler.java (line org.apache.cassandra.net.ResponseVerbHandler) Processing response on a callback from 7D8FA1FD-A2FE-6A54-7BB0-3B129206D1E1@jb08.wetafx.co.nz/192.168.114.67
DEBUG [RequestResponseStage:13] 2011-02-01 17:14:08,188 ReadResponseResolver.java (line org.apache.cassandra.service.ReadResponseResolver) Preprocessed digest response
DEBUG [RequestResponseStage:16] 2011-02-01 17:14:08,188 ResponseVerbHandler.java (line org.apache.cassandra.net.ResponseVerbHandler) Processing response on a callback from 7D8FA1FD-A2FE-6A54-7BB0-3B129206D1E1@/192.168.114.64
DEBUG [RequestResponseStage:16] 2011-02-01 17:14:08,188 ReadResponseResolver.java (line org.apache.cassandra.service.ReadResponseResolver) Preprocessed digest response
INFO [ScheduledTasks:1] 2011-02-01 17:14:15,438 Gossiper.java (line org.apache.cassandra.gms.Gossiper) InetAddress /192.168.114.63 is now dead.
DEBUG [ScheduledTasks:1] 2011-02-01 17:14:15,440 MessagingService.java (line org.apache.cassandra.net.MessagingService) Resetting pool for /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:14:17,442 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [pool-1-thread-1] 2011-02-01 17:14:18,183 CassandraServer.java (line org.apache.cassandra.thrift.CassandraServer) ... timed out
DEBUG [pool-1-thread-2] 2011-02-01 17:14:18,189 CassandraServer.java (line org.apache.cassandra.thrift.CassandraServer) ... timed out
DEBUG [pool-1-thread-1] 2011-02-01 17:14:18,232 ClientState.java (line org.apache.cassandra.service.ClientState) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-2] 2011-02-01 17:14:18,232 ClientState.java (line org.apache.cassandra.service.ClientState) logged out: #<User allow_all groups=[]>
DEBUG [ScheduledTasks:1] 2011-02-01 17:14:29,811 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [ScheduledTasks:1] 2011-02-01 17:15:29,814 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:15:53,637 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:08,667 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:23,697 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [ScheduledTasks:1] 2011-02-01 17:16:29,816 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:36,723 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:50,751 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:03,775 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:19,807 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [ScheduledTasks:1] 2011-02-01 17:17:29,818 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:32,834 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:42,852 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:56,880 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
need to leave work now so may not be able to get further into this until tomorrow.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988879#comment-12988879 ]
Jonathan Ellis commented on CASSANDRA-2081:
-------------------------------------------
Is this RF=3?
What do you see in the Cassandra log when you set log level to debug, for the queries that Hector gives up on?
What are the versions you tried that works/doesn't work? (In description above both versions are given as apache-cassandra-2011-01-28_20-06-01.jar.)
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989841#comment-12989841 ]
Aaron Morton commented on CASSANDRA-2081:
-----------------------------------------
Out of interest, how about making the ExpiringMap check the return of NonBlockingHashMap.put() and assert it was the value passed in. May catch future problems.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991397#comment-12991397 ]
Hudson commented on CASSANDRA-2081:
-----------------------------------
Integrated in Cassandra-0.7 #252 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/252/])
avoid re-using Message object in index queries
patch by Aaron Morton; reviewed by jbellis for CASSANDRA-2081
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Assignee: Aaron Morton
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-2.txt, 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989375#comment-12989375 ]
Aaron Morton commented on CASSANDRA-2081:
-----------------------------------------
My understanding here is the 0.19 node is sending read requests to the 0.1, 0.2 and 0.3 nodes and only getting a reply from the 0.1 node before timing out. The 0.1 node is the first node the request is sent to, so this is the data request the others are digest.
The timeout is the rpc_timeout, and can be seen here...
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,949 ReadCallback.java (line 58) ReadCallback blocking for 2 responses
...10 seconds...
DEBUG [pool-1-thread-1] 2011-02-01 11:48:38,950 CassandraServer.java (line 483) ... timed out
Whats happening on the 0.2 and 0.3 nodes at this point? Are they logging errors or WARN messages about dropped messages ? Can you see any logs about processing messages from the 0.19 node? I'm not sure the down 0.18 node is a factor here.
The client should be retrying when it gets a timeout, which I think you said Hector was doing.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2081:
--------------------------------------
Priority: Critical (was: Blocker)
Affects Version/s: 0.7.1
0.6.10
Fix Version/s: 0.6.12
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989808#comment-12989808 ]
Jonathan Ellis commented on CASSANDRA-2081:
-------------------------------------------
Thibaut is using a 0.7.1 prerelease version where the uuid messages from sendRR have been replaced with unique per-host messages (which are standard int IDs like the old non-shared messages were).
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
> Attachments: 2081-logging.patch
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989100#comment-12989100 ]
Aaron Morton commented on CASSANDRA-2081:
-----------------------------------------
Had a read through the code for think I have an idea what my problem was, not sure if it applies to the previous issue and not sure if its a real bug.
o.a.c.service.ReadCallback.response() will only signal the o.a.c.utils.SimpleCondition if the data request has been received. If the signal is not set after rpc_timeout then ReadCallback.get() will raise a j.u.c.TimeoutException() which comes out of the StorageProxy and it caught in CassandraServer and turned into a o.a.c.thrift.TimedOutException.
So if the node that is asked for the data fails to return, the entire request will timeout even if there are enough nodes to serve the request. I think I've seen this discussed before as by design, and the client should just retry in response to the timeout. Is that correct ?
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2081) Consistency QUORUM
does not work anymore (hector:Could not fullfill request on this host)
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988867#comment-12988867 ]
Thibaut edited comment on CASSANDRA-2081 at 1/31/11 8:10 PM:
-------------------------------------------------------------
My application hangs/blocks forever as I catch all the Hector exceptions and retry when there was an error.
Above log file messages will repeat itself again and again.
There are also no error messages in the cassandra log file.
Also "Could not fullfill request on this host CassandraClient" is an error message I have never seen before.
was (Author: tbritz):
My application hangs/blocks forever as I catch all the Hector exceptions and retry when there was an error.
Above log file messages will repeat itself again and again.
There are also no error messages in the cassandra log file.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989804#comment-12989804 ]
Aaron Morton commented on CASSANDRA-2081:
-----------------------------------------
There is something odd about the way the messages are been described in the logs, e.g. yours say
2011-02-02 12:54:44,554 DEBUG [RequestResponseStage:5] ResponseVerbHandler.java (line 48) Processing response on a callback from 600@/192.168.0.1
mine say
DEBUG [RequestResponseStage:16] 2011-02-01 17:14:08,188 ResponseVerbHandler.java (line org.apache.cassandra.net.ResponseVerbHandler) Processing response on a callback from 7D8FA1FD-A2FE-6A54-7BB0-3B129206D1E1@/192.168.114.6
The MessagingService.sendRR() overload that takes an array (and is called when there are multiple messages) should be updating the message ID for all messages to a shared GUID before sending. This happens the log message about "StorageProxy.java (line 562) reading RangeSliceCommand..." It should be present in the ResponseVerbHandler log message.
Perhaps this means the callback for the message cannot be found. To test the theory can you apply the 2081-logging.patch to the node your are running the request on? It updates ResponseVernHandler to log when a message is lost.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988894#comment-12988894 ]
Brandon Williams commented on CASSANDRA-2081:
---------------------------------------------
I'm not able to reproduce with contrib/stress, can you try that?
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thibaut updated CASSANDRA-2081:
-------------------------------
Summary: Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host) (was: Consistency QUORUM does not work anymore)
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2081.
---------------------------------------
Resolution: Fixed
Reviewer: jbellis
Assignee: Aaron Morton
Thanks for helping track that down everyone!
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Assignee: Aaron Morton
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Reopened: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Morton reopened CASSANDRA-2081:
-------------------------------------
StorageProxy.scan() is also reusing Message objects, with the same result of requests at QUORUM timing out for get_indexed_slice()
Log messages below show a get_indexed_slice() in a 3 node cluster @ QUORUM. The new log message shows the callback has already been removed.
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,893 CassandraServer.java (line 509) scan
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,894 StorageProxy.java (line 703) restricted ranges for query [-1,-1] are [[-1,85070591730234615865843651857942052864], (85070591730234615865843651857942052864,95070591730234615865843651857942052864], (95070591730234615865843651857942052864,108074891939685041992920030907211891412], (108074891939685041992920030907211891412,-1]]
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,919 StorageProxy.java (line 796) scan ranges are [-1,85070591730234615865843651857942052864],(85070591730234615865843651857942052864,95070591730234615865843651857942052864],(95070591730234615865843651857942052864,108074891939685041992920030907211891412],(108074891939685041992920030907211891412,-1]
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,919 ReadCallback.java (line 58) ReadCallback blocking for 2 responses
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,923 StorageProxy.java (line 819) reading org.apache.cassandra.db.IndexScanCommand@26a1b248 from 6289@/127.0.0.3
DEBUG [pool-1-thread-12] 2011-02-08 01:46:32,923 StorageProxy.java (line 819) reading org.apache.cassandra.db.IndexScanCommand@26a1b248 from 6289@/127.0.0.2
DEBUG [RequestResponseStage:1] 2011-02-08 01:46:32,926 ResponseVerbHandler.java (line 51) Processing response on a callback from 6289@/127.0.0.3
DEBUG [RequestResponseStage:2] 2011-02-08 01:46:32,928 ResponseVerbHandler.java (line 41) Callback already removed for 6289
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Assignee: Aaron Morton
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989689#comment-12989689 ]
Thibaut commented on CASSANDRA-2081:
------------------------------------
Debug Output:
========================================
192.168.0.1
--------------------
2011-02-02 12:54:29,117 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line 349) Disseminating load info ...
2011-02-02 12:54:35,016 DEBUG [ReadStage:8] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=} to 567@/192.168.0.7
2011-02-02 12:54:45,015 DEBUG [ReadStage:9] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=} to 600@/192.168.0.7
2011-02-02 12:54:49,679 DEBUG [pool-1-thread-5] ClientState.java (line 91) logged out: #<User allow_all groups=[]>
========================================
192.168.0.2
--------------------
2011-02-02 12:54:30,147 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line 349) Disseminating load info ...
2011-02-02 12:54:34,789 DEBUG [MutationStage:5] RowMutationVerbHandler.java (line 52) Applying RowMutation(keyspace='table_lists', key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334', modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674390000!2419199,]), ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674391000!2419199,])])
2011-02-02 12:54:34,790 DEBUG [MutationStage:5] Table.java (line 397) applying mutation of row 32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
2011-02-02 12:54:34,792 DEBUG [MutationStage:5] RowMutationVerbHandler.java (line 81) RowMutation(keyspace='table_lists', key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334', modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674390000!2419199,]), ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674391000!2419199,])]) applied. Sending response to 566@/192.168.0.7
2011-02-02 12:54:34,953 DEBUG [ReadStage:8] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=} to 567@/192.168.0.7
2011-02-02 12:54:44,965 DEBUG [ReadStage:9] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=} to 600@/192.168.0.7
2011-02-02 12:54:45,057 DEBUG [ScheduledTasks:1] GCInspector.java (line 135) GC for ParNew: 14 ms, 13086296 reclaimed leaving 2087311488 used; max is 4856348672
2011-02-02 12:54:49,613 DEBUG [pool-1-thread-11] ClientState.java (line 91) logged out: #<User allow_all groups=[]>
========================================
192.168.0.3
--------------------
2011-02-02 12:54:30,920 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line 349) Disseminating load info ...
2011-02-02 12:54:34,368 DEBUG [MutationStage:8] RowMutationVerbHandler.java (line 52) Applying RowMutation(keyspace='table_lists', key='36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165', modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674314000!2419199,]), ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674322000!2419199,])])
2011-02-02 12:54:34,369 DEBUG [MutationStage:8] Table.java (line 397) applying mutation of row 36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165
2011-02-02 12:54:34,371 DEBUG [MutationStage:8] RowMutationVerbHandler.java (line 81) RowMutation(keyspace='table_lists', key='36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165', modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674314000!2419199,]), ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674322000!2419199,])]) applied. Sending response to 562@/192.168.0.7
2011-02-02 12:54:34,381 DEBUG [MutationStage:9] RowMutationVerbHandler.java (line 52) Applying RowMutation(keyspace='table_lists', key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334', modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674390000!2419199,]), ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674391000!2419199,])])
2011-02-02 12:54:34,382 DEBUG [MutationStage:9] Table.java (line 397) applying mutation of row 32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334
2011-02-02 12:54:34,383 DEBUG [MutationStage:9] RowMutationVerbHandler.java (line 81) RowMutation(keyspace='table_lists', key='32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334', modifications=[ColumnFamily(table_lists [7461626c655f6c69737473:false:116@1296647674390000!2419199,]), ColumnFamily(table_lists_meta [6e6578745f72657175657374:false:8@1296647674391000!2419199,])]) applied. Sending response to 564@/192.168.0.7
2011-02-02 12:54:34,539 DEBUG [ReadStage:8] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=} to 567@/192.168.0.7
2011-02-02 12:54:44,540 DEBUG [ReadStage:9] RangeSliceVerbHandler.java (line 55) Sending RangeSliceReply{rows=} to 600@/192.168.0.7
2011-02-02 12:54:49,202 DEBUG [pool-1-thread-13] ClientState.java (line 91) logged out: #<User allow_all groups=[]>
========================================
192.168.0.7
--------------------
2011-02-02 12:54:34,059 DEBUG [ScheduledTasks:1] StorageLoadBalancer.java (line 349) Disseminating load info ...
2011-02-02 12:54:34,356 DEBUG [pool-1-thread-1] CassandraServer.java (line 355) batch_mutate
2011-02-02 12:54:34,375 DEBUG [pool-1-thread-1] StorageProxy.java (line 154) insert writing key 36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165 to 561@/192.168.0.5
2011-02-02 12:54:34,375 DEBUG [pool-1-thread-1] StorageProxy.java (line 154) insert writing key 36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165 to 562@/192.168.0.3
2011-02-02 12:54:34,381 DEBUG [pool-1-thread-1] StorageProxy.java (line 154) insert writing key 36306463373934666139396136366665303539636130373063333366323066615f2e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313636345f65626263303033372d356639612d343066382d383833382d356436336433616233366165 to 563@/94.242.198.13
2011-02-02 12:54:34,384 DEBUG [RequestResponseStage:3] ResponseVerbHandler.java (line 48) Processing response on a callback from 561@/192.168.0.5
2011-02-02 12:54:34,386 DEBUG [RequestResponseStage:5] ResponseVerbHandler.java (line 48) Processing response on a callback from 563@/192.168.0.4
2011-02-02 12:54:34,386 DEBUG [RequestResponseStage:4] ResponseVerbHandler.java (line 48) Processing response on a callback from 562@/192.168.0.3
2011-02-02 12:54:34,391 DEBUG [pool-1-thread-2] CassandraServer.java (line 355) batch_mutate
2011-02-02 12:54:34,393 DEBUG [pool-1-thread-2] StorageProxy.java (line 154) insert writing key 32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334 to 564@/192.168.0.3
2011-02-02 12:54:34,394 DEBUG [pool-1-thread-2] StorageProxy.java (line 154) insert writing key 32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334 to 565@/192.168.0.4
2011-02-02 12:54:34,394 DEBUG [pool-1-thread-2] StorageProxy.java (line 154) insert writing key 32383663623561363162363164306231333162326264656232303730303866625f7777772e7768617469736d79697076362e6e65745f686f7374726571756573746c6973745f393232333337303734303230373130313432315f36346334613666322d333737392d343331642d623334382d663636383533316233656334 to 566@/192.168.0.2
2011-02-02 12:54:34,398 DEBUG [RequestResponseStage:6] ResponseVerbHandler.java (line 48) Processing response on a callback from 564@/192.168.0.3
2011-02-02 12:54:34,398 DEBUG [RequestResponseStage:7] ResponseVerbHandler.java (line 48) Processing response on a callback from 565@/192.168.0.4
2011-02-02 12:54:34,399 DEBUG [RequestResponseStage:8] ResponseVerbHandler.java (line 48) Processing response on a callback from 566@/192.168.0.2
2011-02-02 12:54:34,531 DEBUG [pool-1-thread-3] CassandraServer.java (line 445) range_slice
2011-02-02 12:54:34,533 DEBUG [pool-1-thread-3] StorageProxy.java (line 514) RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,], max_keys=250}
2011-02-02 12:54:34,534 DEBUG [pool-1-thread-3] StorageProxy.java (line 705) restricted ranges for query [,] are [[,24], (24,49], (49,6d], (6d,92], (92,b6], (b6,db], (db,ffffffffffffffff], (ffffffffffffffff,]]
2011-02-02 12:54:34,536 DEBUG [pool-1-thread-3] ReadCallback.java (line 58) ReadCallback blocking for 2 responses
2011-02-02 12:54:34,536 DEBUG [pool-1-thread-3] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,24], max_keys=250} from 567@/192.168.0.1
2011-02-02 12:54:34,537 DEBUG [pool-1-thread-3] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,24], max_keys=250} from 567@/192.168.0.2
2011-02-02 12:54:34,537 DEBUG [pool-1-thread-3] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,24], max_keys=250} from 567@/192.168.0.3
2011-02-02 12:54:34,554 DEBUG [RequestResponseStage:1] ResponseVerbHandler.java (line 48) Processing response on a callback from 567@/192.168.0.3
2011-02-02 12:54:44,537 DEBUG [pool-1-thread-3] CassandraServer.java (line 483) ... timed out
2011-02-02 12:54:44,548 DEBUG [pool-1-thread-3] ClientState.java (line 91) logged out: #<User allow_all groups=[]>
2011-02-02 12:54:44,549 DEBUG [pool-1-thread-4] CassandraServer.java (line 445) range_slice
2011-02-02 12:54:44,550 DEBUG [pool-1-thread-4] StorageProxy.java (line 514) RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,], max_keys=250}
2011-02-02 12:54:44,550 DEBUG [pool-1-thread-4] StorageProxy.java (line 705) restricted ranges for query [,] are [[,24], (24,49], (49,6d], (6d,92], (92,b6], (b6,db], (db,ffffffffffffffff], (ffffffffffffffff,]]
2011-02-02 12:54:44,550 DEBUG [pool-1-thread-4] ReadCallback.java (line 58) ReadCallback blocking for 2 responses
2011-02-02 12:54:44,552 DEBUG [pool-1-thread-4] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,24], max_keys=250} from 600@/192.168.0.1
2011-02-02 12:54:44,552 DEBUG [pool-1-thread-4] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,24], max_keys=250} from 600@/192.168.0.2
2011-02-02 12:54:44,552 DEBUG [pool-1-thread-4] StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,24], max_keys=250} from 600@/192.168.0.3
2011-02-02 12:54:44,554 DEBUG [RequestResponseStage:5] ResponseVerbHandler.java (line 48) Processing response on a callback from 600@/192.168.0.1
2011-02-02 12:54:54,550 DEBUG [pool-1-thread-4] CassandraServer.java (line 483) ... timed out
2011-02-02 12:54:54,552 DEBUG [pool-1-thread-4] ClientState.java (line 91) logged out: #<User allow_all groups=[]>
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron Morton updated CASSANDRA-2081:
------------------------------------
Attachment: 2081-logging.patch
adds logging to ResponseVerbHandler to log is the message is received but there is no callback
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
> Attachments: 2081-logging.patch
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989834#comment-12989834 ]
Jonathan Ellis commented on CASSANDRA-2081:
-------------------------------------------
bq. In StorageProxy.fetchRows() line 395 the digestMessage is shared for all non local digest requests.
This is fixed in CASSANDRA-2094, but shouldn't break quorum for RF=3 (since a single digest is all we need)
bq. the log messages above are coming from StorageProxy.getRangeSlice() where the message object is shared for all endpoints
Aha, I didn't notice that. I think this is our bug. Patch attached.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991360#comment-12991360 ]
Aaron Morton commented on CASSANDRA-2081:
-----------------------------------------
Am trying to fix this now.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Assignee: Aaron Morton
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989032#comment-12989032 ]
Aaron Morton commented on CASSANDRA-2081:
-----------------------------------------
I've sort of stumbled onto something similar with an 0.7 install. I need to go home now so cannot dig any deeper and rule out human error, but this is what I have.
5 node 0.7.0 install
1) Load data in using
python stress.py -d jb-cass1,jb-cass2,jb-cass3,jb-cass4,jb-cass5 -o insert -n 1000000 -e QUORUM -t 10 -i 1 -l 3
(use all 5 users, insert 1,000,000 rows with RF 3 and QUORUM and 10 threads, report progress every second)
2) Read back using
python stress.py -d jb-cass2,jb-cass3,jb-cass4,jb-cass5 -o read -n 1000000 -e QUORUM -t 10 -i 1
(note that jb-cass1 is removed from the list)
3) make big bang
Once the read has run a few seconds I ran "reboot -f" on node 1. I expect the read operations to complete, output was
11270,1315,1315,0.00839671943578,9
11631,361,361,0.00746133188792,11
11631,0,0,NaN,12
11631,0,0,NaN,13
11631,0,0,NaN,14
11631,0,0,NaN,15
11631,0,0,NaN,16
11631,0,0,NaN,17
11631,0,0,NaN,18
11631,0,0,NaN,19
Process Reader-10:
Traceback (most recent call last):
File "/vol/apps/python-2.6.4_64/lib/python2.6/multiprocessing/process.py", line 232, in _bootstrap
self.run()
File "stress.py", line 279, in run
r = self.cclient.get_slice(key, parent, p, consistency)
File "/local1/frameworks/cassandra/apache-cassandra-0.7.0-src/contrib/py_stress/cassandra/Cassandra.py", line 432, in get_slice
return self.recv_get_slice()
File "/local1/frameworks/cassandra/apache-cassandra-0.7.0-src/contrib/py_stress/cassandra/Cassandra.py", line 462, in recv_get_slice
raise result.te
All clients died. stress.py is not setting a timeout on the thrift socket, so am guessing this is server side.
I was running DEBUG on all the nodes (but had turned off the line numbers), this is from one. the 114.63 machine is obviously the one I killed.
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,186 StorageService.java (line org.apache.cassandra.service.StorageService) Sorted endpoints are /192.168.114.63,jb08.wetafx.co.nz/192.168.114.67,/192.168.114.64
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,186 QuorumResponseHandler.java (line org.apache.cassandra.service.QuorumResponseHandler) QuorumResponseHandler blocking for 2 responses
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,186 StorageProxy.java (line org.apache.cassandra.service.StorageProxy) strongread reading digest for SliceFromReadCommand(table='Keyspace1', key='30323334343534', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) from 6624@jb08.wetafx.co.nz/192.168.114.67
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,187 StorageProxy.java (line org.apache.cassandra.service.StorageProxy) strongread reading data for SliceFromReadCommand(table='Keyspace1', key='30323334343534', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) from 6623@/192.168.114.63
DEBUG [pool-1-thread-2] 2011-02-01 17:14:08,187 StorageProxy.java (line org.apache.cassandra.service.StorageProxy) strongread reading digest for SliceFromReadCommand(table='Keyspace1', key='30323334343534', column_parent='QueryPath(columnFamilyName='Standard1', superColumnName='null', columnName='null')', start='', finish='', reversed=false, count=5) from 6624@/192.168.114.64
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 0 of 5: 4330:false:34@1296532428248604
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 1 of 5: 4331:false:34@1296532428248637
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 2 of 5: 4332:false:34@1296532428248640
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 3 of 5: 4333:false:34@1296532428248642
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 SliceQueryFilter.java (line org.apache.cassandra.db.filter.SliceQueryFilter) collecting 4 of 5: 4334:false:34@1296532428248656
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 ReadVerbHandler.java (line org.apache.cassandra.db.ReadVerbHandler) digest is 220b82e28c2bb4be869c168243d75f01
DEBUG [ReadStage:19] 2011-02-01 17:14:08,187 ReadVerbHandler.java (line org.apache.cassandra.db.ReadVerbHandler) Read key 30323334343534; sending response to 7D8FA1FD-A2FE-6A54-7BB0-3B129206D1E1@jb08.wetafx.co.nz/192.168.114.67
DEBUG [RequestResponseStage:13] 2011-02-01 17:14:08,188 ResponseVerbHandler.java (line org.apache.cassandra.net.ResponseVerbHandler) Processing response on a callback from 7D8FA1FD-A2FE-6A54-7BB0-3B129206D1E1@jb08.wetafx.co.nz/192.168.114.67
DEBUG [RequestResponseStage:13] 2011-02-01 17:14:08,188 ReadResponseResolver.java (line org.apache.cassandra.service.ReadResponseResolver) Preprocessed digest response
DEBUG [RequestResponseStage:16] 2011-02-01 17:14:08,188 ResponseVerbHandler.java (line org.apache.cassandra.net.ResponseVerbHandler) Processing response on a callback from 7D8FA1FD-A2FE-6A54-7BB0-3B129206D1E1@/192.168.114.64
DEBUG [RequestResponseStage:16] 2011-02-01 17:14:08,188 ReadResponseResolver.java (line org.apache.cassandra.service.ReadResponseResolver) Preprocessed digest response
INFO [ScheduledTasks:1] 2011-02-01 17:14:15,438 Gossiper.java (line org.apache.cassandra.gms.Gossiper) InetAddress /192.168.114.63 is now dead.
DEBUG [ScheduledTasks:1] 2011-02-01 17:14:15,440 MessagingService.java (line org.apache.cassandra.net.MessagingService) Resetting pool for /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:14:17,442 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [pool-1-thread-1] 2011-02-01 17:14:18,183 CassandraServer.java (line org.apache.cassandra.thrift.CassandraServer) ... timed out
DEBUG [pool-1-thread-2] 2011-02-01 17:14:18,189 CassandraServer.java (line org.apache.cassandra.thrift.CassandraServer) ... timed out
DEBUG [pool-1-thread-1] 2011-02-01 17:14:18,232 ClientState.java (line org.apache.cassandra.service.ClientState) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-2] 2011-02-01 17:14:18,232 ClientState.java (line org.apache.cassandra.service.ClientState) logged out: #<User allow_all groups=[]>
DEBUG [ScheduledTasks:1] 2011-02-01 17:14:29,811 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [ScheduledTasks:1] 2011-02-01 17:15:29,814 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:15:53,637 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:08,667 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:23,697 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [ScheduledTasks:1] 2011-02-01 17:16:29,816 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:36,723 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:16:50,751 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:03,775 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:19,807 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [ScheduledTasks:1] 2011-02-01 17:17:29,818 StorageLoadBalancer.java (line org.apache.cassandra.service.StorageLoadBalancer) Disseminating load info ...
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:32,834 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:42,852 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
DEBUG [WRITE-/192.168.114.63] 2011-02-01 17:17:56,880 OutboundTcpConnection.java (line org.apache.cassandra.net.OutboundTcpConnection) attempting to connect to /192.168.114.63
need to leave work now so may not be able to get further into this until tomorrow.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12991399#comment-12991399 ]
Jonathan Ellis commented on CASSANDRA-2081:
-------------------------------------------
bq. it looks like it sends the same message to all endpoints in the local DC
It sends the same message to each entry in the list... which is always going to be a list of size one. So the good news is I don't think it's incorrect, but the bad news is we broke CASSANDRA-1530.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Assignee: Aaron Morton
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-2.txt, 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988867#comment-12988867 ]
Thibaut commented on CASSANDRA-2081:
------------------------------------
My application hangs/blocks forever as I catch all the Hector exceptions and retry when there was an error.
Above log file messages will repeat itself again and again.
> Consistency QUORUM does not work anymore
> ----------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989400#comment-12989400 ]
Thibaut commented on CASSANDRA-2081:
------------------------------------
There is absolutely no load on the cluster, so it's strange that the timeout is being triggered. I will put the other nodes into debug mode tomorrow as well and post the DEBUG output of 0.1, 0.2, and 0.3
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Issue Comment Edited: (CASSANDRA-2081) Consistency QUORUM
does not work anymore (hector:Could not fullfill request on this host)
Posted by "Aaron Morton (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989100#comment-12989100 ]
Aaron Morton edited comment on CASSANDRA-2081 at 2/1/11 8:54 AM:
-----------------------------------------------------------------
Had a read through the code for think I have an idea what my problem was, not sure if it applies to the previous issue and not sure if its a real bug.
o.a.c.service.ReadCallback.response() will only signal the o.a.c.utils.SimpleCondition if the data request has been received. If the signal is not set after rpc_timeout then ReadCallback.get() will raise a j.u.c.TimeoutException() which comes out of the StorageProxy and is caught in CassandraServer and turned into a o.a.c.thrift.TimedOutException.
So if the node that is asked for the data fails to return, the entire request will timeout even if there are enough nodes to serve the request. I think I've seen this discussed before as by design, and the client should just retry in response to the timeout. Is that correct ?
was (Author: amorton):
Had a read through the code for think I have an idea what my problem was, not sure if it applies to the previous issue and not sure if its a real bug.
o.a.c.service.ReadCallback.response() will only signal the o.a.c.utils.SimpleCondition if the data request has been received. If the signal is not set after rpc_timeout then ReadCallback.get() will raise a j.u.c.TimeoutException() which comes out of the StorageProxy and it caught in CassandraServer and turned into a o.a.c.thrift.TimedOutException.
So if the node that is asked for the data fails to return, the entire request will timeout even if there are enough nodes to serve the request. I think I've seen this discussed before as by design, and the client should just retry in response to the timeout. Is that correct ?
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Michel Conrad (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990040#comment-12990040 ]
Michel Conrad commented on CASSANDRA-2081:
------------------------------------------
Jonathan, I tried your patch and it fixes the problem. Thanks.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990162#comment-12990162 ]
Hudson commented on CASSANDRA-2081:
-----------------------------------
Integrated in Cassandra-0.7 #238 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/238/])
add debug log message for missing callback
patch by Aaron Morton and jbellis for CASSANDRA-2081
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Assignee: Aaron Morton
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990156#comment-12990156 ]
Hudson commented on CASSANDRA-2081:
-----------------------------------
Integrated in Cassandra-0.6 #56 (See [https://hudson.apache.org/hudson/job/Cassandra-0.6/56/])
fix range slice ConsistencyLevel > ONE
patch by jbellis; tested by Michel Conrad and Thibaut for CASSANDRA-2081
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Assignee: Aaron Morton
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989818#comment-12989818 ]
Thibaut commented on CASSANDRA-2081:
------------------------------------
Is there an archive of older builds from hudson? I could try out a few versions between 24th january and the 28th to pinpoint the revision that is causing this. This won't take long.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
> Attachments: 2081-logging.patch
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12988885#comment-12988885 ]
Thibaut commented on CASSANDRA-2081:
------------------------------------
RF=3
I will enable the debug log level tomorrow for cassandra, switch back to apache-cassandra-2011-01-28_20-06-01.jar and post you the results.
The last version that I tried that worked was apache-cassandra-2011-01-24_06-01-26.jar. apache-cassandra-2011-01-28_20-06-01.jar doesn't work anymore.
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12990152#comment-12990152 ]
Hudson commented on CASSANDRA-2081:
-----------------------------------
Integrated in Cassandra-0.7 #237 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/237/])
fix range slice ConsistencyLevel > ONE
patch by jbellis; tested by Michel Conrad and Thibaut for CASSANDRA-2081
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.6.10, 0.7.1
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Assignee: Aaron Morton
> Priority: Critical
> Fix For: 0.6.12, 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12989150#comment-12989150 ]
Thibaut commented on CASSANDRA-2081:
------------------------------------
Brandon, I haven't yet run stress test. I can reproduce this error every single time with a single thread accessing my idle cluster.
I also reverted to an older version of hector, but this won't help. As noted before, this error doesn't occur running apache-cassandra-2011-01-24_06-01-26.jar.
Here is the debug output of one of the nodes timing out in my application and not returning an answer:
I only try to access (read/iterator) one single table "table_usersources".
The application runs on node intr1n5 (192.168.0.5) and I added the debug output of intr1n19 (192.168.0.19). The node intr1n18(192.168.0.18) is down and not responding.
Please let me know if you need more information in order to fix this bug. Thanks!
Intr1n19:
INFO [HintedHandoff:1] 2011-02-01 11:47:39,772 HintedHandOffManager.java (line 249) Finished hinted handoff of 0 rows to endpoint /192.168.0.15
INFO [ScheduledTasks:1] 2011-02-01 11:47:40,773 Gossiper.java (line 205) InetAddress /192.168.0.10 is now dead.
DEBUG [ScheduledTasks:1] 2011-02-01 11:47:40,773 MessagingService.java (line 176) Resetting pool for /192.168.0.10
INFO [HintedHandoff:1] 2011-02-01 11:47:40,775 HintedHandOffManager.java (line 192) Started hinted handoff for endpoint /192.168.0.10
INFO [GossipStage:1] 2011-02-01 11:47:40,775 Gossiper.java (line 579) InetAddress /192.168.0.10 is now UP
INFO [HintedHandoff:1] 2011-02-01 11:47:40,775 HintedHandOffManager.java (line 249) Finished hinted handoff of 0 rows to endpoint /192.168.0.10
DEBUG [WRITE-intr1n4/192.168.0.4] 2011-02-01 11:47:41,479 OutboundTcpConnection.java (line 159) attempting to connect to intr1n4/192.168.0.4
DEBUG [WRITE-intr1n20/192.168.0.20] 2011-02-01 11:47:42,729 OutboundTcpConnection.java (line 159) attempting to connect to intr1n20/192.168.0.20
DEBUG [WRITE-intr1n11/192.168.0.11] 2011-02-01 11:47:42,776 OutboundTcpConnection.java (line 159) attempting to connect to intr1n11/192.168.0.11
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:47:46,781 OutboundTcpConnection.java (line 159) attempting to connect to intr1n18/192.168.0.18
DEBUG [WRITE-intr1n15/192.168.0.15] 2011-02-01 11:47:50,786 OutboundTcpConnection.java (line 159) attempting to connect to intr1n15/192.168.0.15
DEBUG [WRITE-intr1n10/192.168.0.10] 2011-02-01 11:47:53,698 OutboundTcpConnection.java (line 159) attempting to connect to intr1n10/192.168.0.10
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:15,413 GCInspector.java (line 135) GC for ParNew: 32 ms, 124741344 reclaimed leaving 880060312 used; max is 3289776128
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:24,853 FileUtils.java (line 48) Deleting LocationInfo-f-79-Index.db
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:24,853 FileUtils.java (line 48) Deleting LocationInfo-f-79-Filter.db
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:24,854 FileUtils.java (line 48) Deleting LocationInfo-f-79-Statistics.db
INFO [ScheduledTasks:1] 2011-02-01 11:48:24,854 SSTable.java (line 147) Deleted /hd2/cassandra_md5/data/system/LocationInfo-f-79
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:48:28,820 OutboundTcpConnection.java (line 159) attempting to connect to intr1n18/192.168.0.18
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,936 CassandraServer.java (line 445) range_slice
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,943 StorageProxy.java (line 514) RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,], max_keys=250}
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,943 StorageProxy.java (line 705) restricted ranges for query [,] are [[,0cc], (0cc,199], (199,266], (266,333], (333,400], (400,4cc], (4cc,599], (599,666], (666,733], (733,7ff], (7ff,8cc], (8cc,999], (999,a66], (a66,b33], (b33,c00], (c00,ccc], (ccc,d99], (d99,e66], (e66,f33], (f33,ffffffffffffffff], (ffffffffffffffff,]]
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,949 ReadCallback.java (line 58) ReadCallback blocking for 2 responses
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,949 StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,0cc], max_keys=250} from 269@/192.168.0.1
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,949 StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,0cc], max_keys=250} from 269@/192.168.0.2
DEBUG [pool-1-thread-1] 2011-02-01 11:48:28,950 StorageProxy.java (line 562) reading RangeSliceCommand{keyspace='table_usersources', column_family='table_usersources_meta', super_column=null, predicate=SlicePredicate(column_names:[java.nio.HeapByteBuffer[pos=76 lim=88 cap=65536]]), range=[,0cc], max_keys=250} from 269@/192.168.0.3
DEBUG [RequestResponseStage:1] 2011-02-01 11:48:28,954 ResponseVerbHandler.java (line 48) Processing response on a callback from 269@/192.168.0.1
DEBUG [ScheduledTasks:1] 2011-02-01 11:48:38,771 StorageLoadBalancer.java (line 349) Disseminating load info ...
DEBUG [pool-1-thread-1] 2011-02-01 11:48:38,950 CassandraServer.java (line 483) ... timed out
DEBUG [pool-1-thread-1] 2011-02-01 11:48:38,957 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:48:44,835 OutboundTcpConnection.java (line 159) attempting to connect to intr1n18/192.168.0.18
DEBUG [pool-1-thread-3] 2011-02-01 11:48:56,275 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-6] 2011-02-01 11:48:56,275 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-7] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-11] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-5] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-2] 2011-02-01 11:48:56,277 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-8] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-10] 2011-02-01 11:48:56,277 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-12] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-15] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-14] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-9] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-16] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-13] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [pool-1-thread-4] 2011-02-01 11:48:56,278 ClientState.java (line 91) logged out: #<User allow_all groups=[]>
DEBUG [WRITE-intr1n18/192.168.0.18] 2011-02-01 11:48:57,845 OutboundTcpConnection.java (line 159) attempting to connect to intr1n18/192.168.0.18
Application:
11:48:25,616 INFO ~ Registering JMX me.prettyprint.cassandra.service:ServiceType=hector,MonitorType=hector
11:48:25,652 INFO ~ get connection for table_lists: consistency: ONE
11:48:25,695 INFO ~ get connection for table_lists: consistency: ONE
11:48:25,825 INFO ~ Downed Host Retry service started with queue size -1 and retry delay 10s
11:48:28,887 ERROR ~ Unable to open transport to intr1n18(192.168.0.18):9160
org.apache.thrift.transport.TTransportException: java.net.NoRouteToHostException: No route to host
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:111)
at me.prettyprint.cassandra.connection.ConcurrentHClientPool.<init>(ConcurrentHClientPool.java:44)
at me.prettyprint.cassandra.connection.HConnectionManager.<init>(HConnectionManager.java:63)
at me.prettyprint.cassandra.service.AbstractCluster.<init>(AbstractCluster.java:62)
at me.prettyprint.cassandra.service.AbstractCluster.<init>(AbstractCluster.java:58)
at me.prettyprint.cassandra.service.ThriftCluster.<init>(ThriftCluster.java:17)
at me.prettyprint.hector.api.factory.HFactory.createCluster(HFactory.java:157)
at me.prettyprint.hector.api.factory.HFactory.getOrCreateCluster(HFactory.java:136)
....
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 23 more
11:48:28,889 ERROR ~ Could not start connection pool for host intr1n18(192.168.0.18):9160
11:48:28,889 INFO ~ Host detected as down was added to retry queue: intr1n18(192.168.0.18):9160
11:48:28,897 INFO ~ get connection for table_usersources: consistency: QUORUM
11:48:38,014 ERROR ~ Unable to open transport to intr1n18(192.168.0.18):9160
org.apache.thrift.transport.TTransportException: java.net.NoRouteToHostException: No route to host
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:111)
at me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.verifyConnection(CassandraHostRetryService.java:116)
at me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.run(CassandraHostRetryService.java:96)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 13 more
11:48:38,019 ERROR ~ Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
11:48:38,020 INFO ~ Downed Host retry status false with host: intr1n18(192.168.0.18):9160
11:48:38,956 ERROR ~ Could not fullfill request on this host CassandraClient<intr1n19:9160-594>
11:48:38,956 ERROR ~ Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:161)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:143)
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:159)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices(KeyspaceServiceImpl.java:165)
at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:67)
at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:63)
at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execute(ThriftRangeSlicesQuery.java:62)
...
Caused by: TimedOutException()
at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12104)
at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:732)
at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:704)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:149)
... 23 more
11:48:48,961 ERROR ~ Could not fullfill request on this host CassandraClient<intr1n17:9160-577>
11:48:48,961 ERROR ~ Exception:
me.prettyprint.hector.api.exceptions.HTimedOutException: TimedOutException()
at me.prettyprint.cassandra.service.ExceptionsTranslatorImpl.translate(ExceptionsTranslatorImpl.java:32)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:161)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:143)
at me.prettyprint.cassandra.service.Operation.executeAndSetResult(Operation.java:101)
at me.prettyprint.cassandra.connection.HConnectionManager.operateWithFailover(HConnectionManager.java:159)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.operateWithFailover(KeyspaceServiceImpl.java:129)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl.getRangeSlices(KeyspaceServiceImpl.java:165)
at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:67)
at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery$1.doInKeyspace(ThriftRangeSlicesQuery.java:63)
at me.prettyprint.cassandra.model.KeyspaceOperationCallback.doInKeyspaceAndMeasure(KeyspaceOperationCallback.java:20)
at me.prettyprint.cassandra.model.ExecutingKeyspace.doExecute(ExecutingKeyspace.java:85)
at me.prettyprint.cassandra.model.thrift.ThriftRangeSlicesQuery.execute(ThriftRangeSlicesQuery.java:62)
...
Caused by: TimedOutException()
at org.apache.cassandra.thrift.Cassandra$get_range_slices_result.read(Cassandra.java:12104)
at org.apache.cassandra.thrift.Cassandra$Client.recv_get_range_slices(Cassandra.java:732)
at org.apache.cassandra.thrift.Cassandra$Client.get_range_slices(Cassandra.java:704)
at me.prettyprint.cassandra.service.KeyspaceServiceImpl$3.execute(KeyspaceServiceImpl.java:149)
... 23 more
11:48:51,025 ERROR ~ Unable to open transport to intr1n18(192.168.0.18):9160
org.apache.thrift.transport.TTransportException: java.net.NoRouteToHostException: No route to host
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at org.apache.thrift.transport.TFramedTransport.open(TFramedTransport.java:81)
at me.prettyprint.cassandra.connection.HThriftClient.open(HThriftClient.java:111)
at me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.verifyConnection(CassandraHostRetryService.java:116)
at me.prettyprint.cassandra.connection.CassandraHostRetryService$RetryRunner.run(CassandraHostRetryService.java:96)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.net.NoRouteToHostException: No route to host
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:333)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:195)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 13 more
11:48:51,025 ERROR ~ Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
11:48:51,025 INFO ~ Downed Host retry status false with host: intr1n18(192.168.0.18):9160
Address Status State Load Owns Token
ffffffffffffffff
192.168.0.1 Up Normal 11.26 GB 5.00% 0cc
192.168.0.2 Up Normal 11.23 GB 5.00% 199
192.168.0.3 Up Normal 11.58 GB 5.00% 266
192.168.0.4 Up Normal 6.77 GB 5.00% 333
192.168.0.5 Up Normal 6.86 GB 5.00% 400
192.168.0.6 Up Normal 6.81 GB 5.00% 4cc
192.168.0.7 Up Normal 6.88 GB 5.00% 599
192.168.0.8 Up Normal 6.84 GB 5.00% 666
192.168.0.9 Up Normal 6.52 GB 5.00% 733
192.168.0.10 Up Normal 5.17 GB 5.00% 7ff
192.168.0.11 Up Normal 6.75 GB 5.00% 8cc
192.168.0.12 Up Normal 7.06 GB 5.00% 999
192.168.0.13 Up Normal 7.27 GB 5.00% a66
192.168.0.14 Up Normal 7.71 GB 5.00% b33
192.168.0.15 Up Normal 7.46 GB 5.00% c00
192.168.0.16 Up Normal 6.94 GB 5.00% ccc
192.168.0.17 Up Normal 6.45 GB 5.00% d99
192.168.0.18 Down Normal ? 5.00% e66
192.168.0.19 Up Normal 6.26 GB 5.00% f33
192.168.0.20 Up Normal 6.33 GB 5.00% ffffffffffffffff
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (CASSANDRA-2081) Consistency QUORUM does not work
anymore (hector:Could not fullfill request on this host)
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2081:
--------------------------------------
Attachment: 2081.txt
> Consistency QUORUM does not work anymore (hector:Could not fullfill request on this host)
> -----------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2081
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2081
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: linux, hector + cassandra
> Reporter: Thibaut
> Priority: Blocker
> Fix For: 0.7.1
>
> Attachments: 2081-logging.patch, 2081.txt
>
>
> I'm using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25.
> Using consistency level Quorum won't work anymore (tested it on read). Consisteny level ONE still works though
> I have tried this with one dead node in my cluster.
> If I restart cassandra with an older svn revision (apache-cassandra-2011-01-28_20-06-01.jar), I can access the cluster with consistency level QUORUM again, while still using apache-cassandra-2011-01-28_20-06-01.jar and hector 7.0.25 in my application.
> 11/01/31 19:54:38 ERROR connection.CassandraHostRetryService: Downed intr1n18(192.168.0.18):9160 host still appears to be down: Unable to open transport to intr1n18(192.168.0.18):9160 , java.net.NoRouteToHostException: No route to host
> 11/01/31 19:54:38 INFO connection.CassandraHostRetryService: Downed Host retry status false with host: intr1n18(192.168.0.18):9160
> 11/01/31 19:54:45 ERROR connection.HConnectionManager: Could not fullfill request on this host CassandraClient<intr1n11:9160-483>
> intr1n11 is marked as up however and I can also access the node through the cassandra cli.
> 192.168.0.1 Up Normal 8.02 GB 5.00% 0cc
> 192.168.0.2 Up Normal 7.96 GB 5.00% 199
> 192.168.0.3 Up Normal 8.24 GB 5.00% 266
> 192.168.0.4 Up Normal 4.94 GB 5.00% 333
> 192.168.0.5 Up Normal 5.02 GB 5.00% 400
> 192.168.0.6 Up Normal 5 GB 5.00% 4cc
> 192.168.0.7 Up Normal 5.1 GB 5.00% 599
> 192.168.0.8 Up Normal 5.07 GB 5.00% 666
> 192.168.0.9 Up Normal 4.78 GB 5.00% 733
> 192.168.0.10 Up Normal 4.34 GB 5.00% 7ff
> 192.168.0.11 Up Normal 5.01 GB 5.00% 8cc
> 192.168.0.12 Up Normal 5.31 GB 5.00% 999
> 192.168.0.13 Up Normal 5.56 GB 5.00% a66
> 192.168.0.14 Up Normal 5.82 GB 5.00% b33
> 192.168.0.15 Up Normal 5.57 GB 5.00% c00
> 192.168.0.16 Up Normal 5.03 GB 5.00% ccc
> 192.168.0.17 Up Normal 4.77 GB 5.00% d99
> 192.168.0.18 Down Normal ? 5.00% e66
> 192.168.0.19 Up Normal 4.78 GB 5.00% f33
> 192.168.0.20 Up Normal 4.83 GB 5.00% ffffffffffffffff
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira