You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Carlos Scheidecker <na...@gmail.com> on 2015/07/22 23:45:27 UTC
Upgraded to Cassandra 2.2.0 nodes not seeing each other
All,
I have a 4 node Cassandra system running on 4 Ubuntu boxes. After updating
to Cassandra 2.2.0 and keeping the same cassandra.yaml file, the nodes
cannot see each other.
When I do a nodetool status it only reports as being up the machine where I
had issue the command.
In other words, all the machines cannot communicate to each other any
longer. Nodetool status behave the same on each machine.
I am trying to debug that, hopefully only something on the configuration
that has changed.
Any ideas?
Thanks.
C.
Re: Upgraded to Cassandra 2.2.0 nodes not seeing each other
Posted by Carlos Scheidecker <na...@gmail.com>.
I agreed Michael. I was generating stuff for it again, Looks like they had
the SSL stack changed. I came from 2.1.6 to 2.2.0. Thanks.
On Wed, Jul 22, 2015 at 5:45 PM, Michael Shuler <mi...@pbandjelly.org>
wrote:
> What version of Cassandra did you upgrade to 2.2.0 *from*?
>
> This would help with looking at config differences, changelogs, etc.
>
> It seems you have some pretty clear SSL connection errors, according to
> the logs, which at least helps with seeing why the nodes can't talk to each
> other. I'm not terribly familiar with using SSL with Cassandra, but it
> seems clear that you have an incorrect server_encryption_options:
> cipher_suites: configuration.
>
> --
> Kind regards,
> Michael
>
> On 07/22/2015 06:33 PM, Carlos Scheidecker wrote:
>
>> Thanks for the reply, Michael!
>>
>> Yes, I did followed the upgrade nodes.
>>
>> I am running Ubuntu Ubuntu 14.04.2 LTS on all and
>> kernel 3.13.0-57-generic on all.
>>
>> I have 4 machines: .31, .32, .33 and .34. If I run nodetool status from
>> .34 I now see all the others as DN the same happens if I log in from the
>> others:
>>
>> DN 192.168.1.31 ? 256 ?
>> 1f8000f5-026c-42c7-8189-cf19fbede566 RAC1
>> DN 192.168.1.32 ? 256 ?
>> 12478d45-3d5e-418b-a0dc-dba6d4307af3 RAC1
>> DN 192.168.1.33 ? 256 ?
>> 994172b3-cd36-4558-a4b8-054cfac027f3 RAC1
>> UN 192.168.1.34 1.7 MB 256 ?
>> b66be1f3-bb4a-49bd-9835-5c8ee2a71e5c RAC1
>>
>> If I do a netstat -atn from .34 I get:
>>
>> tcp 0 0 127.0.1.1:53 <http://127.0.1.1:53>
>> 0.0.0.0:* LISTEN
>> tcp 0 0 0.0.0.0:22 <http://0.0.0.0:22>
>> 0.0.0.0:* LISTEN
>> tcp 0 0 127.0.0.1:631 <http://127.0.0.1:631>
>> 0.0.0.0:* LISTEN
>> tcp 0 0 192.168.1.34:7001 <http://192.168.1.34:7001>
>> 0.0.0.0:* LISTEN
>> tcp 0 0 127.0.0.1:7199 <http://127.0.0.1:7199>
>> 0.0.0.0:* LISTEN
>> tcp 0 0 192.168.1.34:9160 <http://192.168.1.34:9160>
>> 0.0.0.0:* LISTEN
>> tcp 0 0 127.0.0.1:59441 <http://127.0.0.1:59441>
>> 0.0.0.0:* LISTEN
>> tcp 0 0 192.168.1.34:52951 <http://192.168.1.34:52951>
>> 192.168.1.31:7001 <http://192.168.1.31:7001> ESTABLISHED
>>
>> On the logs I now have the following errors (/var/log/syslog.log):
>>
>> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:48,764 SSLFactory.java:163 - Filtering out
>>
>> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
>> as it isnt supported by the socket
>> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:48,764 OutboundTcpConnection.java:229 - error
>> processing a message intended for /192.168.1.31 <http://192.168.1.31>
>> java.lang.NullPointerException: null
>> at
>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
>> ~[guava-16.0.jar:na]
>> at
>>
>> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.run(OutboundTcpConnection.java:218)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:48,764 OutboundTcpConnection.java:316 - error writing
>> to /192.168.1.31 <http://192.168.1.31>
>> java.lang.NullPointerException: null
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.run(OutboundTcpConnection.java:219)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> WARN [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
>> 2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out
>>
>> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
>> as it isnt supported by the socket
>> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out
>>
>> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
>> as it isnt supported by the socket
>> ERROR [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
>> 2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error
>> processing a message intended for /192.168.1.33 <http://192.168.1.33>
>> java.lang.NullPointerException: null
>> at
>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
>> ~[guava-16.0.jar:na]
>> at
>>
>> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.run(OutboundTcpConnection.java:218)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error
>> processing a message intended for /192.168.1.31 <http://192.168.1.31>
>> java.lang.NullPointerException: null
>> at
>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
>> ~[guava-16.0.jar:na]
>> at
>>
>> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.run(OutboundTcpConnection.java:218)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:50,763 OutboundTcpConnection.java:316 - error writing
>> to /192.168.1.31 <http://192.168.1.31>
>> java.lang.NullPointerException: null
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.run(OutboundTcpConnection.java:219)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:51,766 SSLFactory.java:163 - Filtering out
>>
>> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
>> as it isnt supported by the socket
>> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:51,767 OutboundTcpConnection.java:229 - error
>> processing a message intended for /192.168.1.31 <http://192.168.1.31>
>> java.lang.NullPointerException: null
>> at
>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
>> ~[guava-16.0.jar:na]
>> at
>>
>> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.run(OutboundTcpConnection.java:218)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> ERROR [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
>> 2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing
>> to /192.168.1.33 <http://192.168.1.33>
>> java.lang.NullPointerException: null
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.run(OutboundTcpConnection.java:219)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing
>> to /192.168.1.31 <http://192.168.1.31>
>> java.lang.NullPointerException: null
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.run(OutboundTcpConnection.java:219)
>> [apache-cassandra-2.2.0.jar:2.2.0]
>> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:53,767 SSLFactory.java:163 - Filtering out
>>
>> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
>> as it isnt supported by the socket
>> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
>> 2015-07-22 17:29:53,767 OutboundTcpConnection.java:229 - error
>> processing a message intended for /192.168.1.31 <http://192.168.1.31>
>> java.lang.NullPointerException: null
>> at
>> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
>> ~[guava-16.0.jar:na]
>> at
>>
>> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>> at
>> org.apache.cassandra.net
>> .OutboundTcpConnection.run(OutboundTcpConnection.java:218)
>>
>> ~[apache-cassandra-2.2.0.jar:2.2.0]
>>
>>
>> On my cassandra.yaml I have the end point snitch:
>>
>> endpoint_snitch: GossipingPropertyFileSnitch
>>
>> On the cassandra-rackdc.properties
>> # Licensed to the Apache Software Foundation (ASF) under one
>> # or more contributor license agreements. See the NOTICE file
>> # distributed with this work for additional information
>> # regarding copyright ownership. The ASF licenses this file
>> # to you under the Apache License, Version 2.0 (the
>> # "License"); you may not use this file except in compliance
>> # with the License. You may obtain a copy of the License at
>> #
>> # http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>>
>> # These properties are used with GossipingPropertyFileSnitch and will
>> # indicate the rack and dc for this node
>> dc=DC1
>> rack=RAC1
>>
>> # Add a suffix to a datacenter name. Used by the Ec2Snitch and
>> Ec2MultiRegionSnitch
>> # to append a string to the EC2 region name.
>> #dc_suffix=
>>
>> # Uncomment the following line to make this snitch prefer the internal
>> ip when possible, as the Ec2MultiRegionSnitch does.
>> # prefer_local=true
>>
>> On my cassandra-topology.properties
>>
>> # Licensed to the Apache Software Foundation (ASF) under one
>> # or more contributor license agreements. See the NOTICE file
>> # distributed with this work for additional information
>> # regarding copyright ownership. The ASF licenses this file
>> # to you under the Apache License, Version 2.0 (the
>> # "License"); you may not use this file except in compliance
>> # with the License. You may obtain a copy of the License at
>> #
>> # http://www.apache.org/licenses/LICENSE-2.0
>> #
>> # Unless required by applicable law or agreed to in writing, software
>> # distributed under the License is distributed on an "AS IS" BASIS,
>> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
>> # See the License for the specific language governing permissions and
>> # limitations under the License.
>>
>> # Cassandra Node IP=Data Center:Rack
>> 192.168.1.31=DC1:RAC1
>> 192.168.1.32=DC1:RAC1
>> 192.168.1.33=DC1:RAC1
>> 192.168.1.34=DC1:RAC1
>>
>> # default for unknown nodes
>> default=DC1:RAC1
>>
>>
>>
>>
>>
>> On Wed, Jul 22, 2015 at 5:23 PM, Michael Shuler <michael@pbandjelly.org
>> <ma...@pbandjelly.org>> wrote:
>>
>> On 07/22/2015 04:45 PM, Carlos Scheidecker wrote:
>>
>> I have a 4 node Cassandra system running on 4 Ubuntu boxes. After
>> updating to Cassandra 2.2.0 and keeping the same cassandra.yaml
>> file,
>> the nodes cannot see each other.
>>
>>
>> What version did you upgrade from?
>>
>> Usually, when upgrading, it is probably a good idea to start with
>> the default cassandra.yaml from the new version (2.2.0 in your case)
>> and edit the necessary items from your old version; i.e. num_tokens,
>> initial_token, listen_address, broadcast_address, etc. You are
>> perhaps missing some sort of default setting that 2.2.0 is looking
>> for?
>>
>> When I do a nodetool status it only reports as being up the
>> machine
>> where I had issue the command.
>>
>> In other words, all the machines cannot communicate to each
>> other any
>> longer. Nodetool status behave the same on each machine.
>>
>> I am trying to debug that, hopefully only something on the
>> configuration
>> that has changed.
>>
>> Any ideas?
>>
>>
>> Anything helpful in the system.log on each of your nodes?
>>
>> Did you follow all the upgrade notes from your previous release to
>> 2.2.0?
>>
>> https://github.com/apache/cassandra/blob/cassandra-2.2.0/NEWS.txt
>>
>> --
>> Kind regards,
>> Michael
>>
>>
>>
>
Re: Upgraded to Cassandra 2.2.0 nodes not seeing each other
Posted by Michael Shuler <mi...@pbandjelly.org>.
What version of Cassandra did you upgrade to 2.2.0 *from*?
This would help with looking at config differences, changelogs, etc.
It seems you have some pretty clear SSL connection errors, according to
the logs, which at least helps with seeing why the nodes can't talk to
each other. I'm not terribly familiar with using SSL with Cassandra, but
it seems clear that you have an incorrect server_encryption_options:
cipher_suites: configuration.
--
Kind regards,
Michael
On 07/22/2015 06:33 PM, Carlos Scheidecker wrote:
> Thanks for the reply, Michael!
>
> Yes, I did followed the upgrade nodes.
>
> I am running Ubuntu Ubuntu 14.04.2 LTS on all and
> kernel 3.13.0-57-generic on all.
>
> I have 4 machines: .31, .32, .33 and .34. If I run nodetool status from
> .34 I now see all the others as DN the same happens if I log in from the
> others:
>
> DN 192.168.1.31 ? 256 ?
> 1f8000f5-026c-42c7-8189-cf19fbede566 RAC1
> DN 192.168.1.32 ? 256 ?
> 12478d45-3d5e-418b-a0dc-dba6d4307af3 RAC1
> DN 192.168.1.33 ? 256 ?
> 994172b3-cd36-4558-a4b8-054cfac027f3 RAC1
> UN 192.168.1.34 1.7 MB 256 ?
> b66be1f3-bb4a-49bd-9835-5c8ee2a71e5c RAC1
>
> If I do a netstat -atn from .34 I get:
>
> tcp 0 0 127.0.1.1:53 <http://127.0.1.1:53>
> 0.0.0.0:* LISTEN
> tcp 0 0 0.0.0.0:22 <http://0.0.0.0:22>
> 0.0.0.0:* LISTEN
> tcp 0 0 127.0.0.1:631 <http://127.0.0.1:631>
> 0.0.0.0:* LISTEN
> tcp 0 0 192.168.1.34:7001 <http://192.168.1.34:7001>
> 0.0.0.0:* LISTEN
> tcp 0 0 127.0.0.1:7199 <http://127.0.0.1:7199>
> 0.0.0.0:* LISTEN
> tcp 0 0 192.168.1.34:9160 <http://192.168.1.34:9160>
> 0.0.0.0:* LISTEN
> tcp 0 0 127.0.0.1:59441 <http://127.0.0.1:59441>
> 0.0.0.0:* LISTEN
> tcp 0 0 192.168.1.34:52951 <http://192.168.1.34:52951>
> 192.168.1.31:7001 <http://192.168.1.31:7001> ESTABLISHED
>
> On the logs I now have the following errors (/var/log/syslog.log):
>
> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:48,764 SSLFactory.java:163 - Filtering out
> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
> as it isnt supported by the socket
> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:48,764 OutboundTcpConnection.java:229 - error
> processing a message intended for /192.168.1.31 <http://192.168.1.31>
> java.lang.NullPointerException: null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:48,764 OutboundTcpConnection.java:316 - error writing
> to /192.168.1.31 <http://192.168.1.31>
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
> [apache-cassandra-2.2.0.jar:2.2.0]
> WARN [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
> 2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out
> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
> as it isnt supported by the socket
> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:49,764 SSLFactory.java:163 - Filtering out
> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
> as it isnt supported by the socket
> ERROR [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
> 2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error
> processing a message intended for /192.168.1.33 <http://192.168.1.33>
> java.lang.NullPointerException: null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:49,764 OutboundTcpConnection.java:229 - error
> processing a message intended for /192.168.1.31 <http://192.168.1.31>
> java.lang.NullPointerException: null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:50,763 OutboundTcpConnection.java:316 - error writing
> to /192.168.1.31 <http://192.168.1.31>
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
> [apache-cassandra-2.2.0.jar:2.2.0]
> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:51,766 SSLFactory.java:163 - Filtering out
> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
> as it isnt supported by the socket
> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:51,767 OutboundTcpConnection.java:229 - error
> processing a message intended for /192.168.1.31 <http://192.168.1.31>
> java.lang.NullPointerException: null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> ERROR [MessagingService-Outgoing-/192.168.1.33 <http://192.168.1.33>]
> 2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing
> to /192.168.1.33 <http://192.168.1.33>
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
> [apache-cassandra-2.2.0.jar:2.2.0]
> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:52,764 OutboundTcpConnection.java:316 - error writing
> to /192.168.1.31 <http://192.168.1.31>
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
> [apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
> [apache-cassandra-2.2.0.jar:2.2.0]
> WARN [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:53,767 SSLFactory.java:163 - Filtering out
> TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
> as it isnt supported by the socket
> ERROR [MessagingService-Outgoing-/192.168.1.31 <http://192.168.1.31>]
> 2015-07-22 17:29:53,767 OutboundTcpConnection.java:229 - error
> processing a message intended for /192.168.1.31 <http://192.168.1.31>
> java.lang.NullPointerException: null
> at
> com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213) ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
> ~[apache-cassandra-2.2.0.jar:2.2.0]
>
>
> On my cassandra.yaml I have the end point snitch:
>
> endpoint_snitch: GossipingPropertyFileSnitch
>
> On the cassandra-rackdc.properties
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements. See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership. The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
>
> # These properties are used with GossipingPropertyFileSnitch and will
> # indicate the rack and dc for this node
> dc=DC1
> rack=RAC1
>
> # Add a suffix to a datacenter name. Used by the Ec2Snitch and
> Ec2MultiRegionSnitch
> # to append a string to the EC2 region name.
> #dc_suffix=
>
> # Uncomment the following line to make this snitch prefer the internal
> ip when possible, as the Ec2MultiRegionSnitch does.
> # prefer_local=true
>
> On my cassandra-topology.properties
>
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements. See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership. The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
>
> # Cassandra Node IP=Data Center:Rack
> 192.168.1.31=DC1:RAC1
> 192.168.1.32=DC1:RAC1
> 192.168.1.33=DC1:RAC1
> 192.168.1.34=DC1:RAC1
>
> # default for unknown nodes
> default=DC1:RAC1
>
>
>
>
>
> On Wed, Jul 22, 2015 at 5:23 PM, Michael Shuler <michael@pbandjelly.org
> <ma...@pbandjelly.org>> wrote:
>
> On 07/22/2015 04:45 PM, Carlos Scheidecker wrote:
>
> I have a 4 node Cassandra system running on 4 Ubuntu boxes. After
> updating to Cassandra 2.2.0 and keeping the same cassandra.yaml
> file,
> the nodes cannot see each other.
>
>
> What version did you upgrade from?
>
> Usually, when upgrading, it is probably a good idea to start with
> the default cassandra.yaml from the new version (2.2.0 in your case)
> and edit the necessary items from your old version; i.e. num_tokens,
> initial_token, listen_address, broadcast_address, etc. You are
> perhaps missing some sort of default setting that 2.2.0 is looking for?
>
> When I do a nodetool status it only reports as being up the machine
> where I had issue the command.
>
> In other words, all the machines cannot communicate to each
> other any
> longer. Nodetool status behave the same on each machine.
>
> I am trying to debug that, hopefully only something on the
> configuration
> that has changed.
>
> Any ideas?
>
>
> Anything helpful in the system.log on each of your nodes?
>
> Did you follow all the upgrade notes from your previous release to
> 2.2.0?
>
> https://github.com/apache/cassandra/blob/cassandra-2.2.0/NEWS.txt
>
> --
> Kind regards,
> Michael
>
>
Re: Upgraded to Cassandra 2.2.0 nodes not seeing each other
Posted by Carlos Scheidecker <na...@gmail.com>.
Thanks for the reply, Michael!
Yes, I did followed the upgrade nodes.
I am running Ubuntu Ubuntu 14.04.2 LTS on all and kernel 3.13.0-57-generic
on all.
I have 4 machines: .31, .32, .33 and .34. If I run nodetool status from .34
I now see all the others as DN the same happens if I log in from the others:
DN 192.168.1.31 ? 256 ?
1f8000f5-026c-42c7-8189-cf19fbede566 RAC1
DN 192.168.1.32 ? 256 ?
12478d45-3d5e-418b-a0dc-dba6d4307af3 RAC1
DN 192.168.1.33 ? 256 ?
994172b3-cd36-4558-a4b8-054cfac027f3 RAC1
UN 192.168.1.34 1.7 MB 256 ?
b66be1f3-bb4a-49bd-9835-5c8ee2a71e5c RAC1
If I do a netstat -atn from .34 I get:
tcp 0 0 127.0.1.1:53 0.0.0.0:* LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.34:7001 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:7199 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.34:9160 0.0.0.0:* LISTEN
tcp 0 0 127.0.0.1:59441 0.0.0.0:* LISTEN
tcp 0 0 192.168.1.34:52951 192.168.1.31:7001
ESTABLISHED
On the logs I now have the following errors (/var/log/syslog.log):
WARN [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:48,764
SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:48,764
OutboundTcpConnection.java:229 - error processing a message intended for /
192.168.1.31
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:48,764
OutboundTcpConnection.java:316 - error writing to /192.168.1.31
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
WARN [MessagingService-Outgoing-/192.168.1.33] 2015-07-22 17:29:49,764
SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
WARN [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:49,764
SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.33] 2015-07-22 17:29:49,764
OutboundTcpConnection.java:229 - error processing a message intended for /
192.168.1.33
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:49,764
OutboundTcpConnection.java:229 - error processing a message intended for /
192.168.1.31
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:50,763
OutboundTcpConnection.java:316 - error writing to /192.168.1.31
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
WARN [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:51,766
SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:51,767
OutboundTcpConnection.java:229 - error processing a message intended for /
192.168.1.31
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.33] 2015-07-22 17:29:52,764
OutboundTcpConnection.java:316 - error writing to /192.168.1.33
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:52,764
OutboundTcpConnection.java:316 - error writing to /192.168.1.31
java.lang.NullPointerException: null
at
org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:323)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:285)
[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:219)
[apache-cassandra-2.2.0.jar:2.2.0]
WARN [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:53,767
SSLFactory.java:163 - Filtering out
TLS_DHE_RSA_WITH_AES_256_CBC_SHA,TLS_RSA_WITH_AES_256_CBC_SHA,TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA
as it isnt supported by the socket
ERROR [MessagingService-Outgoing-/192.168.1.31] 2015-07-22 17:29:53,767
OutboundTcpConnection.java:229 - error processing a message intended for /
192.168.1.31
java.lang.NullPointerException: null
at
com.google.common.base.Preconditions.checkNotNull(Preconditions.java:213)
~[guava-16.0.jar:na]
at
org.apache.cassandra.io.util.BufferedDataOutputStreamPlus.<init>(BufferedDataOutputStreamPlus.java:74)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.connect(OutboundTcpConnection.java:404)
~[apache-cassandra-2.2.0.jar:2.2.0]
at
org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
~[apache-cassandra-2.2.0.jar:2.2.0]
On my cassandra.yaml I have the end point snitch:
endpoint_snitch: GossipingPropertyFileSnitch
On the cassandra-rackdc.properties
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# These properties are used with GossipingPropertyFileSnitch and will
# indicate the rack and dc for this node
dc=DC1
rack=RAC1
# Add a suffix to a datacenter name. Used by the Ec2Snitch and
Ec2MultiRegionSnitch
# to append a string to the EC2 region name.
#dc_suffix=
# Uncomment the following line to make this snitch prefer the internal ip
when possible, as the Ec2MultiRegionSnitch does.
# prefer_local=true
On my cassandra-topology.properties
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership. The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Cassandra Node IP=Data Center:Rack
192.168.1.31=DC1:RAC1
192.168.1.32=DC1:RAC1
192.168.1.33=DC1:RAC1
192.168.1.34=DC1:RAC1
# default for unknown nodes
default=DC1:RAC1
On Wed, Jul 22, 2015 at 5:23 PM, Michael Shuler <mi...@pbandjelly.org>
wrote:
> On 07/22/2015 04:45 PM, Carlos Scheidecker wrote:
>
>> I have a 4 node Cassandra system running on 4 Ubuntu boxes. After
>> updating to Cassandra 2.2.0 and keeping the same cassandra.yaml file,
>> the nodes cannot see each other.
>>
>
> What version did you upgrade from?
>
> Usually, when upgrading, it is probably a good idea to start with the
> default cassandra.yaml from the new version (2.2.0 in your case) and edit
> the necessary items from your old version; i.e. num_tokens, initial_token,
> listen_address, broadcast_address, etc. You are perhaps missing some sort
> of default setting that 2.2.0 is looking for?
>
> When I do a nodetool status it only reports as being up the machine
>> where I had issue the command.
>>
>> In other words, all the machines cannot communicate to each other any
>> longer. Nodetool status behave the same on each machine.
>>
>> I am trying to debug that, hopefully only something on the configuration
>> that has changed.
>>
>> Any ideas?
>>
>
> Anything helpful in the system.log on each of your nodes?
>
> Did you follow all the upgrade notes from your previous release to 2.2.0?
>
> https://github.com/apache/cassandra/blob/cassandra-2.2.0/NEWS.txt
>
> --
> Kind regards,
> Michael
>
Re: Upgraded to Cassandra 2.2.0 nodes not seeing each other
Posted by Michael Shuler <mi...@pbandjelly.org>.
On 07/22/2015 04:45 PM, Carlos Scheidecker wrote:
> I have a 4 node Cassandra system running on 4 Ubuntu boxes. After
> updating to Cassandra 2.2.0 and keeping the same cassandra.yaml file,
> the nodes cannot see each other.
What version did you upgrade from?
Usually, when upgrading, it is probably a good idea to start with the
default cassandra.yaml from the new version (2.2.0 in your case) and
edit the necessary items from your old version; i.e. num_tokens,
initial_token, listen_address, broadcast_address, etc. You are perhaps
missing some sort of default setting that 2.2.0 is looking for?
> When I do a nodetool status it only reports as being up the machine
> where I had issue the command.
>
> In other words, all the machines cannot communicate to each other any
> longer. Nodetool status behave the same on each machine.
>
> I am trying to debug that, hopefully only something on the configuration
> that has changed.
>
> Any ideas?
Anything helpful in the system.log on each of your nodes?
Did you follow all the upgrade notes from your previous release to 2.2.0?
https://github.com/apache/cassandra/blob/cassandra-2.2.0/NEWS.txt
--
Kind regards,
Michael