You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Benjamin Coverston (JIRA)" <ji...@apache.org> on 2012/06/15 17:44:43 UTC
[jira] [Created] (CASSANDRA-4345) New (bootstrapping) Supplant
Healthy Nodes
Benjamin Coverston created CASSANDRA-4345:
---------------------------------------------
Summary: New (bootstrapping) Supplant Healthy Nodes
Key: CASSANDRA-4345
URL: https://issues.apache.org/jira/browse/CASSANDRA-4345
Project: Cassandra
Issue Type: Improvement
Affects Versions: 1.0.9
Reporter: Benjamin Coverston
Copied a config from an existing node and fired up a new node, which happily inserted itself at token 0 of a running ring. The surprising and worrisome part is that EVERY node started throwing:
java.lang.NullPointerException
ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,948 Cassandra.java (line 3041) Internal error processing get_slice
java.lang.NullPointerException
ERROR [RPC-Thread:205427] 2012-06-14 19:16:31,949 Cassandra.java (line 3041) Internal error processing get_slice
java.lang.NullPointerException
ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,952 Cassandra.java (line 3041) Internal error processing get_slice
java.lang.NullPointerException
---
Resulting in:
INFO [GossipStage:1] 2012-06-14 18:24:37,472 Gossiper.java (line 838) Node /192.168.88.48 is now part of the cluster
INFO [GossipStage:1] 2012-06-14 18:24:37,473 Gossiper.java (line 804) InetAddress /192.168.88.48 is now UP
INFO [GossipStage:1] 2012-06-14 18:24:37,473 StorageService.java (line 1008) Nodes /192.168.88.48 and /192.168.88.70 have the same token 0. /192.168.88
.48 is the new owner
WARN [GossipStage:1] 2012-06-14 18:24:37,474 TokenMetadata.java (line 135) Token 0 changing ownership from /192.168.88.70 to /192.168.88.48
INFO [GossipStage:1] 2012-06-14 18:24:37,475 ColumnFamilyStore.java (line 705) Enqueuing flush of Memtable-LocationInfo@961917618(20/25 serialized/live
bytes, 1 ops)
INFO [FlushWriter:1272] 2012-06-14 18:24:37,475 Memtable.java (line 246) Writing Memtable-LocationInfo@961917618(20/25 serialized/live bytes, 1 ops)
INFO [FlushWriter:1272] 2012-06-14 18:24:37,492 Memtable.java (line 283) Completed flushing /cass_ssd/system/LocationInfo-hc-23-Data.db (74 bytes)
ERROR [RPC-Thread:200943] 2012-06-14 18:24:38,007 Cassandra.java (line 3041) Internal error processing get_slice
java.lang.NullPointerException
at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:104)
at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122)
at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:93)
at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100)
at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:2002)
at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1996)
at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:604)
at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:564)
at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:128)
at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:283)
at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:365)
at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:326)
at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033)
at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
at org.apache.cassandra.thrift.CustomTHsHaServer$Invocation.run(CustomTHsHaServer.java:105)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4345) New (bootstrapping) Supplant
Healthy Nodes
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13296076#comment-13296076 ]
Jonathan Ellis commented on CASSANDRA-4345:
-------------------------------------------
Looks like we have a check for "token exists in the ring" for move, but not for bootstrap.
> New (bootstrapping) Supplant Healthy Nodes
> ------------------------------------------
>
> Key: CASSANDRA-4345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4345
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.0.9
> Reporter: Benjamin Coverston
> Labels: datastax_qa
>
> Copied a config from an existing node and fired up a new node, which happily inserted itself at token 0 of a running ring. The surprising and worrisome part is that EVERY node started throwing:
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,948 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205427] 2012-06-14 19:16:31,949 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,952 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ---
> Resulting in:
> INFO [GossipStage:1] 2012-06-14 18:24:37,472 Gossiper.java (line 838) Node /192.168.88.48 is now part of the cluster
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 Gossiper.java (line 804) InetAddress /192.168.88.48 is now UP
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 StorageService.java (line 1008) Nodes /192.168.88.48 and /192.168.88.70 have the same token 0. /192.168.88
> .48 is the new owner
> WARN [GossipStage:1] 2012-06-14 18:24:37,474 TokenMetadata.java (line 135) Token 0 changing ownership from /192.168.88.70 to /192.168.88.48
> INFO [GossipStage:1] 2012-06-14 18:24:37,475 ColumnFamilyStore.java (line 705) Enqueuing flush of Memtable-LocationInfo@961917618(20/25 serialized/live
> bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,475 Memtable.java (line 246) Writing Memtable-LocationInfo@961917618(20/25 serialized/live bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,492 Memtable.java (line 283) Completed flushing /cass_ssd/system/LocationInfo-hc-23-Data.db (74 bytes)
> ERROR [RPC-Thread:200943] 2012-06-14 18:24:38,007 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:104)
> at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122)
> at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:93)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:2002)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1996)
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:604)
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:564)
> at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:128)
> at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:283)
> at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:365)
> at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:326)
> at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033)
> at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
> at org.apache.cassandra.thrift.CustomTHsHaServer$Invocation.run(CustomTHsHaServer.java:105)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4345) New (bootstrapping) Supplant
Healthy Nodes
Posted by "Joaquin Casares (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295827#comment-13295827 ]
Joaquin Casares commented on CASSANDRA-4345:
--------------------------------------------
I was able to reproduce this error as well.
> New (bootstrapping) Supplant Healthy Nodes
> ------------------------------------------
>
> Key: CASSANDRA-4345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4345
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.0.9
> Reporter: Benjamin Coverston
> Labels: datastax_qa
>
> Copied a config from an existing node and fired up a new node, which happily inserted itself at token 0 of a running ring. The surprising and worrisome part is that EVERY node started throwing:
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,948 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205427] 2012-06-14 19:16:31,949 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,952 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ---
> Resulting in:
> INFO [GossipStage:1] 2012-06-14 18:24:37,472 Gossiper.java (line 838) Node /192.168.88.48 is now part of the cluster
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 Gossiper.java (line 804) InetAddress /192.168.88.48 is now UP
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 StorageService.java (line 1008) Nodes /192.168.88.48 and /192.168.88.70 have the same token 0. /192.168.88
> .48 is the new owner
> WARN [GossipStage:1] 2012-06-14 18:24:37,474 TokenMetadata.java (line 135) Token 0 changing ownership from /192.168.88.70 to /192.168.88.48
> INFO [GossipStage:1] 2012-06-14 18:24:37,475 ColumnFamilyStore.java (line 705) Enqueuing flush of Memtable-LocationInfo@961917618(20/25 serialized/live
> bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,475 Memtable.java (line 246) Writing Memtable-LocationInfo@961917618(20/25 serialized/live bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,492 Memtable.java (line 283) Completed flushing /cass_ssd/system/LocationInfo-hc-23-Data.db (74 bytes)
> ERROR [RPC-Thread:200943] 2012-06-14 18:24:38,007 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:104)
> at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122)
> at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:93)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:2002)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1996)
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:604)
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:564)
> at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:128)
> at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:283)
> at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:365)
> at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:326)
> at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033)
> at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
> at org.apache.cassandra.thrift.CustomTHsHaServer$Invocation.run(CustomTHsHaServer.java:105)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4345) New (bootstrapping)
Supplant Healthy Nodes
Posted by "Joaquin Casares (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295827#comment-13295827 ]
Joaquin Casares edited comment on CASSANDRA-4345 at 6/15/12 7:08 PM:
---------------------------------------------------------------------
I was able to reproduce the original node getting kicked out as well while auto_bootstrap set to true on the new, clean machine.
was (Author: j.casares):
I was able to reproduce this error as well.
> New (bootstrapping) Supplant Healthy Nodes
> ------------------------------------------
>
> Key: CASSANDRA-4345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4345
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.0.9
> Reporter: Benjamin Coverston
> Labels: datastax_qa
>
> Copied a config from an existing node and fired up a new node, which happily inserted itself at token 0 of a running ring. The surprising and worrisome part is that EVERY node started throwing:
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,948 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205427] 2012-06-14 19:16:31,949 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,952 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ---
> Resulting in:
> INFO [GossipStage:1] 2012-06-14 18:24:37,472 Gossiper.java (line 838) Node /192.168.88.48 is now part of the cluster
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 Gossiper.java (line 804) InetAddress /192.168.88.48 is now UP
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 StorageService.java (line 1008) Nodes /192.168.88.48 and /192.168.88.70 have the same token 0. /192.168.88
> .48 is the new owner
> WARN [GossipStage:1] 2012-06-14 18:24:37,474 TokenMetadata.java (line 135) Token 0 changing ownership from /192.168.88.70 to /192.168.88.48
> INFO [GossipStage:1] 2012-06-14 18:24:37,475 ColumnFamilyStore.java (line 705) Enqueuing flush of Memtable-LocationInfo@961917618(20/25 serialized/live
> bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,475 Memtable.java (line 246) Writing Memtable-LocationInfo@961917618(20/25 serialized/live bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,492 Memtable.java (line 283) Completed flushing /cass_ssd/system/LocationInfo-hc-23-Data.db (74 bytes)
> ERROR [RPC-Thread:200943] 2012-06-14 18:24:38,007 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:104)
> at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122)
> at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:93)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:2002)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1996)
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:604)
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:564)
> at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:128)
> at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:283)
> at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:365)
> at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:326)
> at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033)
> at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
> at org.apache.cassandra.thrift.CustomTHsHaServer$Invocation.run(CustomTHsHaServer.java:105)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4345) New (bootstrapping) Supplant
Healthy Nodes
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-4345.
---------------------------------------
Resolution: Duplicate
Split into CASSANDRA-4348 and CASSANDRA-4349.
> New (bootstrapping) Supplant Healthy Nodes
> ------------------------------------------
>
> Key: CASSANDRA-4345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4345
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.0.9
> Reporter: Benjamin Coverston
> Labels: datastax_qa
>
> Copied a config from an existing node and fired up a new node, which happily inserted itself at token 0 of a running ring. The surprising and worrisome part is that EVERY node started throwing:
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,948 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205427] 2012-06-14 19:16:31,949 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,952 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ---
> Resulting in:
> INFO [GossipStage:1] 2012-06-14 18:24:37,472 Gossiper.java (line 838) Node /192.168.88.48 is now part of the cluster
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 Gossiper.java (line 804) InetAddress /192.168.88.48 is now UP
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 StorageService.java (line 1008) Nodes /192.168.88.48 and /192.168.88.70 have the same token 0. /192.168.88
> .48 is the new owner
> WARN [GossipStage:1] 2012-06-14 18:24:37,474 TokenMetadata.java (line 135) Token 0 changing ownership from /192.168.88.70 to /192.168.88.48
> INFO [GossipStage:1] 2012-06-14 18:24:37,475 ColumnFamilyStore.java (line 705) Enqueuing flush of Memtable-LocationInfo@961917618(20/25 serialized/live
> bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,475 Memtable.java (line 246) Writing Memtable-LocationInfo@961917618(20/25 serialized/live bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,492 Memtable.java (line 283) Completed flushing /cass_ssd/system/LocationInfo-hc-23-Data.db (74 bytes)
> ERROR [RPC-Thread:200943] 2012-06-14 18:24:38,007 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:104)
> at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122)
> at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:93)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:2002)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1996)
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:604)
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:564)
> at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:128)
> at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:283)
> at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:365)
> at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:326)
> at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033)
> at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
> at org.apache.cassandra.thrift.CustomTHsHaServer$Invocation.run(CustomTHsHaServer.java:105)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4345) New (bootstrapping) Supplant
Healthy Nodes
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295737#comment-13295737 ]
Jonathan Ellis commented on CASSANDRA-4345:
-------------------------------------------
While we could improve the error message, PFS shouldn't just Make Something Up when given an unknown node to replicate to.
> New (bootstrapping) Supplant Healthy Nodes
> ------------------------------------------
>
> Key: CASSANDRA-4345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4345
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.0.9
> Reporter: Benjamin Coverston
>
> Copied a config from an existing node and fired up a new node, which happily inserted itself at token 0 of a running ring. The surprising and worrisome part is that EVERY node started throwing:
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,948 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205427] 2012-06-14 19:16:31,949 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,952 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ---
> Resulting in:
> INFO [GossipStage:1] 2012-06-14 18:24:37,472 Gossiper.java (line 838) Node /192.168.88.48 is now part of the cluster
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 Gossiper.java (line 804) InetAddress /192.168.88.48 is now UP
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 StorageService.java (line 1008) Nodes /192.168.88.48 and /192.168.88.70 have the same token 0. /192.168.88
> .48 is the new owner
> WARN [GossipStage:1] 2012-06-14 18:24:37,474 TokenMetadata.java (line 135) Token 0 changing ownership from /192.168.88.70 to /192.168.88.48
> INFO [GossipStage:1] 2012-06-14 18:24:37,475 ColumnFamilyStore.java (line 705) Enqueuing flush of Memtable-LocationInfo@961917618(20/25 serialized/live
> bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,475 Memtable.java (line 246) Writing Memtable-LocationInfo@961917618(20/25 serialized/live bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,492 Memtable.java (line 283) Completed flushing /cass_ssd/system/LocationInfo-hc-23-Data.db (74 bytes)
> ERROR [RPC-Thread:200943] 2012-06-14 18:24:38,007 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:104)
> at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122)
> at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:93)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:2002)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1996)
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:604)
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:564)
> at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:128)
> at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:283)
> at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:365)
> at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:326)
> at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033)
> at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
> at org.apache.cassandra.thrift.CustomTHsHaServer$Invocation.run(CustomTHsHaServer.java:105)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4345) New (bootstrapping) Supplant
Healthy Nodes
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13295863#comment-13295863 ]
Brandon Williams commented on CASSANDRA-4345:
---------------------------------------------
If you mean the NPE, that is no surprise.
> New (bootstrapping) Supplant Healthy Nodes
> ------------------------------------------
>
> Key: CASSANDRA-4345
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4345
> Project: Cassandra
> Issue Type: Improvement
> Affects Versions: 1.0.9
> Reporter: Benjamin Coverston
> Labels: datastax_qa
>
> Copied a config from an existing node and fired up a new node, which happily inserted itself at token 0 of a running ring. The surprising and worrisome part is that EVERY node started throwing:
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,948 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205427] 2012-06-14 19:16:31,949 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ERROR [RPC-Thread:205459] 2012-06-14 19:16:31,952 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> ---
> Resulting in:
> INFO [GossipStage:1] 2012-06-14 18:24:37,472 Gossiper.java (line 838) Node /192.168.88.48 is now part of the cluster
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 Gossiper.java (line 804) InetAddress /192.168.88.48 is now UP
> INFO [GossipStage:1] 2012-06-14 18:24:37,473 StorageService.java (line 1008) Nodes /192.168.88.48 and /192.168.88.70 have the same token 0. /192.168.88
> .48 is the new owner
> WARN [GossipStage:1] 2012-06-14 18:24:37,474 TokenMetadata.java (line 135) Token 0 changing ownership from /192.168.88.70 to /192.168.88.48
> INFO [GossipStage:1] 2012-06-14 18:24:37,475 ColumnFamilyStore.java (line 705) Enqueuing flush of Memtable-LocationInfo@961917618(20/25 serialized/live
> bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,475 Memtable.java (line 246) Writing Memtable-LocationInfo@961917618(20/25 serialized/live bytes, 1 ops)
> INFO [FlushWriter:1272] 2012-06-14 18:24:37,492 Memtable.java (line 283) Completed flushing /cass_ssd/system/LocationInfo-hc-23-Data.db (74 bytes)
> ERROR [RPC-Thread:200943] 2012-06-14 18:24:38,007 Cassandra.java (line 3041) Internal error processing get_slice
> java.lang.NullPointerException
> at org.apache.cassandra.locator.PropertyFileSnitch.getDatacenter(PropertyFileSnitch.java:104)
> at org.apache.cassandra.locator.DynamicEndpointSnitch.getDatacenter(DynamicEndpointSnitch.java:122)
> at org.apache.cassandra.locator.NetworkTopologyStrategy.calculateNaturalEndpoints(NetworkTopologyStrategy.java:93)
> at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:100)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:2002)
> at org.apache.cassandra.service.StorageService.getLiveNaturalEndpoints(StorageService.java:1996)
> at org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:604)
> at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:564)
> at org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:128)
> at org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:283)
> at org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:365)
> at org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:326)
> at org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033)
> at org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
> at org.apache.thrift.server.TNonblockingServer$FrameBuffer.invoke(TNonblockingServer.java:631)
> at org.apache.cassandra.thrift.CustomTHsHaServer$Invocation.run(CustomTHsHaServer.java:105)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira