You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Abhishek Verma <ve...@uber.com> on 2016/10/10 23:34:55 UTC
Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
datacenter fails because of streaming errors
Hi Cassandra users,
We are trying to upgrade our Cassandra version from 2.2.5 to 3.0.8 (running
on Mesos, but that's besides the point). We have two datacenters, so in
order to preserve our data, we are trying to upgrade one datacenter at a
time.
Initially both DCs (dc1 and dc2) are running 2.2.5. The idea is to tear
down dc1 completely (delete all the data in it), bring it up with 3.0.8,
let data replicate from dc2 to dc1, and then tear down dc2, bring it up
with 3.0.8 and replicate data from dc1.
I am able to reproduce the problem on bare metal clusters running on 3
nodes. I am using Oracle's server-jre-8u74-linux-x64 JRE.
*Node A*: Downloaded 2.2.5-bin.tar.gz, changed the seeds to include its own
IP address, changed listen_address and rpc_address to its own IP and
changed endpoint_snitch to GossipingPropertyFileSnitch. I
changed conf/cassandra-rackdc.properties to
dc=dc2
rack=rack2
This node started up fine and is UN in nodetool status in dc2.
I used CQL shell to create a table and insert 3 rows:
verma@xxxxx:~/apache-cassandra-2.2.5$ bin/cqlsh $HOSTNAME
Connected to Test Cluster at xxxxx:9042.
[cqlsh 5.0.1 | Cassandra 2.2.5 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> desc tmp
CREATE KEYSPACE tmp WITH replication = {'class': 'NetworkTopologyStrategy',
'dc1': '1', 'dc2': '1'} AND durable_writes = true;
CREATE TABLE tmp.map (
key text PRIMARY KEY,
value text
)...;
cqlsh> select * from tmp.map;
key | value
-----+-------
k1 | v1
k3 | v3
k2 | v2
*Node B:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include itself
and node A, changed listen_address and rpc_address to its own IP, changed
endpoint_snitch to GossipingPropertyFileSnitch. I did not change
conf/cassandra-rackdc.properties and its contents are
dc=dc1
rack=rack1
In the logs, I see:
INFO [main] 2016-10-10 22:42:42,850 MessagingService.java:557 - Starting
Messaging Service on /10.164.32.29:7000 (eth0)
INFO [main] 2016-10-10 22:42:42,864 StorageService.java:784 - This node
will not auto bootstrap because it is configured to be a seed node.
So I start a third node:
*Node C:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include node A
and node B, changed listen_address and rpc_address to its own IP, changed
endpoint_snitch to GossipingPropertyFileSnitch. I did not change
conf/cassandra-rackdc.properties.
Now, nodetool status shows:
verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UJ <Node C IP> 87.81 KB 256 ?
9064832d-ed5c-4c42-ad5a-f754b52b670c rack1
UN <Node B IP> 107.72 KB 256 100.0%
28b1043f-115b-46a5-b6b6-8609829cde76 rack1
Datacenter: dc2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN <Node A IP> 73.2 KB 256 100.0%
09cc542c-2299-45a5-a4d1-159c239ded37 rack2
Nodetool describe cluster shows:
verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool describecluster
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
c2a2bb4f-7d31-3fb8-a216-00b41a643650: [<Node B IP>, <Node C IP>]
9770e3c5-3135-32e2-b761-65a0f6d8824e: [<Node A IP>]
Note that there are two schema versions and they don't match.
I see the following in the system.log:
INFO [InternalResponseStage:1] 2016-10-10 22:48:36,055
ColumnFamilyStore.java:390 - Initializing system_auth.roles
INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
waiting for schema information to complete
INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
schema complete, ready to bootstrap
INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
waiting for pending range calculation
INFO [main] 2016-10-10 22:48:36,317 StorageService.java:1149 - JOINING:
calculation complete, ready to bootstrap
INFO [main] 2016-10-10 22:48:36,319 StorageService.java:1149 - JOINING:
getting bootstrap token
INFO [main] 2016-10-10 22:48:36,357 StorageService.java:1149 - JOINING:
sleeping 30000 ms for pending range setup
INFO [main] 2016-10-10 22:49:06,358 StorageService.java:1149 - JOINING:
Starting to bootstrap...
INFO [main] 2016-10-10 22:49:06,494 StreamResultFuture.java:87 - [Stream
#bfb5e470-8f3b-11e6-b69a-1b451159408e] Executing streaming plan for
Bootstrap
INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,495
StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
Starting streaming to /<Node A IP>
INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,495
StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
Starting streaming to /<Node B IP>
INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,500
StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
ID#0] Beginning stream session with /<Node B IP>
INFO [STREAM-IN-/<Node B IP>] 2016-10-10 22:49:06,590
StreamResultFuture.java:183 - [Stream
#bfb5e470-8f3b-11e6-b69a-1b451159408e] Session with /<Node B IP> is complete
INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,635
StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
ID#0] Beginning stream session with /<Node A IP>
ERROR [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
StreamSession.java:528 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
Streaming error occurred
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_102]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
~[na:1.8.0_102]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_102]
at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_102]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
~[na:1.8.0_102]
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
~[na:1.8.0_102]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
~[na:1.8.0_102]
at
java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
~[na:1.8.0_102]
at
org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:287)
~[apache-cassandra-3.0.8.jar:3.0.8]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
INFO [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
StreamResultFuture.java:183 - [Stream
#bfb5e470-8f3b-11e6-b69a-1b451159408e] Session with /<Node A IP> is complete
WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
StreamResultFuture.java:210 - [Stream
#bfb5e470-8f3b-11e6-b69a-1b451159408e] Stream failed
WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
StorageService.java:1208 - Error during bootstrap.
org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.0.8.jar:3.0.8]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:534)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:305)
[apache-cassandra-3.0.8.jar:3.0.8]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
ERROR [main] 2016-10-10 22:49:06,641 StorageService.java:1218 - Error while
waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException:
org.apache.cassandra.streaming.StreamException: Stream failed
at
com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
~[guava-18.0.jar:na]
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1213)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:889)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:663)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:528)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:339)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
[apache-cassandra-3.0.8.jar:3.0.8]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-3.0.8.jar:3.0.8]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-18.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-18.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:534)
~[apache-cassandra-3.0.8.jar:3.0.8]
at
org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:305)
~[apache-cassandra-3.0.8.jar:3.0.8]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
WARN [main] 2016-10-10 22:49:06,646 StorageService.java:944 - Some data
streaming failed. Use nodetool to check bootstrap state and resume. For
more, see `nodetool help bootstrap`. IN_PROGRESS
INFO [main] 2016-10-10 22:49:06,647 CassandraDaemon.java:644 - Waiting for
gossip to settle before accepting client requests...
INFO [main] 2016-10-10 22:49:14,648 CassandraDaemon.java:675 - No gossip
backlog; proceeding
INFO [main] 2016-10-10 22:49:14,694 NativeTransportService.java:70 - Netty
using native Epoll event loop
INFO [main] 2016-10-10 22:49:14,726 Server.java:159 - Using Netty Version:
[netty-buffer=netty-buffer-4.0.23.Final.208198c,
netty-codec=netty-codec-4.0.23.Final.208198c,
netty-codec-http=netty-codec-http-4.0.23.Final.208198c,
netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c,
netty-common=netty-common-4.0.23.Final.208198c,
netty-handler=netty-handler-4.0.23.Final.208198c,
netty-transport=netty-transport-4.0.23.Final.208198c,
netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c,
netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c,
netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c]
INFO [main] 2016-10-10 22:49:14,726 Server.java:160 - Starting listening
for CQL clients on /<Node C IP>:9042 (unencrypted)...
INFO [main] 2016-10-10 22:49:14,748 CassandraDaemon.java:477 - Not
starting RPC server as requested. Use JMX
(StorageService->startRPCServer()) or nodetool (enablethrift) to start it
I tried resuming bootstrap but it fails with the same streaming errors:
verma@<Node C>:~/apache-cassandra-3.0.8$ bin/nodetool bootstrap resume
Resuming bootstrap
[2016-10-10 23:15:11,816] session with /<Node B IP> complete (progress: 0%)
[2016-10-10 23:15:11,939] session with /<Node A IP> complete (progress: 0%)
[2016-10-10 23:15:11,940] Stream failed
and I see the same error in the system.log:
StreamSession.java:528 - [Stream #64b73a20-8f3f-11e6-b69a-1b451159408e]
Streaming error occurred
java.io.IOException: Connection reset by peer
...
Does Cassandra support upgrading from 2.2.5 to 3.0.8 in this way? Am I
missing something?
Thanks for your time.
-Abhishek.
Re: Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
datacenter fails because of streaming errors
Posted by Abhishek Verma <ve...@uber.com>.
Thanks Jonathan, Utkarsh and Jeff.
We will try to find a way for our Mesos framework to support upgrading the
nodes in-place.
On Mon, Oct 10, 2016 at 5:11 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> During the upgrade you'll want to avoid the following operations that
> result in data streaming:
>
> 1. Bootstrapping nodes
> 2. Decomissioning nodes
> 3. Repair
>
>
> On Mon, Oct 10, 2016 at 5:00 PM Jeff Jirsa <je...@crowdstrike.com>
> wrote:
>
>>
>>
>> No need to cc dev@, user@ is the right list for this question.
>>
>>
>>
>> As Jon mentioned, you can’t stream (bootstrap/rebuild/repair) across
>> major versions, so don’t try to destroy the cluster – just upgrade in
>> place. It IS a good idea to do one DC at a time, but an in-place upgrade is
>> pretty straightforward – flush, drain, stop Cassandra, replace binaries,
>> start Cassandra, run nodetool upgradesstables -a.
>>
>>
>>
>> Note that you can run nodetool upgradesstables on more than one node at a
>> time if you can tolerate the hit to your read latencies.
>>
>>
>>
>> It IS common, I imagine, for there to be schema mismatches temporarily
>> while you have a mixed version cluster – this isn’t necessarily a huge
>> problem, but do try to get to 3.0.8 as quickly as possible once you start,
>> and if you can avoid administrative tasks (such as those that will change
>> the schema) during the process, that’s generally advisable.
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> *From: *Abhishek Verma <ve...@uber.com>
>> *Reply-To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
>> *Date: *Monday, October 10, 2016 at 4:34 PM
>> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>, "
>> dev@cassandra.apache.org" <de...@cassandra.apache.org>
>> *Subject: *Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
>> datacenter fails because of streaming errors
>>
>>
>>
>> Hi Cassandra users,
>>
>>
>>
>> We are trying to upgrade our Cassandra version from 2.2.5 to 3.0.8
>> (running on Mesos, but that's besides the point). We have two datacenters,
>> so in order to preserve our data, we are trying to upgrade one datacenter
>> at a time.
>>
>>
>>
>> Initially both DCs (dc1 and dc2) are running 2.2.5. The idea is to tear
>> down dc1 completely (delete all the data in it), bring it up with 3.0.8,
>> let data replicate from dc2 to dc1, and then tear down dc2, bring it up
>> with 3.0.8 and replicate data from dc1.
>>
>>
>>
>> I am able to reproduce the problem on bare metal clusters running on 3
>> nodes. I am using Oracle's server-jre-8u74-linux-x64 JRE.
>>
>>
>>
>> *Node A*: Downloaded 2.2.5-bin.tar.gz, changed the seeds to include its
>> own IP address, changed listen_address and rpc_address to its own IP and
>> changed endpoint_snitch to GossipingPropertyFileSnitch. I
>> changed conf/cassandra-rackdc.properties to
>>
>> dc=dc2
>>
>> rack=rack2
>>
>> This node started up fine and is UN in nodetool status in dc2.
>>
>>
>>
>> I used CQL shell to create a table and insert 3 rows:
>>
>> verma@xxxxx:~/apache-cassandra-2.2.5$ bin/cqlsh $HOSTNAME
>>
>> Connected to Test Cluster at xxxxx:9042.
>>
>> [cqlsh 5.0.1 | Cassandra 2.2.5 | CQL spec 3.3.1 | Native protocol v4]
>>
>> Use HELP for help.
>>
>> cqlsh> desc tmp
>>
>>
>>
>> CREATE KEYSPACE tmp WITH replication = {'class':
>> 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1'} AND durable_writes =
>> true;
>>
>>
>>
>> CREATE TABLE tmp.map (
>>
>> key text PRIMARY KEY,
>>
>> value text
>>
>> )...;
>>
>> cqlsh> select * from tmp.map;
>>
>>
>>
>> key | value
>>
>> -----+-------
>>
>> k1 | v1
>>
>> k3 | v3
>>
>> k2 | v2
>>
>>
>>
>>
>>
>> *Node B:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include
>> itself and node A, changed listen_address and rpc_address to its own IP,
>> changed endpoint_snitch to GossipingPropertyFileSnitch. I did not change
>> conf/cassandra-rackdc.properties and its contents are
>>
>> dc=dc1
>>
>> rack=rack1
>>
>>
>>
>> In the logs, I see:
>>
>> INFO [main] 2016-10-10 22:42:42,850 MessagingService.java:557 - Starting
>> Messaging Service on /10.164.32.29:7000
>> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.164.32.29-3A7000&d=DQMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=aDxgr_M4K2TNU2MqOBLrREEaFoUbWmFpnP4l67CROag&s=t4xNW-nnA0EVZWnt-PSAoq4vx6Lep2ObgkG3ZuUUg7c&e=>
>> (eth0)
>>
>> INFO [main] 2016-10-10 22:42:42,864 StorageService.java:784 - This node
>> will not auto bootstrap because it is configured to be a seed node.
>>
>>
>>
>> So I start a third node:
>>
>> *Node C:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include node
>> A and node B, changed listen_address and rpc_address to its own IP, changed
>> endpoint_snitch to GossipingPropertyFileSnitch. I did not change
>> conf/cassandra-rackdc.properties.
>>
>> Now, nodetool status shows:
>>
>>
>>
>> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool status
>>
>> Datacenter: dc1
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UJ <Node C IP> 87.81 KB 256 ?
>> 9064832d-ed5c-4c42-ad5a-f754b52b670c rack1
>>
>> UN <Node B IP> 107.72 KB 256 100.0%
>> 28b1043f-115b-46a5-b6b6-8609829cde76 rack1
>>
>> Datacenter: dc2
>>
>> ===============
>>
>> Status=Up/Down
>>
>> |/ State=Normal/Leaving/Joining/Moving
>>
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>>
>> UN <Node A IP> 73.2 KB 256 100.0%
>> 09cc542c-2299-45a5-a4d1-159c239ded37 rack2
>>
>>
>>
>> Nodetool describe cluster shows:
>>
>> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool describecluster
>>
>> Cluster Information:
>>
>> Name: Test Cluster
>>
>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>>
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>>
>> Schema versions:
>>
>> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [<Node B IP>, <Node C IP>]
>>
>>
>>
>> 9770e3c5-3135-32e2-b761-65a0f6d8824e: [<Node A IP>]
>>
>>
>>
>> Note that there are two schema versions and they don't match.
>>
>>
>>
>> I see the following in the system.log:
>>
>>
>>
>> INFO [InternalResponseStage:1] 2016-10-10 22:48:36,055
>> ColumnFamilyStore.java:390 - Initializing system_auth.roles
>>
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> waiting for schema information to complete
>>
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> schema complete, ready to bootstrap
>>
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> waiting for pending range calculation
>>
>> INFO [main] 2016-10-10 22:48:36,317 StorageService.java:1149 - JOINING:
>> calculation complete, ready to bootstrap
>>
>> INFO [main] 2016-10-10 22:48:36,319 StorageService.java:1149 - JOINING:
>> getting bootstrap token
>>
>> INFO [main] 2016-10-10 22:48:36,357 StorageService.java:1149 - JOINING:
>> sleeping 30000 ms for pending range setup
>>
>> INFO [main] 2016-10-10 22:49:06,358 StorageService.java:1149 - JOINING:
>> Starting to bootstrap...
>>
>> INFO [main] 2016-10-10 22:49:06,494 StreamResultFuture.java:87 - [Stream
>> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Executing streaming plan for
>> Bootstrap
>>
>> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,495
>> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Starting streaming to /<Node A IP>
>>
>> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,495
>> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Starting streaming to /<Node B IP>
>>
>> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,500
>> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
>> ID#0] Beginning stream session with /<Node B IP>
>>
>> INFO [STREAM-IN-/<Node B IP>] 2016-10-10 22:49:06,590
>> StreamResultFuture.java:183 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Session with /<Node B IP> is complete
>>
>> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,635
>> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
>> ID#0] Beginning stream session with /<Node A IP>
>>
>> ERROR [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
>> StreamSession.java:528 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Streaming error occurred
>>
>> java.io.IOException: Connection reset by peer
>>
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_102]
>>
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> ~[na:1.8.0_102]
>>
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> ~[na:1.8.0_102]
>>
>> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_102]
>>
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>> ~[na:1.8.0_102]
>>
>> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
>> ~[na:1.8.0_102]
>>
>> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>> ~[na:1.8.0_102]
>>
>> at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>> ~[na:1.8.0_102]
>>
>> at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:287)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>>
>> INFO [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
>> StreamResultFuture.java:183 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Session with /<Node A IP> is complete
>>
>> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
>> StreamResultFuture.java:210 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Stream failed
>>
>> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
>> StorageService.java:1208 - Error during bootstrap.
>>
>> org.apache.cassandra.streaming.StreamException: Stream failed
>>
>> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.
>> onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.8.jar:
>> 3.0.8]
>>
>> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>> [guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.MoreExecutors$
>> DirectExecutor.execute(MoreExecutors.java:457) [guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.ExecutionList.
>> executeListener(ExecutionList.java:156) [guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.ExecutionList.
>> execute(ExecutionList.java:145) [guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.AbstractFuture.
>> setException(AbstractFuture.java:202) [guava-18.0.jar:na]
>>
>> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamResultFuture.
>> handleSessionComplete(StreamResultFuture.java:187)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamSession.
>> closeSession(StreamSession.java:429) [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamSession.
>> onError(StreamSession.java:534) [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:305)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>>
>> ERROR [main] 2016-10-10 22:49:06,641 StorageService.java:1218 - Error
>> while waiting on bootstrap to complete. Bootstrap will have to be restarted.
>>
>> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
>> Stream failed
>>
>> at com.google.common.util.concurrent.AbstractFuture$
>> Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.AbstractFuture$
>> Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
>> ~[guava-18.0.jar:na]
>>
>> at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1213)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:889)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:663)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:528)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:339)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>>
>> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>>
>> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.
>> onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.8.jar:
>> 3.0.8]
>>
>> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>> ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.MoreExecutors$
>> DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.ExecutionList.
>> executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.ExecutionList.
>> execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
>>
>> at com.google.common.util.concurrent.AbstractFuture.
>> setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
>>
>> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamResultFuture.
>> handleSessionComplete(StreamResultFuture.java:187)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamSession.
>> closeSession(StreamSession.java:429) ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.StreamSession.
>> onError(StreamSession.java:534) ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:305)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>>
>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
>>
>> WARN [main] 2016-10-10 22:49:06,646 StorageService.java:944 - Some data
>> streaming failed. Use nodetool to check bootstrap state and resume. For
>> more, see `nodetool help bootstrap`. IN_PROGRESS
>>
>> INFO [main] 2016-10-10 22:49:06,647 CassandraDaemon.java:644 - Waiting
>> for gossip to settle before accepting client requests...
>>
>> INFO [main] 2016-10-10 22:49:14,648 CassandraDaemon.java:675 - No gossip
>> backlog; proceeding
>>
>> INFO [main] 2016-10-10 22:49:14,694 NativeTransportService.java:70 -
>> Netty using native Epoll event loop
>>
>> INFO [main] 2016-10-10 22:49:14,726 Server.java:159 - Using Netty
>> Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c,
>> netty-codec=netty-codec-4.0.23.Final.208198c,
>> netty-codec-http=netty-codec-http-4.0.23.Final.208198c,
>> netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c,
>> netty-common=netty-common-4.0.23.Final.208198c,
>> netty-handler=netty-handler-4.0.23.Final.208198c, netty-transport=netty-
>> transport-4.0.23.Final.208198c, netty-transport-rxtx=netty-
>> transport-rxtx-4.0.23.Final.208198c, netty-transport-sctp=netty-
>> transport-sctp-4.0.23.Final.208198c, netty-transport-udt=netty-
>> transport-udt-4.0.23.Final.208198c]
>>
>> INFO [main] 2016-10-10 22:49:14,726 Server.java:160 - Starting listening
>> for CQL clients on /<Node C IP>:9042 (unencrypted)...
>>
>> INFO [main] 2016-10-10 22:49:14,748 CassandraDaemon.java:477 - Not
>> starting RPC server as requested. Use JMX (StorageService->startRPCServer())
>> or nodetool (enablethrift) to start it
>>
>>
>>
>> I tried resuming bootstrap but it fails with the same streaming errors:
>>
>>
>>
>> verma@<Node C>:~/apache-cassandra-3.0.8$ bin/nodetool bootstrap resume
>>
>> Resuming bootstrap
>>
>> [2016-10-10 23:15:11,816] session with /<Node B IP> complete (progress:
>> 0%)
>>
>> [2016-10-10 23:15:11,939] session with /<Node A IP> complete (progress:
>> 0%)
>>
>> [2016-10-10 23:15:11,940] Stream failed
>>
>>
>>
>> and I see the same error in the system.log:
>>
>>
>>
>> StreamSession.java:528 - [Stream #64b73a20-8f3f-11e6-b69a-1b451159408e]
>> Streaming error occurred
>>
>> java.io.IOException: Connection reset by peer
>>
>> ...
>>
>>
>>
>> Does Cassandra support upgrading from 2.2.5 to 3.0.8 in this way? Am I
>> missing something?
>>
>>
>>
>> Thanks for your time.
>>
>> -Abhishek.
>> ____________________________________________________________________
>> CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential
>> and may be legally privileged. If you are not the intended recipient, do
>> not disclose, copy, distribute, or use this email or any attachments. If
>> you have received this in error please let the sender know and then delete
>> the email and all attachments.
>>
>
Re: Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
datacenter fails because of streaming errors
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
During the upgrade you'll want to avoid the following operations that
result in data streaming:
1. Bootstrapping nodes
2. Decomissioning nodes
3. Repair
On Mon, Oct 10, 2016 at 5:00 PM Jeff Jirsa <je...@crowdstrike.com>
wrote:
>
>
> No need to cc dev@, user@ is the right list for this question.
>
>
>
> As Jon mentioned, you can’t stream (bootstrap/rebuild/repair) across major
> versions, so don’t try to destroy the cluster – just upgrade in place. It
> IS a good idea to do one DC at a time, but an in-place upgrade is pretty
> straightforward – flush, drain, stop Cassandra, replace binaries, start
> Cassandra, run nodetool upgradesstables -a.
>
>
>
> Note that you can run nodetool upgradesstables on more than one node at a
> time if you can tolerate the hit to your read latencies.
>
>
>
> It IS common, I imagine, for there to be schema mismatches temporarily
> while you have a mixed version cluster – this isn’t necessarily a huge
> problem, but do try to get to 3.0.8 as quickly as possible once you start,
> and if you can avoid administrative tasks (such as those that will change
> the schema) during the process, that’s generally advisable.
>
>
>
>
>
>
>
>
>
> *From: *Abhishek Verma <ve...@uber.com>
> *Reply-To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>
> *Date: *Monday, October 10, 2016 at 4:34 PM
> *To: *"user@cassandra.apache.org" <us...@cassandra.apache.org>, "
> dev@cassandra.apache.org" <de...@cassandra.apache.org>
> *Subject: *Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
> datacenter fails because of streaming errors
>
>
>
> Hi Cassandra users,
>
>
>
> We are trying to upgrade our Cassandra version from 2.2.5 to 3.0.8
> (running on Mesos, but that's besides the point). We have two datacenters,
> so in order to preserve our data, we are trying to upgrade one datacenter
> at a time.
>
>
>
> Initially both DCs (dc1 and dc2) are running 2.2.5. The idea is to tear
> down dc1 completely (delete all the data in it), bring it up with 3.0.8,
> let data replicate from dc2 to dc1, and then tear down dc2, bring it up
> with 3.0.8 and replicate data from dc1.
>
>
>
> I am able to reproduce the problem on bare metal clusters running on 3
> nodes. I am using Oracle's server-jre-8u74-linux-x64 JRE.
>
>
>
> *Node A*: Downloaded 2.2.5-bin.tar.gz, changed the seeds to include its
> own IP address, changed listen_address and rpc_address to its own IP and
> changed endpoint_snitch to GossipingPropertyFileSnitch. I
> changed conf/cassandra-rackdc.properties to
>
> dc=dc2
>
> rack=rack2
>
> This node started up fine and is UN in nodetool status in dc2.
>
>
>
> I used CQL shell to create a table and insert 3 rows:
>
> verma@xxxxx:~/apache-cassandra-2.2.5$ bin/cqlsh $HOSTNAME
>
> Connected to Test Cluster at xxxxx:9042.
>
> [cqlsh 5.0.1 | Cassandra 2.2.5 | CQL spec 3.3.1 | Native protocol v4]
>
> Use HELP for help.
>
> cqlsh> desc tmp
>
>
>
> CREATE KEYSPACE tmp WITH replication = {'class':
> 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1'} AND durable_writes =
> true;
>
>
>
> CREATE TABLE tmp.map (
>
> key text PRIMARY KEY,
>
> value text
>
> )...;
>
> cqlsh> select * from tmp.map;
>
>
>
> key | value
>
> -----+-------
>
> k1 | v1
>
> k3 | v3
>
> k2 | v2
>
>
>
>
>
> *Node B:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include
> itself and node A, changed listen_address and rpc_address to its own IP,
> changed endpoint_snitch to GossipingPropertyFileSnitch. I did not change
> conf/cassandra-rackdc.properties and its contents are
>
> dc=dc1
>
> rack=rack1
>
>
>
> In the logs, I see:
>
> INFO [main] 2016-10-10 22:42:42,850 MessagingService.java:557 - Starting
> Messaging Service on /10.164.32.29:7000
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__10.164.32.29-3A7000&d=DQMFaQ&c=08AGY6txKsvMOP6lYkHQpPMRA1U6kqhAwGa8-0QCg3M&r=yfYEBHVkX6l0zImlOIBID0gmhluYPD5Jje-3CtaT3ow&m=aDxgr_M4K2TNU2MqOBLrREEaFoUbWmFpnP4l67CROag&s=t4xNW-nnA0EVZWnt-PSAoq4vx6Lep2ObgkG3ZuUUg7c&e=>
> (eth0)
>
> INFO [main] 2016-10-10 22:42:42,864 StorageService.java:784 - This node
> will not auto bootstrap because it is configured to be a seed node.
>
>
>
> So I start a third node:
>
> *Node C:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include node
> A and node B, changed listen_address and rpc_address to its own IP, changed
> endpoint_snitch to GossipingPropertyFileSnitch. I did not change
> conf/cassandra-rackdc.properties.
>
> Now, nodetool status shows:
>
>
>
> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool status
>
> Datacenter: dc1
>
> ===============
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> -- Address Load Tokens Owns (effective) Host ID
> Rack
>
> UJ <Node C IP> 87.81 KB 256 ?
> 9064832d-ed5c-4c42-ad5a-f754b52b670c rack1
>
> UN <Node B IP> 107.72 KB 256 100.0%
> 28b1043f-115b-46a5-b6b6-8609829cde76 rack1
>
> Datacenter: dc2
>
> ===============
>
> Status=Up/Down
>
> |/ State=Normal/Leaving/Joining/Moving
>
> -- Address Load Tokens Owns (effective) Host ID
> Rack
>
> UN <Node A IP> 73.2 KB 256 100.0%
> 09cc542c-2299-45a5-a4d1-159c239ded37 rack2
>
>
>
> Nodetool describe cluster shows:
>
> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool describecluster
>
> Cluster Information:
>
> Name: Test Cluster
>
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>
> Schema versions:
>
> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [<Node B IP>, <Node C IP>]
>
>
>
> 9770e3c5-3135-32e2-b761-65a0f6d8824e: [<Node A IP>]
>
>
>
> Note that there are two schema versions and they don't match.
>
>
>
> I see the following in the system.log:
>
>
>
> INFO [InternalResponseStage:1] 2016-10-10 22:48:36,055
> ColumnFamilyStore.java:390 - Initializing system_auth.roles
>
> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> waiting for schema information to complete
>
> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> schema complete, ready to bootstrap
>
> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> waiting for pending range calculation
>
> INFO [main] 2016-10-10 22:48:36,317 StorageService.java:1149 - JOINING:
> calculation complete, ready to bootstrap
>
> INFO [main] 2016-10-10 22:48:36,319 StorageService.java:1149 - JOINING:
> getting bootstrap token
>
> INFO [main] 2016-10-10 22:48:36,357 StorageService.java:1149 - JOINING:
> sleeping 30000 ms for pending range setup
>
> INFO [main] 2016-10-10 22:49:06,358 StorageService.java:1149 - JOINING:
> Starting to bootstrap...
>
> INFO [main] 2016-10-10 22:49:06,494 StreamResultFuture.java:87 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Executing streaming plan for
> Bootstrap
>
> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,495
> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
> Starting streaming to /<Node A IP>
>
> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,495
> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
> Starting streaming to /<Node B IP>
>
> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,500
> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
> ID#0] Beginning stream session with /<Node B IP>
>
> INFO [STREAM-IN-/<Node B IP>] 2016-10-10 22:49:06,590
> StreamResultFuture.java:183 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Session with /<Node B IP> is complete
>
> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,635
> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
> ID#0] Beginning stream session with /<Node A IP>
>
> ERROR [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
> StreamSession.java:528 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
> Streaming error occurred
>
> java.io.IOException: Connection reset by peer
>
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_102]
>
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> ~[na:1.8.0_102]
>
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_102]
>
> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_102]
>
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> ~[na:1.8.0_102]
>
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
> ~[na:1.8.0_102]
>
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_102]
>
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_102]
>
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:287)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>
> INFO [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
> StreamResultFuture.java:183 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Session with /<Node A IP> is complete
>
> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
> StreamResultFuture.java:210 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Stream failed
>
> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
> StorageService.java:1208 - Error during bootstrap.
>
> org.apache.cassandra.streaming.StreamException: Stream failed
>
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> [guava-18.0.jar:na]
>
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> [guava-18.0.jar:na]
>
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> [guava-18.0.jar:na]
>
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> [guava-18.0.jar:na]
>
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> [guava-18.0.jar:na]
>
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:534)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:305)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>
> ERROR [main] 2016-10-10 22:49:06,641 StorageService.java:1218 - Error
> while waiting on bootstrap to complete. Bootstrap will have to be restarted.
>
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
>
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
>
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
>
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
>
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1213)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:889)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:663)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:528)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:339)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
>
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
>
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
>
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
>
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
>
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
>
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:534)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:305)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
>
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
>
> WARN [main] 2016-10-10 22:49:06,646 StorageService.java:944 - Some data
> streaming failed. Use nodetool to check bootstrap state and resume. For
> more, see `nodetool help bootstrap`. IN_PROGRESS
>
> INFO [main] 2016-10-10 22:49:06,647 CassandraDaemon.java:644 - Waiting
> for gossip to settle before accepting client requests...
>
> INFO [main] 2016-10-10 22:49:14,648 CassandraDaemon.java:675 - No gossip
> backlog; proceeding
>
> INFO [main] 2016-10-10 22:49:14,694 NativeTransportService.java:70 -
> Netty using native Epoll event loop
>
> INFO [main] 2016-10-10 22:49:14,726 Server.java:159 - Using Netty
> Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c,
> netty-codec=netty-codec-4.0.23.Final.208198c,
> netty-codec-http=netty-codec-http-4.0.23.Final.208198c,
> netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c,
> netty-common=netty-common-4.0.23.Final.208198c,
> netty-handler=netty-handler-4.0.23.Final.208198c,
> netty-transport=netty-transport-4.0.23.Final.208198c,
> netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c,
> netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c,
> netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c]
>
> INFO [main] 2016-10-10 22:49:14,726 Server.java:160 - Starting listening
> for CQL clients on /<Node C IP>:9042 (unencrypted)...
>
> INFO [main] 2016-10-10 22:49:14,748 CassandraDaemon.java:477 - Not
> starting RPC server as requested. Use JMX
> (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
>
>
>
> I tried resuming bootstrap but it fails with the same streaming errors:
>
>
>
> verma@<Node C>:~/apache-cassandra-3.0.8$ bin/nodetool bootstrap resume
>
> Resuming bootstrap
>
> [2016-10-10 23:15:11,816] session with /<Node B IP> complete (progress: 0%)
>
> [2016-10-10 23:15:11,939] session with /<Node A IP> complete (progress: 0%)
>
> [2016-10-10 23:15:11,940] Stream failed
>
>
>
> and I see the same error in the system.log:
>
>
>
> StreamSession.java:528 - [Stream #64b73a20-8f3f-11e6-b69a-1b451159408e]
> Streaming error occurred
>
> java.io.IOException: Connection reset by peer
>
> ...
>
>
>
> Does Cassandra support upgrading from 2.2.5 to 3.0.8 in this way? Am I
> missing something?
>
>
>
> Thanks for your time.
>
> -Abhishek.
> ____________________________________________________________________
> CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and
> may be legally privileged. If you are not the intended recipient, do not
> disclose, copy, distribute, or use this email or any attachments. If you
> have received this in error please let the sender know and then delete the
> email and all attachments.
>
Re: Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
datacenter fails because of streaming errors
Posted by Jeff Jirsa <je...@crowdstrike.com>.
No need to cc dev@, user@ is the right list for this question.
As Jon mentioned, you can’t stream (bootstrap/rebuild/repair) across major versions, so don’t try to destroy the cluster – just upgrade in place. It IS a good idea to do one DC at a time, but an in-place upgrade is pretty straightforward – flush, drain, stop Cassandra, replace binaries, start Cassandra, run nodetool upgradesstables -a.
Note that you can run nodetool upgradesstables on more than one node at a time if you can tolerate the hit to your read latencies.
It IS common, I imagine, for there to be schema mismatches temporarily while you have a mixed version cluster – this isn’t necessarily a huge problem, but do try to get to 3.0.8 as quickly as possible once you start, and if you can avoid administrative tasks (such as those that will change the schema) during the process, that’s generally advisable.
From: Abhishek Verma <ve...@uber.com>
Reply-To: "user@cassandra.apache.org" <us...@cassandra.apache.org>
Date: Monday, October 10, 2016 at 4:34 PM
To: "user@cassandra.apache.org" <us...@cassandra.apache.org>, "dev@cassandra.apache.org" <de...@cassandra.apache.org>
Subject: Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8 datacenter fails because of streaming errors
Hi Cassandra users,
We are trying to upgrade our Cassandra version from 2.2.5 to 3.0.8 (running on Mesos, but that's besides the point). We have two datacenters, so in order to preserve our data, we are trying to upgrade one datacenter at a time.
Initially both DCs (dc1 and dc2) are running 2.2.5. The idea is to tear down dc1 completely (delete all the data in it), bring it up with 3.0.8, let data replicate from dc2 to dc1, and then tear down dc2, bring it up with 3.0.8 and replicate data from dc1.
I am able to reproduce the problem on bare metal clusters running on 3 nodes. I am using Oracle's server-jre-8u74-linux-x64 JRE.
Node A: Downloaded 2.2.5-bin.tar.gz, changed the seeds to include its own IP address, changed listen_address and rpc_address to its own IP and changed endpoint_snitch to GossipingPropertyFileSnitch. I changed conf/cassandra-rackdc.properties to
dc=dc2
rack=rack2
This node started up fine and is UN in nodetool status in dc2.
I used CQL shell to create a table and insert 3 rows:
verma@xxxxx:~/apache-cassandra-2.2.5$ bin/cqlsh $HOSTNAME
Connected to Test Cluster at xxxxx:9042.
[cqlsh 5.0.1 | Cassandra 2.2.5 | CQL spec 3.3.1 | Native protocol v4]
Use HELP for help.
cqlsh> desc tmp
CREATE KEYSPACE tmp WITH replication = {'class': 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1'} AND durable_writes = true;
CREATE TABLE tmp.map (
key text PRIMARY KEY,
value text
)...;
cqlsh> select * from tmp.map;
key | value
-----+-------
k1 | v1
k3 | v3
k2 | v2
Node B: Downloaded 3.0.8-bin.tar.gz, changed the seeds to include itself and node A, changed listen_address and rpc_address to its own IP, changed endpoint_snitch to GossipingPropertyFileSnitch. I did not change conf/cassandra-rackdc.properties and its contents are
dc=dc1
rack=rack1
In the logs, I see:
INFO [main] 2016-10-10 22:42:42,850 MessagingService.java:557 - Starting Messaging Service on /10.164.32.29:7000 (eth0)
INFO [main] 2016-10-10 22:42:42,864 StorageService.java:784 - This node will not auto bootstrap because it is configured to be a seed node.
So I start a third node:
Node C: Downloaded 3.0.8-bin.tar.gz, changed the seeds to include node A and node B, changed listen_address and rpc_address to its own IP, changed endpoint_snitch to GossipingPropertyFileSnitch. I did not change conf/cassandra-rackdc.properties.
Now, nodetool status shows:
verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool status
Datacenter: dc1
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UJ <Node C IP> 87.81 KB 256 ? 9064832d-ed5c-4c42-ad5a-f754b52b670c rack1
UN <Node B IP> 107.72 KB 256 100.0% 28b1043f-115b-46a5-b6b6-8609829cde76 rack1
Datacenter: dc2
===============
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
UN <Node A IP> 73.2 KB 256 100.0% 09cc542c-2299-45a5-a4d1-159c239ded37 rack2
Nodetool describe cluster shows:
verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool describecluster
Cluster Information:
Name: Test Cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
c2a2bb4f-7d31-3fb8-a216-00b41a643650: [<Node B IP>, <Node C IP>]
9770e3c5-3135-32e2-b761-65a0f6d8824e: [<Node A IP>]
Note that there are two schema versions and they don't match.
I see the following in the system.log:
INFO [InternalResponseStage:1] 2016-10-10 22:48:36,055 ColumnFamilyStore.java:390 - Initializing system_auth.roles
INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING: waiting for schema information to complete
INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING: schema complete, ready to bootstrap
INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING: waiting for pending range calculation
INFO [main] 2016-10-10 22:48:36,317 StorageService.java:1149 - JOINING: calculation complete, ready to bootstrap
INFO [main] 2016-10-10 22:48:36,319 StorageService.java:1149 - JOINING: getting bootstrap token
INFO [main] 2016-10-10 22:48:36,357 StorageService.java:1149 - JOINING: sleeping 30000 ms for pending range setup
INFO [main] 2016-10-10 22:49:06,358 StorageService.java:1149 - JOINING: Starting to bootstrap...
INFO [main] 2016-10-10 22:49:06,494 StreamResultFuture.java:87 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e] Executing streaming plan for Bootstrap
INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,495 StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e] Starting streaming to /<Node A IP>
INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,495 StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e] Starting streaming to /<Node B IP>
INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,500 StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e, ID#0] Beginning stream session with /<Node B IP>
INFO [STREAM-IN-/<Node B IP>] 2016-10-10 22:49:06,590 StreamResultFuture.java:183 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e] Session with /<Node B IP> is complete
INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,635 StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e, ID#0] Beginning stream session with /<Node A IP>
ERROR [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639 StreamSession.java:528 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e] Streaming error occurred
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_102]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[na:1.8.0_102]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_102]
at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_102]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380) ~[na:1.8.0_102]
at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206) ~[na:1.8.0_102]
at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) ~[na:1.8.0_102]
at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385) ~[na:1.8.0_102]
at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:287) ~[apache-cassandra-3.0.8.jar:3.0.8]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
INFO [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639 StreamResultFuture.java:183 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e] Session with /<Node A IP> is complete
WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640 StreamResultFuture.java:210 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e] Stream failed
WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640 StorageService.java:1208 - Error during bootstrap.
org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.8.jar:3.0.8]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) [guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) [guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) [guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) [guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) [guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211) [apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187) [apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429) [apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:534) [apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:305) [apache-cassandra-3.0.8.jar:3.0.8]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
ERROR [main] 2016-10-10 22:49:06,641 StorageService.java:1218 - Error while waiting on bootstrap to complete. Bootstrap will have to be restarted.
java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException: Stream failed
at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116) ~[guava-18.0.jar:na]
at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1213) [apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:889) [apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:663) [apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.service.StorageService.initServer(StorageService.java:528) [apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:339) [apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557) [apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685) [apache-cassandra-3.0.8.jar:3.0.8]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.8.jar:3.0.8]
at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:534) ~[apache-cassandra-3.0.8.jar:3.0.8]
at org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:305) ~[apache-cassandra-3.0.8.jar:3.0.8]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
WARN [main] 2016-10-10 22:49:06,646 StorageService.java:944 - Some data streaming failed. Use nodetool to check bootstrap state and resume. For more, see `nodetool help bootstrap`. IN_PROGRESS
INFO [main] 2016-10-10 22:49:06,647 CassandraDaemon.java:644 - Waiting for gossip to settle before accepting client requests...
INFO [main] 2016-10-10 22:49:14,648 CassandraDaemon.java:675 - No gossip backlog; proceeding
INFO [main] 2016-10-10 22:49:14,694 NativeTransportService.java:70 - Netty using native Epoll event loop
INFO [main] 2016-10-10 22:49:14,726 Server.java:159 - Using Netty Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c, netty-codec=netty-codec-4.0.23.Final.208198c, netty-codec-http=netty-codec-http-4.0.23.Final.208198c, netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c, netty-common=netty-common-4.0.23.Final.208198c, netty-handler=netty-handler-4.0.23.Final.208198c, netty-transport=netty-transport-4.0.23.Final.208198c, netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c, netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c, netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c]
INFO [main] 2016-10-10 22:49:14,726 Server.java:160 - Starting listening for CQL clients on /<Node C IP>:9042 (unencrypted)...
INFO [main] 2016-10-10 22:49:14,748 CassandraDaemon.java:477 - Not starting RPC server as requested. Use JMX (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
I tried resuming bootstrap but it fails with the same streaming errors:
verma@<Node C>:~/apache-cassandra-3.0.8$ bin/nodetool bootstrap resume
Resuming bootstrap
[2016-10-10 23:15:11,816] session with /<Node B IP> complete (progress: 0%)
[2016-10-10 23:15:11,939] session with /<Node A IP> complete (progress: 0%)
[2016-10-10 23:15:11,940] Stream failed
and I see the same error in the system.log:
StreamSession.java:528 - [Stream #64b73a20-8f3f-11e6-b69a-1b451159408e] Streaming error occurred
java.io.IOException: Connection reset by peer
...
Does Cassandra support upgrading from 2.2.5 to 3.0.8 in this way? Am I missing something?
Thanks for your time.
-Abhishek.
____________________________________________________________________
CONFIDENTIALITY NOTE: This e-mail and any attachments are confidential and may be legally privileged. If you are not the intended recipient, do not disclose, copy, distribute, or use this email or any attachments. If you have received this in error please let the sender know and then delete the email and all attachments.
Re: Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
datacenter fails because of streaming errors
Posted by Utkarsh Sengar <ut...@gmail.com>.
As Johathan said, you need to upgrade cassandra directly and use "nodetool
upgradesstables".
Datastax has an excellent resource on upgrading cassandra
https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgdCassandra.html,
specifically
https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdCassandraDetails.html
Make sure you have a snapshot from which you can restore using "nodetool
snapshot". We upgraded from 1.x to 2.x and the upgrade went south, had to
restore from snapshot.
Thanks,
-Utkarsh
On Mon, Oct 10, 2016 at 4:46 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> You can't stream between major versions. Don't tear down your first data
> center, upgrade it instead.
> On Mon, Oct 10, 2016 at 4:35 PM Abhishek Verma <ve...@uber.com> wrote:
>
>> Hi Cassandra users,
>>
>> We are trying to upgrade our Cassandra version from 2.2.5 to 3.0.8
>> (running on Mesos, but that's besides the point). We have two datacenters,
>> so in order to preserve our data, we are trying to upgrade one datacenter
>> at a time.
>>
>> Initially both DCs (dc1 and dc2) are running 2.2.5. The idea is to tear
>> down dc1 completely (delete all the data in it), bring it up with 3.0.8,
>> let data replicate from dc2 to dc1, and then tear down dc2, bring it up
>> with 3.0.8 and replicate data from dc1.
>>
>> I am able to reproduce the problem on bare metal clusters running on 3
>> nodes. I am using Oracle's server-jre-8u74-linux-x64 JRE.
>>
>> *Node A*: Downloaded 2.2.5-bin.tar.gz, changed the seeds to include its
>> own IP address, changed listen_address and rpc_address to its own IP and
>> changed endpoint_snitch to GossipingPropertyFileSnitch. I
>> changed conf/cassandra-rackdc.properties to
>> dc=dc2
>> rack=rack2
>> This node started up fine and is UN in nodetool status in dc2.
>>
>> I used CQL shell to create a table and insert 3 rows:
>> verma@xxxxx:~/apache-cassandra-2.2.5$ bin/cqlsh $HOSTNAME
>> Connected to Test Cluster at xxxxx:9042.
>> [cqlsh 5.0.1 | Cassandra 2.2.5 | CQL spec 3.3.1 | Native protocol v4]
>> Use HELP for help.
>> cqlsh> desc tmp
>>
>> CREATE KEYSPACE tmp WITH replication = {'class':
>> 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1'} AND durable_writes =
>> true;
>>
>> CREATE TABLE tmp.map (
>> key text PRIMARY KEY,
>> value text
>> )...;
>> cqlsh> select * from tmp.map;
>>
>> key | value
>> -----+-------
>> k1 | v1
>> k3 | v3
>> k2 | v2
>>
>>
>> *Node B:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include
>> itself and node A, changed listen_address and rpc_address to its own IP,
>> changed endpoint_snitch to GossipingPropertyFileSnitch. I did not change
>> conf/cassandra-rackdc.properties and its contents are
>> dc=dc1
>> rack=rack1
>>
>> In the logs, I see:
>> INFO [main] 2016-10-10 22:42:42,850 MessagingService.java:557 - Starting
>> Messaging Service on /10.164.32.29:7000 (eth0)
>> INFO [main] 2016-10-10 22:42:42,864 StorageService.java:784 - This node
>> will not auto bootstrap because it is configured to be a seed node.
>>
>> So I start a third node:
>> *Node C:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include node
>> A and node B, changed listen_address and rpc_address to its own IP, changed
>> endpoint_snitch to GossipingPropertyFileSnitch. I did not change
>> conf/cassandra-rackdc.properties.
>> Now, nodetool status shows:
>>
>> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool status
>> Datacenter: dc1
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> UJ <Node C IP> 87.81 KB 256 ?
>> 9064832d-ed5c-4c42-ad5a-f754b52b670c rack1
>> UN <Node B IP> 107.72 KB 256 100.0%
>> 28b1043f-115b-46a5-b6b6-8609829cde76 rack1
>> Datacenter: dc2
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> UN <Node A IP> 73.2 KB 256 100.0%
>> 09cc542c-2299-45a5-a4d1-159c239ded37 rack2
>>
>> Nodetool describe cluster shows:
>> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool describecluster
>> Cluster Information:
>> Name: Test Cluster
>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Schema versions:
>> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [<Node B IP>, <Node C IP>]
>>
>> 9770e3c5-3135-32e2-b761-65a0f6d8824e: [<Node A IP>]
>>
>> Note that there are two schema versions and they don't match.
>>
>> I see the following in the system.log:
>>
>> INFO [InternalResponseStage:1] 2016-10-10 22:48:36,055
>> ColumnFamilyStore.java:390 - Initializing system_auth.roles
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> waiting for schema information to complete
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> schema complete, ready to bootstrap
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> waiting for pending range calculation
>> INFO [main] 2016-10-10 22:48:36,317 StorageService.java:1149 - JOINING:
>> calculation complete, ready to bootstrap
>> INFO [main] 2016-10-10 22:48:36,319 StorageService.java:1149 - JOINING:
>> getting bootstrap token
>> INFO [main] 2016-10-10 22:48:36,357 StorageService.java:1149 - JOINING:
>> sleeping 30000 ms for pending range setup
>> INFO [main] 2016-10-10 22:49:06,358 StorageService.java:1149 - JOINING:
>> Starting to bootstrap...
>> INFO [main] 2016-10-10 22:49:06,494 StreamResultFuture.java:87 - [Stream
>> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Executing streaming plan for
>> Bootstrap
>> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,495
>> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Starting streaming to /<Node A IP>
>> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,495
>> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Starting streaming to /<Node B IP>
>> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,500
>> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
>> ID#0] Beginning stream session with /<Node B IP>
>> INFO [STREAM-IN-/<Node B IP>] 2016-10-10 22:49:06,590
>> StreamResultFuture.java:183 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Session with /<Node B IP> is complete
>> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,635
>> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
>> ID#0] Beginning stream session with /<Node A IP>
>> ERROR [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
>> StreamSession.java:528 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Streaming error occurred
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_102]
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> ~[na:1.8.0_102]
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> ~[na:1.8.0_102]
>> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_102]
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>> ~[na:1.8.0_102]
>> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
>> ~[na:1.8.0_102]
>> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>> ~[na:1.8.0_102]
>> at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>> ~[na:1.8.0_102]
>> at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:287)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>> INFO [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
>> StreamResultFuture.java:183 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Session with /<Node A IP> is complete
>> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
>> StreamResultFuture.java:210 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Stream failed
>> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
>> StorageService.java:1208 - Error during bootstrap.
>> org.apache.cassandra.streaming.StreamException: Stream failed
>> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.
>> onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.8.jar:
>> 3.0.8]
>> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>> [guava-18.0.jar:na]
>> at com.google.common.util.concurrent.MoreExecutors$
>> DirectExecutor.execute(MoreExecutors.java:457) [guava-18.0.jar:na]
>> at com.google.common.util.concurrent.ExecutionList.
>> executeListener(ExecutionList.java:156) [guava-18.0.jar:na]
>> at com.google.common.util.concurrent.ExecutionList.
>> execute(ExecutionList.java:145) [guava-18.0.jar:na]
>> at com.google.common.util.concurrent.AbstractFuture.
>> setException(AbstractFuture.java:202) [guava-18.0.jar:na]
>> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamResultFuture.
>> handleSessionComplete(StreamResultFuture.java:187)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamSession.
>> closeSession(StreamSession.java:429) [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamSession.
>> onError(StreamSession.java:534) [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:305)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>> ERROR [main] 2016-10-10 22:49:06,641 StorageService.java:1218 - Error
>> while waiting on bootstrap to complete. Bootstrap will have to be restarted.
>> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
>> Stream failed
>> at com.google.common.util.concurrent.AbstractFuture$
>> Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.AbstractFuture$
>> Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
>> ~[guava-18.0.jar:na]
>> at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1213)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:889)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:663)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:528)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:339)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.
>> onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.8.jar:
>> 3.0.8]
>> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>> ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.MoreExecutors$
>> DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.ExecutionList.
>> executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.ExecutionList.
>> execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.AbstractFuture.
>> setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
>> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamResultFuture.
>> handleSessionComplete(StreamResultFuture.java:187)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamSession.
>> closeSession(StreamSession.java:429) ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamSession.
>> onError(StreamSession.java:534) ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:305)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
>> WARN [main] 2016-10-10 22:49:06,646 StorageService.java:944 - Some data
>> streaming failed. Use nodetool to check bootstrap state and resume. For
>> more, see `nodetool help bootstrap`. IN_PROGRESS
>> INFO [main] 2016-10-10 22:49:06,647 CassandraDaemon.java:644 - Waiting
>> for gossip to settle before accepting client requests...
>> INFO [main] 2016-10-10 22:49:14,648 CassandraDaemon.java:675 - No gossip
>> backlog; proceeding
>> INFO [main] 2016-10-10 22:49:14,694 NativeTransportService.java:70 -
>> Netty using native Epoll event loop
>> INFO [main] 2016-10-10 22:49:14,726 Server.java:159 - Using Netty
>> Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c,
>> netty-codec=netty-codec-4.0.23.Final.208198c,
>> netty-codec-http=netty-codec-http-4.0.23.Final.208198c,
>> netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c,
>> netty-common=netty-common-4.0.23.Final.208198c,
>> netty-handler=netty-handler-4.0.23.Final.208198c, netty-transport=netty-
>> transport-4.0.23.Final.208198c, netty-transport-rxtx=netty-
>> transport-rxtx-4.0.23.Final.208198c, netty-transport-sctp=netty-
>> transport-sctp-4.0.23.Final.208198c, netty-transport-udt=netty-
>> transport-udt-4.0.23.Final.208198c]
>> INFO [main] 2016-10-10 22:49:14,726 Server.java:160 - Starting listening
>> for CQL clients on /<Node C IP>:9042 (unencrypted)...
>> INFO [main] 2016-10-10 22:49:14,748 CassandraDaemon.java:477 - Not
>> starting RPC server as requested. Use JMX (StorageService->startRPCServer())
>> or nodetool (enablethrift) to start it
>>
>> I tried resuming bootstrap but it fails with the same streaming errors:
>>
>> verma@<Node C>:~/apache-cassandra-3.0.8$ bin/nodetool bootstrap resume
>> Resuming bootstrap
>> [2016-10-10 23:15:11,816] session with /<Node B IP> complete (progress:
>> 0%)
>> [2016-10-10 23:15:11,939] session with /<Node A IP> complete (progress:
>> 0%)
>> [2016-10-10 23:15:11,940] Stream failed
>>
>> and I see the same error in the system.log:
>>
>> StreamSession.java:528 - [Stream #64b73a20-8f3f-11e6-b69a-1b451159408e]
>> Streaming error occurred
>> java.io.IOException: Connection reset by peer
>> ...
>>
>> Does Cassandra support upgrading from 2.2.5 to 3.0.8 in this way? Am I
>> missing something?
>>
>> Thanks for your time.
>> -Abhishek.
>>
>
--
Thanks,
-Utkarsh
Re: Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
datacenter fails because of streaming errors
Posted by Utkarsh Sengar <ut...@gmail.com>.
As Johathan said, you need to upgrade cassandra directly and use "nodetool
upgradesstables".
Datastax has an excellent resource on upgrading cassandra
https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgdCassandra.html,
specifically
https://docs.datastax.com/en/latest-upgrade/upgrade/cassandra/upgrdCassandraDetails.html
Make sure you have a snapshot from which you can restore using "nodetool
snapshot". We upgraded from 1.x to 2.x and the upgrade went south, had to
restore from snapshot.
Thanks,
-Utkarsh
On Mon, Oct 10, 2016 at 4:46 PM, Jonathan Haddad <jo...@jonhaddad.com> wrote:
> You can't stream between major versions. Don't tear down your first data
> center, upgrade it instead.
> On Mon, Oct 10, 2016 at 4:35 PM Abhishek Verma <ve...@uber.com> wrote:
>
>> Hi Cassandra users,
>>
>> We are trying to upgrade our Cassandra version from 2.2.5 to 3.0.8
>> (running on Mesos, but that's besides the point). We have two datacenters,
>> so in order to preserve our data, we are trying to upgrade one datacenter
>> at a time.
>>
>> Initially both DCs (dc1 and dc2) are running 2.2.5. The idea is to tear
>> down dc1 completely (delete all the data in it), bring it up with 3.0.8,
>> let data replicate from dc2 to dc1, and then tear down dc2, bring it up
>> with 3.0.8 and replicate data from dc1.
>>
>> I am able to reproduce the problem on bare metal clusters running on 3
>> nodes. I am using Oracle's server-jre-8u74-linux-x64 JRE.
>>
>> *Node A*: Downloaded 2.2.5-bin.tar.gz, changed the seeds to include its
>> own IP address, changed listen_address and rpc_address to its own IP and
>> changed endpoint_snitch to GossipingPropertyFileSnitch. I
>> changed conf/cassandra-rackdc.properties to
>> dc=dc2
>> rack=rack2
>> This node started up fine and is UN in nodetool status in dc2.
>>
>> I used CQL shell to create a table and insert 3 rows:
>> verma@xxxxx:~/apache-cassandra-2.2.5$ bin/cqlsh $HOSTNAME
>> Connected to Test Cluster at xxxxx:9042.
>> [cqlsh 5.0.1 | Cassandra 2.2.5 | CQL spec 3.3.1 | Native protocol v4]
>> Use HELP for help.
>> cqlsh> desc tmp
>>
>> CREATE KEYSPACE tmp WITH replication = {'class':
>> 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1'} AND durable_writes =
>> true;
>>
>> CREATE TABLE tmp.map (
>> key text PRIMARY KEY,
>> value text
>> )...;
>> cqlsh> select * from tmp.map;
>>
>> key | value
>> -----+-------
>> k1 | v1
>> k3 | v3
>> k2 | v2
>>
>>
>> *Node B:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include
>> itself and node A, changed listen_address and rpc_address to its own IP,
>> changed endpoint_snitch to GossipingPropertyFileSnitch. I did not change
>> conf/cassandra-rackdc.properties and its contents are
>> dc=dc1
>> rack=rack1
>>
>> In the logs, I see:
>> INFO [main] 2016-10-10 22:42:42,850 MessagingService.java:557 - Starting
>> Messaging Service on /10.164.32.29:7000 (eth0)
>> INFO [main] 2016-10-10 22:42:42,864 StorageService.java:784 - This node
>> will not auto bootstrap because it is configured to be a seed node.
>>
>> So I start a third node:
>> *Node C:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include node
>> A and node B, changed listen_address and rpc_address to its own IP, changed
>> endpoint_snitch to GossipingPropertyFileSnitch. I did not change
>> conf/cassandra-rackdc.properties.
>> Now, nodetool status shows:
>>
>> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool status
>> Datacenter: dc1
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> UJ <Node C IP> 87.81 KB 256 ?
>> 9064832d-ed5c-4c42-ad5a-f754b52b670c rack1
>> UN <Node B IP> 107.72 KB 256 100.0%
>> 28b1043f-115b-46a5-b6b6-8609829cde76 rack1
>> Datacenter: dc2
>> ===============
>> Status=Up/Down
>> |/ State=Normal/Leaving/Joining/Moving
>> -- Address Load Tokens Owns (effective) Host ID
>> Rack
>> UN <Node A IP> 73.2 KB 256 100.0%
>> 09cc542c-2299-45a5-a4d1-159c239ded37 rack2
>>
>> Nodetool describe cluster shows:
>> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool describecluster
>> Cluster Information:
>> Name: Test Cluster
>> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
>> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
>> Schema versions:
>> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [<Node B IP>, <Node C IP>]
>>
>> 9770e3c5-3135-32e2-b761-65a0f6d8824e: [<Node A IP>]
>>
>> Note that there are two schema versions and they don't match.
>>
>> I see the following in the system.log:
>>
>> INFO [InternalResponseStage:1] 2016-10-10 22:48:36,055
>> ColumnFamilyStore.java:390 - Initializing system_auth.roles
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> waiting for schema information to complete
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> schema complete, ready to bootstrap
>> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
>> waiting for pending range calculation
>> INFO [main] 2016-10-10 22:48:36,317 StorageService.java:1149 - JOINING:
>> calculation complete, ready to bootstrap
>> INFO [main] 2016-10-10 22:48:36,319 StorageService.java:1149 - JOINING:
>> getting bootstrap token
>> INFO [main] 2016-10-10 22:48:36,357 StorageService.java:1149 - JOINING:
>> sleeping 30000 ms for pending range setup
>> INFO [main] 2016-10-10 22:49:06,358 StorageService.java:1149 - JOINING:
>> Starting to bootstrap...
>> INFO [main] 2016-10-10 22:49:06,494 StreamResultFuture.java:87 - [Stream
>> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Executing streaming plan for
>> Bootstrap
>> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,495
>> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Starting streaming to /<Node A IP>
>> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,495
>> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Starting streaming to /<Node B IP>
>> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,500
>> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
>> ID#0] Beginning stream session with /<Node B IP>
>> INFO [STREAM-IN-/<Node B IP>] 2016-10-10 22:49:06,590
>> StreamResultFuture.java:183 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Session with /<Node B IP> is complete
>> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,635
>> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
>> ID#0] Beginning stream session with /<Node A IP>
>> ERROR [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
>> StreamSession.java:528 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Streaming error occurred
>> java.io.IOException: Connection reset by peer
>> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_102]
>> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>> ~[na:1.8.0_102]
>> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
>> ~[na:1.8.0_102]
>> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_102]
>> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
>> ~[na:1.8.0_102]
>> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
>> ~[na:1.8.0_102]
>> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>> ~[na:1.8.0_102]
>> at java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>> ~[na:1.8.0_102]
>> at org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:287)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>> INFO [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
>> StreamResultFuture.java:183 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Session with /<Node A IP> is complete
>> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
>> StreamResultFuture.java:210 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
>> Stream failed
>> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
>> StorageService.java:1208 - Error during bootstrap.
>> org.apache.cassandra.streaming.StreamException: Stream failed
>> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.
>> onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.8.jar:
>> 3.0.8]
>> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>> [guava-18.0.jar:na]
>> at com.google.common.util.concurrent.MoreExecutors$
>> DirectExecutor.execute(MoreExecutors.java:457) [guava-18.0.jar:na]
>> at com.google.common.util.concurrent.ExecutionList.
>> executeListener(ExecutionList.java:156) [guava-18.0.jar:na]
>> at com.google.common.util.concurrent.ExecutionList.
>> execute(ExecutionList.java:145) [guava-18.0.jar:na]
>> at com.google.common.util.concurrent.AbstractFuture.
>> setException(AbstractFuture.java:202) [guava-18.0.jar:na]
>> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamResultFuture.
>> handleSessionComplete(StreamResultFuture.java:187)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamSession.
>> closeSession(StreamSession.java:429) [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamSession.
>> onError(StreamSession.java:534) [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:305)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
>> ERROR [main] 2016-10-10 22:49:06,641 StorageService.java:1218 - Error
>> while waiting on bootstrap to complete. Bootstrap will have to be restarted.
>> java.util.concurrent.ExecutionException: org.apache.cassandra.streaming.StreamException:
>> Stream failed
>> at com.google.common.util.concurrent.AbstractFuture$
>> Sync.getValue(AbstractFuture.java:299) ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.AbstractFuture$
>> Sync.get(AbstractFuture.java:286) ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
>> ~[guava-18.0.jar:na]
>> at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1213)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:889)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:663)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:528)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:339)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
>> [apache-cassandra-3.0.8.jar:3.0.8]
>> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.
>> onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-3.0.8.jar:
>> 3.0.8]
>> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
>> ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.MoreExecutors$
>> DirectExecutor.execute(MoreExecutors.java:457) ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.ExecutionList.
>> executeListener(ExecutionList.java:156) ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.ExecutionList.
>> execute(ExecutionList.java:145) ~[guava-18.0.jar:na]
>> at com.google.common.util.concurrent.AbstractFuture.
>> setException(AbstractFuture.java:202) ~[guava-18.0.jar:na]
>> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamResultFuture.
>> handleSessionComplete(StreamResultFuture.java:187)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamSession.
>> closeSession(StreamSession.java:429) ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.StreamSession.
>> onError(StreamSession.java:534) ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at org.apache.cassandra.streaming.ConnectionHandler$
>> IncomingMessageHandler.run(ConnectionHandler.java:305)
>> ~[apache-cassandra-3.0.8.jar:3.0.8]
>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
>> WARN [main] 2016-10-10 22:49:06,646 StorageService.java:944 - Some data
>> streaming failed. Use nodetool to check bootstrap state and resume. For
>> more, see `nodetool help bootstrap`. IN_PROGRESS
>> INFO [main] 2016-10-10 22:49:06,647 CassandraDaemon.java:644 - Waiting
>> for gossip to settle before accepting client requests...
>> INFO [main] 2016-10-10 22:49:14,648 CassandraDaemon.java:675 - No gossip
>> backlog; proceeding
>> INFO [main] 2016-10-10 22:49:14,694 NativeTransportService.java:70 -
>> Netty using native Epoll event loop
>> INFO [main] 2016-10-10 22:49:14,726 Server.java:159 - Using Netty
>> Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c,
>> netty-codec=netty-codec-4.0.23.Final.208198c,
>> netty-codec-http=netty-codec-http-4.0.23.Final.208198c,
>> netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c,
>> netty-common=netty-common-4.0.23.Final.208198c,
>> netty-handler=netty-handler-4.0.23.Final.208198c, netty-transport=netty-
>> transport-4.0.23.Final.208198c, netty-transport-rxtx=netty-
>> transport-rxtx-4.0.23.Final.208198c, netty-transport-sctp=netty-
>> transport-sctp-4.0.23.Final.208198c, netty-transport-udt=netty-
>> transport-udt-4.0.23.Final.208198c]
>> INFO [main] 2016-10-10 22:49:14,726 Server.java:160 - Starting listening
>> for CQL clients on /<Node C IP>:9042 (unencrypted)...
>> INFO [main] 2016-10-10 22:49:14,748 CassandraDaemon.java:477 - Not
>> starting RPC server as requested. Use JMX (StorageService->startRPCServer())
>> or nodetool (enablethrift) to start it
>>
>> I tried resuming bootstrap but it fails with the same streaming errors:
>>
>> verma@<Node C>:~/apache-cassandra-3.0.8$ bin/nodetool bootstrap resume
>> Resuming bootstrap
>> [2016-10-10 23:15:11,816] session with /<Node B IP> complete (progress:
>> 0%)
>> [2016-10-10 23:15:11,939] session with /<Node A IP> complete (progress:
>> 0%)
>> [2016-10-10 23:15:11,940] Stream failed
>>
>> and I see the same error in the system.log:
>>
>> StreamSession.java:528 - [Stream #64b73a20-8f3f-11e6-b69a-1b451159408e]
>> Streaming error occurred
>> java.io.IOException: Connection reset by peer
>> ...
>>
>> Does Cassandra support upgrading from 2.2.5 to 3.0.8 in this way? Am I
>> missing something?
>>
>> Thanks for your time.
>> -Abhishek.
>>
>
--
Thanks,
-Utkarsh
Re: Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
datacenter fails because of streaming errors
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
You can't stream between major versions. Don't tear down your first data
center, upgrade it instead.
On Mon, Oct 10, 2016 at 4:35 PM Abhishek Verma <ve...@uber.com> wrote:
> Hi Cassandra users,
>
> We are trying to upgrade our Cassandra version from 2.2.5 to 3.0.8
> (running on Mesos, but that's besides the point). We have two datacenters,
> so in order to preserve our data, we are trying to upgrade one datacenter
> at a time.
>
> Initially both DCs (dc1 and dc2) are running 2.2.5. The idea is to tear
> down dc1 completely (delete all the data in it), bring it up with 3.0.8,
> let data replicate from dc2 to dc1, and then tear down dc2, bring it up
> with 3.0.8 and replicate data from dc1.
>
> I am able to reproduce the problem on bare metal clusters running on 3
> nodes. I am using Oracle's server-jre-8u74-linux-x64 JRE.
>
> *Node A*: Downloaded 2.2.5-bin.tar.gz, changed the seeds to include its
> own IP address, changed listen_address and rpc_address to its own IP and
> changed endpoint_snitch to GossipingPropertyFileSnitch. I
> changed conf/cassandra-rackdc.properties to
> dc=dc2
> rack=rack2
> This node started up fine and is UN in nodetool status in dc2.
>
> I used CQL shell to create a table and insert 3 rows:
> verma@xxxxx:~/apache-cassandra-2.2.5$ bin/cqlsh $HOSTNAME
> Connected to Test Cluster at xxxxx:9042.
> [cqlsh 5.0.1 | Cassandra 2.2.5 | CQL spec 3.3.1 | Native protocol v4]
> Use HELP for help.
> cqlsh> desc tmp
>
> CREATE KEYSPACE tmp WITH replication = {'class':
> 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1'} AND durable_writes =
> true;
>
> CREATE TABLE tmp.map (
> key text PRIMARY KEY,
> value text
> )...;
> cqlsh> select * from tmp.map;
>
> key | value
> -----+-------
> k1 | v1
> k3 | v3
> k2 | v2
>
>
> *Node B:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include
> itself and node A, changed listen_address and rpc_address to its own IP,
> changed endpoint_snitch to GossipingPropertyFileSnitch. I did not change
> conf/cassandra-rackdc.properties and its contents are
> dc=dc1
> rack=rack1
>
> In the logs, I see:
> INFO [main] 2016-10-10 22:42:42,850 MessagingService.java:557 - Starting
> Messaging Service on /10.164.32.29:7000 (eth0)
> INFO [main] 2016-10-10 22:42:42,864 StorageService.java:784 - This node
> will not auto bootstrap because it is configured to be a seed node.
>
> So I start a third node:
> *Node C:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include node
> A and node B, changed listen_address and rpc_address to its own IP, changed
> endpoint_snitch to GossipingPropertyFileSnitch. I did not change
> conf/cassandra-rackdc.properties.
> Now, nodetool status shows:
>
> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool status
> Datacenter: dc1
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UJ <Node C IP> 87.81 KB 256 ?
> 9064832d-ed5c-4c42-ad5a-f754b52b670c rack1
> UN <Node B IP> 107.72 KB 256 100.0%
> 28b1043f-115b-46a5-b6b6-8609829cde76 rack1
> Datacenter: dc2
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN <Node A IP> 73.2 KB 256 100.0%
> 09cc542c-2299-45a5-a4d1-159c239ded37 rack2
>
> Nodetool describe cluster shows:
> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool describecluster
> Cluster Information:
> Name: Test Cluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [<Node B IP>, <Node C IP>]
>
> 9770e3c5-3135-32e2-b761-65a0f6d8824e: [<Node A IP>]
>
> Note that there are two schema versions and they don't match.
>
> I see the following in the system.log:
>
> INFO [InternalResponseStage:1] 2016-10-10 22:48:36,055
> ColumnFamilyStore.java:390 - Initializing system_auth.roles
> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> waiting for schema information to complete
> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> schema complete, ready to bootstrap
> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> waiting for pending range calculation
> INFO [main] 2016-10-10 22:48:36,317 StorageService.java:1149 - JOINING:
> calculation complete, ready to bootstrap
> INFO [main] 2016-10-10 22:48:36,319 StorageService.java:1149 - JOINING:
> getting bootstrap token
> INFO [main] 2016-10-10 22:48:36,357 StorageService.java:1149 - JOINING:
> sleeping 30000 ms for pending range setup
> INFO [main] 2016-10-10 22:49:06,358 StorageService.java:1149 - JOINING:
> Starting to bootstrap...
> INFO [main] 2016-10-10 22:49:06,494 StreamResultFuture.java:87 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Executing streaming plan for
> Bootstrap
> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,495
> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
> Starting streaming to /<Node A IP>
> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,495
> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
> Starting streaming to /<Node B IP>
> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,500
> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
> ID#0] Beginning stream session with /<Node B IP>
> INFO [STREAM-IN-/<Node B IP>] 2016-10-10 22:49:06,590
> StreamResultFuture.java:183 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Session with /<Node B IP> is complete
> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,635
> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
> ID#0] Beginning stream session with /<Node A IP>
> ERROR [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
> StreamSession.java:528 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
> Streaming error occurred
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_102]
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> ~[na:1.8.0_102]
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_102]
> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_102]
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> ~[na:1.8.0_102]
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
> ~[na:1.8.0_102]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_102]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_102]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:287)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> INFO [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
> StreamResultFuture.java:183 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Session with /<Node A IP> is complete
> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
> StreamResultFuture.java:210 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Stream failed
> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
> StorageService.java:1208 - Error during bootstrap.
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> [guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> [guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> [guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> [guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> [guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:534)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:305)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> ERROR [main] 2016-10-10 22:49:06,641 StorageService.java:1218 - Error
> while waiting on bootstrap to complete. Bootstrap will have to be restarted.
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1213)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:889)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:663)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:528)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:339)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:534)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:305)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
> WARN [main] 2016-10-10 22:49:06,646 StorageService.java:944 - Some data
> streaming failed. Use nodetool to check bootstrap state and resume. For
> more, see `nodetool help bootstrap`. IN_PROGRESS
> INFO [main] 2016-10-10 22:49:06,647 CassandraDaemon.java:644 - Waiting
> for gossip to settle before accepting client requests...
> INFO [main] 2016-10-10 22:49:14,648 CassandraDaemon.java:675 - No gossip
> backlog; proceeding
> INFO [main] 2016-10-10 22:49:14,694 NativeTransportService.java:70 -
> Netty using native Epoll event loop
> INFO [main] 2016-10-10 22:49:14,726 Server.java:159 - Using Netty
> Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c,
> netty-codec=netty-codec-4.0.23.Final.208198c,
> netty-codec-http=netty-codec-http-4.0.23.Final.208198c,
> netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c,
> netty-common=netty-common-4.0.23.Final.208198c,
> netty-handler=netty-handler-4.0.23.Final.208198c,
> netty-transport=netty-transport-4.0.23.Final.208198c,
> netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c,
> netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c,
> netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c]
> INFO [main] 2016-10-10 22:49:14,726 Server.java:160 - Starting listening
> for CQL clients on /<Node C IP>:9042 (unencrypted)...
> INFO [main] 2016-10-10 22:49:14,748 CassandraDaemon.java:477 - Not
> starting RPC server as requested. Use JMX
> (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
>
> I tried resuming bootstrap but it fails with the same streaming errors:
>
> verma@<Node C>:~/apache-cassandra-3.0.8$ bin/nodetool bootstrap resume
> Resuming bootstrap
> [2016-10-10 23:15:11,816] session with /<Node B IP> complete (progress: 0%)
> [2016-10-10 23:15:11,939] session with /<Node A IP> complete (progress: 0%)
> [2016-10-10 23:15:11,940] Stream failed
>
> and I see the same error in the system.log:
>
> StreamSession.java:528 - [Stream #64b73a20-8f3f-11e6-b69a-1b451159408e]
> Streaming error occurred
> java.io.IOException: Connection reset by peer
> ...
>
> Does Cassandra support upgrading from 2.2.5 to 3.0.8 in this way? Am I
> missing something?
>
> Thanks for your time.
> -Abhishek.
>
Re: Bootstrapping data from Cassandra 2.2.5 datacenter to 3.0.8
datacenter fails because of streaming errors
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
You can't stream between major versions. Don't tear down your first data
center, upgrade it instead.
On Mon, Oct 10, 2016 at 4:35 PM Abhishek Verma <ve...@uber.com> wrote:
> Hi Cassandra users,
>
> We are trying to upgrade our Cassandra version from 2.2.5 to 3.0.8
> (running on Mesos, but that's besides the point). We have two datacenters,
> so in order to preserve our data, we are trying to upgrade one datacenter
> at a time.
>
> Initially both DCs (dc1 and dc2) are running 2.2.5. The idea is to tear
> down dc1 completely (delete all the data in it), bring it up with 3.0.8,
> let data replicate from dc2 to dc1, and then tear down dc2, bring it up
> with 3.0.8 and replicate data from dc1.
>
> I am able to reproduce the problem on bare metal clusters running on 3
> nodes. I am using Oracle's server-jre-8u74-linux-x64 JRE.
>
> *Node A*: Downloaded 2.2.5-bin.tar.gz, changed the seeds to include its
> own IP address, changed listen_address and rpc_address to its own IP and
> changed endpoint_snitch to GossipingPropertyFileSnitch. I
> changed conf/cassandra-rackdc.properties to
> dc=dc2
> rack=rack2
> This node started up fine and is UN in nodetool status in dc2.
>
> I used CQL shell to create a table and insert 3 rows:
> verma@xxxxx:~/apache-cassandra-2.2.5$ bin/cqlsh $HOSTNAME
> Connected to Test Cluster at xxxxx:9042.
> [cqlsh 5.0.1 | Cassandra 2.2.5 | CQL spec 3.3.1 | Native protocol v4]
> Use HELP for help.
> cqlsh> desc tmp
>
> CREATE KEYSPACE tmp WITH replication = {'class':
> 'NetworkTopologyStrategy', 'dc1': '1', 'dc2': '1'} AND durable_writes =
> true;
>
> CREATE TABLE tmp.map (
> key text PRIMARY KEY,
> value text
> )...;
> cqlsh> select * from tmp.map;
>
> key | value
> -----+-------
> k1 | v1
> k3 | v3
> k2 | v2
>
>
> *Node B:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include
> itself and node A, changed listen_address and rpc_address to its own IP,
> changed endpoint_snitch to GossipingPropertyFileSnitch. I did not change
> conf/cassandra-rackdc.properties and its contents are
> dc=dc1
> rack=rack1
>
> In the logs, I see:
> INFO [main] 2016-10-10 22:42:42,850 MessagingService.java:557 - Starting
> Messaging Service on /10.164.32.29:7000 (eth0)
> INFO [main] 2016-10-10 22:42:42,864 StorageService.java:784 - This node
> will not auto bootstrap because it is configured to be a seed node.
>
> So I start a third node:
> *Node C:* Downloaded 3.0.8-bin.tar.gz, changed the seeds to include node
> A and node B, changed listen_address and rpc_address to its own IP, changed
> endpoint_snitch to GossipingPropertyFileSnitch. I did not change
> conf/cassandra-rackdc.properties.
> Now, nodetool status shows:
>
> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool status
> Datacenter: dc1
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UJ <Node C IP> 87.81 KB 256 ?
> 9064832d-ed5c-4c42-ad5a-f754b52b670c rack1
> UN <Node B IP> 107.72 KB 256 100.0%
> 28b1043f-115b-46a5-b6b6-8609829cde76 rack1
> Datacenter: dc2
> ===============
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN <Node A IP> 73.2 KB 256 100.0%
> 09cc542c-2299-45a5-a4d1-159c239ded37 rack2
>
> Nodetool describe cluster shows:
> verma@xxxxxxx:~/apache-cassandra-3.0.8$ bin/nodetool describecluster
> Cluster Information:
> Name: Test Cluster
> Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
> Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
> Schema versions:
> c2a2bb4f-7d31-3fb8-a216-00b41a643650: [<Node B IP>, <Node C IP>]
>
> 9770e3c5-3135-32e2-b761-65a0f6d8824e: [<Node A IP>]
>
> Note that there are two schema versions and they don't match.
>
> I see the following in the system.log:
>
> INFO [InternalResponseStage:1] 2016-10-10 22:48:36,055
> ColumnFamilyStore.java:390 - Initializing system_auth.roles
> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> waiting for schema information to complete
> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> schema complete, ready to bootstrap
> INFO [main] 2016-10-10 22:48:36,316 StorageService.java:1149 - JOINING:
> waiting for pending range calculation
> INFO [main] 2016-10-10 22:48:36,317 StorageService.java:1149 - JOINING:
> calculation complete, ready to bootstrap
> INFO [main] 2016-10-10 22:48:36,319 StorageService.java:1149 - JOINING:
> getting bootstrap token
> INFO [main] 2016-10-10 22:48:36,357 StorageService.java:1149 - JOINING:
> sleeping 30000 ms for pending range setup
> INFO [main] 2016-10-10 22:49:06,358 StorageService.java:1149 - JOINING:
> Starting to bootstrap...
> INFO [main] 2016-10-10 22:49:06,494 StreamResultFuture.java:87 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Executing streaming plan for
> Bootstrap
> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,495
> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
> Starting streaming to /<Node A IP>
> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,495
> StreamSession.java:242 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
> Starting streaming to /<Node B IP>
> INFO [StreamConnectionEstablisher:2] 2016-10-10 22:49:06,500
> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
> ID#0] Beginning stream session with /<Node B IP>
> INFO [STREAM-IN-/<Node B IP>] 2016-10-10 22:49:06,590
> StreamResultFuture.java:183 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Session with /<Node B IP> is complete
> INFO [StreamConnectionEstablisher:1] 2016-10-10 22:49:06,635
> StreamCoordinator.java:213 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e,
> ID#0] Beginning stream session with /<Node A IP>
> ERROR [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
> StreamSession.java:528 - [Stream #bfb5e470-8f3b-11e6-b69a-1b451159408e]
> Streaming error occurred
> java.io.IOException: Connection reset by peer
> at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[na:1.8.0_102]
> at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
> ~[na:1.8.0_102]
> at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223) ~[na:1.8.0_102]
> at sun.nio.ch.IOUtil.read(IOUtil.java:197) ~[na:1.8.0_102]
> at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
> ~[na:1.8.0_102]
> at sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:206)
> ~[na:1.8.0_102]
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> ~[na:1.8.0_102]
> at
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> ~[na:1.8.0_102]
> at
> org.apache.cassandra.streaming.messages.StreamMessage.deserialize(StreamMessage.java:54)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:287)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> INFO [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,639
> StreamResultFuture.java:183 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Session with /<Node A IP> is complete
> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
> StreamResultFuture.java:210 - [Stream
> #bfb5e470-8f3b-11e6-b69a-1b451159408e] Stream failed
> WARN [STREAM-IN-/<Node A IP>] 2016-10-10 22:49:06,640
> StorageService.java:1208 - Error during bootstrap.
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> [guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> [guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> [guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> [guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> [guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:534)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:305)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_102]
> ERROR [main] 2016-10-10 22:49:06,641 StorageService.java:1218 - Error
> while waiting on bootstrap to complete. Bootstrap will have to be restarted.
> java.util.concurrent.ExecutionException:
> org.apache.cassandra.streaming.StreamException: Stream failed
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:299)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:286)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:116)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1213)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:889)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:663)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:528)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:339)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:557)
> [apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:685)
> [apache-cassandra-3.0.8.jar:3.0.8]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at com.google.common.util.concurrent.Futures$6.run(Futures.java:1310)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$DirectExecutor.execute(MoreExecutors.java:457)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-18.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-18.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:211)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:187)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:429)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.StreamSession.onError(StreamSession.java:534)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at
> org.apache.cassandra.streaming.ConnectionHandler$IncomingMessageHandler.run(ConnectionHandler.java:305)
> ~[apache-cassandra-3.0.8.jar:3.0.8]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_102]
> WARN [main] 2016-10-10 22:49:06,646 StorageService.java:944 - Some data
> streaming failed. Use nodetool to check bootstrap state and resume. For
> more, see `nodetool help bootstrap`. IN_PROGRESS
> INFO [main] 2016-10-10 22:49:06,647 CassandraDaemon.java:644 - Waiting
> for gossip to settle before accepting client requests...
> INFO [main] 2016-10-10 22:49:14,648 CassandraDaemon.java:675 - No gossip
> backlog; proceeding
> INFO [main] 2016-10-10 22:49:14,694 NativeTransportService.java:70 -
> Netty using native Epoll event loop
> INFO [main] 2016-10-10 22:49:14,726 Server.java:159 - Using Netty
> Version: [netty-buffer=netty-buffer-4.0.23.Final.208198c,
> netty-codec=netty-codec-4.0.23.Final.208198c,
> netty-codec-http=netty-codec-http-4.0.23.Final.208198c,
> netty-codec-socks=netty-codec-socks-4.0.23.Final.208198c,
> netty-common=netty-common-4.0.23.Final.208198c,
> netty-handler=netty-handler-4.0.23.Final.208198c,
> netty-transport=netty-transport-4.0.23.Final.208198c,
> netty-transport-rxtx=netty-transport-rxtx-4.0.23.Final.208198c,
> netty-transport-sctp=netty-transport-sctp-4.0.23.Final.208198c,
> netty-transport-udt=netty-transport-udt-4.0.23.Final.208198c]
> INFO [main] 2016-10-10 22:49:14,726 Server.java:160 - Starting listening
> for CQL clients on /<Node C IP>:9042 (unencrypted)...
> INFO [main] 2016-10-10 22:49:14,748 CassandraDaemon.java:477 - Not
> starting RPC server as requested. Use JMX
> (StorageService->startRPCServer()) or nodetool (enablethrift) to start it
>
> I tried resuming bootstrap but it fails with the same streaming errors:
>
> verma@<Node C>:~/apache-cassandra-3.0.8$ bin/nodetool bootstrap resume
> Resuming bootstrap
> [2016-10-10 23:15:11,816] session with /<Node B IP> complete (progress: 0%)
> [2016-10-10 23:15:11,939] session with /<Node A IP> complete (progress: 0%)
> [2016-10-10 23:15:11,940] Stream failed
>
> and I see the same error in the system.log:
>
> StreamSession.java:528 - [Stream #64b73a20-8f3f-11e6-b69a-1b451159408e]
> Streaming error occurred
> java.io.IOException: Connection reset by peer
> ...
>
> Does Cassandra support upgrading from 2.2.5 to 3.0.8 in this way? Am I
> missing something?
>
> Thanks for your time.
> -Abhishek.
>