You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by br...@gmail.com on 2017/08/12 21:58:12 UTC
Dropping down replication factor
Hi folks, hopefully a quick one:
We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch. It's all in one region but spread across 3 availability zones. It was nicely balanced with 4 nodes in each.
But with a couple of failures and subsequent provisions to the wrong az we now have a cluster with :
5 nodes in az A
5 nodes in az B
2 nodes in az C
Not sure why, but when adding a third node in AZ C it fails to stream after getting all the way to completion and no apparent error in logs. I've looked at a couple of bugs referring to scrubbing and possible OOM bugs due to metadata writing at end of streaming (sorry don't have ticket handy). I'm worried I might not be able to do much with these since the disk space usage is high and they are under a lot of load given the small number of them for this rack.
Rather than troubleshoot this further, what I was thinking about doing was:
- drop the replication factor on our keyspace to two
- hopefully this would reduce load on these two remaining nodes
- run repairs/cleanup across the cluster
- then shoot these two nodes in the 'c' rack
- run repairs/cleanup across the cluster
Would this work with minimal/no disruption?
Should I update their "rack" before hand or after ?
What else am I not thinking about?
My main goal atm is to get back to where the cluster is in a clean consistent state that allows nodes to properly bootstrap.
Thanks for your help in advance.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Dropping down replication factor
Posted by Erick Ramirez <fl...@gmail.com>.
I would discourage dropping to RF=2 because if you're using CL=*QUORUM, it
won't be able to tolerate a node outage.
You mentioned a couple of days ago that there's an index file that is
corrupted on 10.40.17.114. Could you try moving out the sstable set
associated with that corrupt file and try again? Though I echo Jeff's
comments and I'm concerned you have a hardware issue on that node since
OpsCenter tables got corrupted too. The replace method certainly sounds
like a good idea.
On Sun, Aug 13, 2017 at 7:58 AM, <br...@gmail.com> wrote:
> Hi folks, hopefully a quick one:
>
> We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch. It's all
> in one region but spread across 3 availability zones. It was nicely
> balanced with 4 nodes in each.
>
> But with a couple of failures and subsequent provisions to the wrong az we
> now have a cluster with :
>
> 5 nodes in az A
> 5 nodes in az B
> 2 nodes in az C
>
> Not sure why, but when adding a third node in AZ C it fails to stream
> after getting all the way to completion and no apparent error in logs.
> I've looked at a couple of bugs referring to scrubbing and possible OOM
> bugs due to metadata writing at end of streaming (sorry don't have ticket
> handy). I'm worried I might not be able to do much with these since the
> disk space usage is high and they are under a lot of load given the small
> number of them for this rack.
>
> Rather than troubleshoot this further, what I was thinking about doing was:
> - drop the replication factor on our keyspace to two
> - hopefully this would reduce load on these two remaining nodes
> - run repairs/cleanup across the cluster
> - then shoot these two nodes in the 'c' rack
> - run repairs/cleanup across the cluster
>
> Would this work with minimal/no disruption?
> Should I update their "rack" before hand or after ?
> What else am I not thinking about?
>
> My main goal atm is to get back to where the cluster is in a clean
> consistent state that allows nodes to properly bootstrap.
>
> Thanks for your help in advance.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
Re: Dropping down replication factor
Posted by Brian Spindler <br...@gmail.com>.
Thanks Kurt.
We had one sstable from a cf of ours. I am actually running a repair on
that cf now and then plan to try and join the additional nodes as you
suggest. I deleted the opscenter corrupt sstables as well but will not
bother repairing that before adding capacity.
Been keeping an eye across all nodes for corrupt exceptions - so far no new
occurrences.
Thanks again.
-B
On Sun, Aug 13, 2017 at 17:52 kurt greaves <ku...@instaclustr.com> wrote:
>
>
> On 14 Aug. 2017 00:59, "Brian Spindler" <br...@gmail.com> wrote:
>
> Do you think with the setup I've described I'd be ok doing that now to
> recover this node?
>
> The node died trying to run the scrub; I've restarted it but I'm not sure
> it's going to get past a scrub/repair, this is why I deleted the other
> files as a brute force method. I think I might have to do the same here
> and then kick off a repair if I can't just replace it?
>
> is it just opscenter keyspace that has corrupt sstables? if so I wouldn't
> worry about repairing too much. If you can get that third node in C to join
> I'd say your best bet is to just do that until you have enough nodes in C.
> Dropping and increasing RF is pretty risky on a live system.
>
> It sounds to me like you stand a good chance of getting the new nodes in C
> to join so I'd pursue that before trying anything more complicated
>
>
> Doing the repair on the node that had the corrupt data deleted should be
> ok?
>
> Yes. as long as you also deleted corrupt SSTables on any other nodes that
> had them.
>
--
-Brian
Re: Dropping down replication factor
Posted by kurt greaves <ku...@instaclustr.com>.
On 14 Aug. 2017 00:59, "Brian Spindler" <br...@gmail.com> wrote:
Do you think with the setup I've described I'd be ok doing that now to
recover this node?
The node died trying to run the scrub; I've restarted it but I'm not sure
it's going to get past a scrub/repair, this is why I deleted the other
files as a brute force method. I think I might have to do the same here
and then kick off a repair if I can't just replace it?
is it just opscenter keyspace that has corrupt sstables? if so I wouldn't
worry about repairing too much. If you can get that third node in C to join
I'd say your best bet is to just do that until you have enough nodes in C.
Dropping and increasing RF is pretty risky on a live system.
It sounds to me like you stand a good chance of getting the new nodes in C
to join so I'd pursue that before trying anything more complicated
Doing the repair on the node that had the corrupt data deleted should be
ok?
Yes. as long as you also deleted corrupt SSTables on any other nodes that
had them.
Re: Dropping down replication factor
Posted by Brian Spindler <br...@gmail.com>.
Do you think with the setup I've described I'd be ok doing that now to
recover this node?
The node died trying to run the scrub; I've restarted it but I'm not sure
it's going to get past a scrub/repair, this is why I deleted the other
files as a brute force method. I think I might have to do the same here
and then kick off a repair if I can't just replace it?
Doing the repair on the node that had the corrupt data deleted should be
ok?
On Sun, Aug 13, 2017 at 10:29 AM Jeff Jirsa <jj...@gmail.com> wrote:
> Running repairs when you have corrupt sstables can spread the corruption
>
> In 2.1.15, corruption is almost certainly from something like a bad disk
> or bad RAM
>
> One way to deal with corruption is to stop the node and replace is (with
> -Dcassandra.replace_address) so you restream data from neighbors. The
> challenge here is making sure you have a healthy replica for streaming
>
> Please make sure you have backups and snapshots if you have corruption
> popping up
>
> If you're using vnodes, once you get rid of the corruption you may
> consider adding another c node with fewer vnodes to try to get it joined
> faster with less data.
>
>
> --
> Jeff Jirsa
>
>
> On Aug 13, 2017, at 7:11 AM, Brian Spindler <br...@gmail.com>
> wrote:
>
> Hi Jeff, I ran the scrub online and that didn't help. I went ahead and
> stopped the node, deleted all the corrupted data files <cf>-<num>-*.db
> files and planned on running a repair when it came back online.
>
> Unrelated I believe, now another CF is corrupted!
>
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /ephemeral/cassandra/data/OpsCenter/rollups300-45c85324387b35238d056678f8fa8b0f/OpsCenter-rollups300-ka-100672-Data.db
> Caused by: org.apache.cassandra.io.compress.CorruptBlockException:
> (/ephemeral/cassandra/data/OpsCenter/rollups300-45c85324387b35238d056678f8fa8b0f/OpsCenter-rollups300-ka-100672-Data.db):
> corruption detected, chunk at 101500 of length 26523398.
>
> Few days ago when troubleshooting this I did change the OpsCenter keyspace
> RF == 2 from 3 since I thought that would help reduce load. Did that cause
> this corruption?
>
> running *'nodetool scrub OpsCenter rollups300'* on that node now
>
> And now I also see this when running nodetool status:
>
> *"Note: Non-system keyspaces don't have the same replication settings,
> effective ownership information is meaningless"*
>
> What to do?
>
> I still can't stream to this new node cause of this corruption. Disk
> space is getting low on these nodes ...
>
> On Sat, Aug 12, 2017 at 9:51 PM Brian Spindler <br...@gmail.com>
> wrote:
>
>> nothing in logs on the node that it was streaming from.
>>
>> however, I think I found the issue on the other node in the C rack:
>>
>> ERROR [STREAM-IN-/10.40.17.114] 2017-08-12 16:48:53,354
>> StreamSession.java:512 - [Stream #08957970-7f7e-11e7-b2a2-a31e21b877e5]
>> Streaming error occurred
>> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
>> /ephemeral/cassandra/data/...
>>
>> I did a 'cat /var/log/cassandra/system.log|grep Corrupt' and it seems
>> it's a single Index.db file and nothing on the other node.
>>
>> I think nodetool scrub or offline sstablescrub might be in order but with
>> the current load I'm not sure I can take it offline for very long.
>>
>> Thanks again for the help.
>>
>>
>> On Sat, Aug 12, 2017 at 9:38 PM Jeffrey Jirsa <jj...@gmail.com> wrote:
>>
>>> Compaction is backed up – that may be normal write load (because of the
>>> rack imbalance), or it may be a secondary index build. Hard to say for
>>> sure. ‘nodetool compactionstats’ if you’re able to provide it. The jstack
>>> probably not necessary, streaming is being marked as failed and it’s
>>> turning itself off. Not sure why streaming is marked as failing, though,
>>> anything on the sending sides?
>>>
>>>
>>>
>>>
>>>
>>> From: Brian Spindler <br...@gmail.com>
>>> Reply-To: <us...@cassandra.apache.org>
>>> Date: Saturday, August 12, 2017 at 6:34 PM
>>> To: <us...@cassandra.apache.org>
>>> Subject: Re: Dropping down replication factor
>>>
>>> Thanks for replying Jeff.
>>>
>>> Responses below.
>>>
>>> On Sat, Aug 12, 2017 at 8:33 PM Jeff Jirsa <jj...@gmail.com> wrote:
>>>
>>>> Answers inline
>>>>
>>>> --
>>>> Jeff Jirsa
>>>>
>>>>
>>>> > On Aug 12, 2017, at 2:58 PM, brian.spindler@gmail.com wrote:
>>>> >
>>>> > Hi folks, hopefully a quick one:
>>>> >
>>>> > We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch.
>>>> It's all in one region but spread across 3 availability zones. It was
>>>> nicely balanced with 4 nodes in each.
>>>> >
>>>> > But with a couple of failures and subsequent provisions to the wrong
>>>> az we now have a cluster with :
>>>> >
>>>> > 5 nodes in az A
>>>> > 5 nodes in az B
>>>> > 2 nodes in az C
>>>> >
>>>> > Not sure why, but when adding a third node in AZ C it fails to stream
>>>> after getting all the way to completion and no apparent error in logs.
>>>> I've looked at a couple of bugs referring to scrubbing and possible OOM
>>>> bugs due to metadata writing at end of streaming (sorry don't have ticket
>>>> handy). I'm worried I might not be able to do much with these since the
>>>> disk space usage is high and they are under a lot of load given the small
>>>> number of them for this rack.
>>>>
>>>> You'll definitely have higher load on az C instances with rf=3 in this
>>>> ratio
>>>
>>>
>>>> Streaming should still work - are you sure it's not busy doing
>>>> something? Like building secondary index or similar? jstack thread dump
>>>> would be useful, or at least nodetool tpstats
>>>>
>>>> Only other thing might be a backup. We do incrementals x1hr and
>>> snapshots x24h; they are shipped to s3 then links are cleaned up. The
>>> error I get on the node I'm trying to add to rack C is:
>>>
>>> ERROR [main] 2017-08-12 23:54:51,546 CassandraDaemon.java:583 -
>>> Exception encountered during startup
>>> java.lang.RuntimeException: Error during boostrap: Stream failed
>>> at
>>> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:87)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1166)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:944)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
>>> [apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
>>> [apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
>>> [apache-cassandra-2.1.15.jar:2.1.15]
>>> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>>> at
>>> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
>>> ~[guava-16.0.jar:na]
>>> at
>>> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>>> ~[guava-16.0.jar:na]
>>> at
>>> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
>>> ~[guava-16.0.jar:na]
>>> at
>>> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
>>> ~[guava-16.0.jar:na]
>>> at
>>> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
>>> ~[guava-16.0.jar:na]
>>> at
>>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:209)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:413)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:700)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:661)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179)
>>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at
>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>>> ~[na:1.8.0_112]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>>> ~[na:1.8.0_112]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>>> ~[na:1.8.0_112]
>>> at
>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>>> ~[na:1.8.0_112]
>>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
>>> WARN [StorageServiceShutdownHook] 2017-08-12 23:54:51,582
>>> Gossiper.java:1462 - No local state or state is in silent shutdown, not
>>> announcing shutdown
>>> INFO [StorageServiceShutdownHook] 2017-08-12 23:54:51,582
>>> MessagingService.java:734 - Waiting for messaging service to quiesce
>>> INFO [ACCEPT-/10.40.17.114] 2017-08-12 23:54:51,583
>>> MessagingService.java:1020 - MessagingService has terminated the accept()
>>> thread
>>>
>>> And I got this on this same node when it was bootstrapping, I ran
>>> 'nodetool netstats' just before it shutdown:
>>>
>>> Receiving 377 files, 161928296443 bytes total. Already received
>>> 377 files, 161928296443 bytes total
>>>
>>> TPStats on host that was streaming the data to this node:
>>>
>>> Pool Name Active Pending Completed Blocked
>>> All time blocked
>>> MutationStage 1 1 4488289014 0
>>> 0
>>> ReadStage 0 0 24486526 0
>>> 0
>>> RequestResponseStage 0 0 3038847374
>>> <(303)%20884-7374> 0 0
>>> ReadRepairStage 0 0 1601576 0
>>> 0
>>> CounterMutationStage 0 0 68403 0
>>> 0
>>> MiscStage 0 0 0 0
>>> 0
>>> AntiEntropySessions 0 0 0 0
>>> 0
>>> HintedHandoff 0 0 18 0
>>> 0
>>> GossipStage 0 0 2786892 0
>>> 0
>>> CacheCleanupExecutor 0 0 0 0
>>> 0
>>> InternalResponseStage 0 0 61115 0
>>> 0
>>> CommitLogArchiver 0 0 0 0
>>> 0
>>> CompactionExecutor 4 83 304167 0
>>> 0
>>> ValidationExecutor 0 0 78249 0
>>> 0
>>> MigrationStage 0 0 94201 0
>>> 0
>>> AntiEntropyStage 0 0 160505 0
>>> 0
>>> PendingRangeCalculator 0 0 30 0
>>> 0
>>> Sampler 0 0 0 0
>>> 0
>>> MemtableFlushWriter 0 0 71270 0
>>> 0
>>> MemtablePostFlush 0 0 175209 0
>>> 0
>>> MemtableReclaimMemory 0 0 81222 0
>>> 0
>>> Native-Transport-Requests 2 0 1983565628 0
>>> 9405444
>>>
>>> Message type Dropped
>>> READ 218
>>> RANGE_SLICE 15
>>> _TRACE 0
>>> MUTATION 2949001
>>> COUNTER_MUTATION 0
>>> BINARY 0
>>> REQUEST_RESPONSE 0
>>> PAGED_RANGE 0
>>> READ_REPAIR 8571
>>>
>>> I can get a jstack if needed.
>>>
>>>>
>>>> >
>>>> > Rather than troubleshoot this further, what I was thinking about
>>>> doing was:
>>>> > - drop the replication factor on our keyspace to two
>>>>
>>>> Repair before you do this, or you'll lose your consistency guarantees
>>>>
>>>
>>> Given the load on the 2 nodes in rack C I'm hoping a repair will succeed.
>>>
>>>
>>>> > - hopefully this would reduce load on these two remaining nodes
>>>>
>>>> It should, racks awareness guarantees on replica per rack if rf==num
>>>> racks, so right now those 2 c machines have 2.5x as much data as the
>>>> others. This will drop that requirement and drop the load significantly
>>>>
>>>> > - run repairs/cleanup across the cluster
>>>> > - then shoot these two nodes in the 'c' rack
>>>>
>>>> Why shoot the c instances? Why not drop RF and then add 2 more C
>>>> instances, then increase RF back to 3, run repair, then Decom the extra
>>>> instances in a and b?
>>>>
>>>>
>>>> Fair point. I was considering staying at RF two but I think with your
>>> points below, I should reconsider.
>>>
>>>
>>>> > - run repairs/cleanup across the cluster
>>>> >
>>>> > Would this work with minimal/no disruption?
>>>>
>>>> The big risk of running rf=2 is that quorum==all - any gc pause or node
>>>> restarting will make you lose HA or strong consistency guarantees.
>>>>
>>>> > Should I update their "rack" before hand or after ?
>>>>
>>>> You can't change a node's rack once it's in the cluster, it SHOULD
>>>> refuse to start if you do that
>>>>
>>>> Got it.
>>>
>>>
>>>> > What else am I not thinking about?
>>>> >
>>>> > My main goal atm is to get back to where the cluster is in a clean
>>>> consistent state that allows nodes to properly bootstrap.
>>>> >
>>>> > Thanks for your help in advance.
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>> > For additional commands, e-mail: user-help@cassandra.apache.org
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>>
>>>>
Re: Dropping down replication factor
Posted by Jeff Jirsa <jj...@gmail.com>.
Running repairs when you have corrupt sstables can spread the corruption
In 2.1.15, corruption is almost certainly from something like a bad disk or bad RAM
One way to deal with corruption is to stop the node and replace is (with -Dcassandra.replace_address) so you restream data from neighbors. The challenge here is making sure you have a healthy replica for streaming
Please make sure you have backups and snapshots if you have corruption popping up
If you're using vnodes, once you get rid of the corruption you may consider adding another c node with fewer vnodes to try to get it joined faster with less data.
--
Jeff Jirsa
> On Aug 13, 2017, at 7:11 AM, Brian Spindler <br...@gmail.com> wrote:
>
> Hi Jeff, I ran the scrub online and that didn't help. I went ahead and stopped the node, deleted all the corrupted data files <cf>-<num>-*.db files and planned on running a repair when it came back online.
>
> Unrelated I believe, now another CF is corrupted!
>
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /ephemeral/cassandra/data/OpsCenter/rollups300-45c85324387b35238d056678f8fa8b0f/OpsCenter-rollups300-ka-100672-Data.db
> Caused by: org.apache.cassandra.io.compress.CorruptBlockException: (/ephemeral/cassandra/data/OpsCenter/rollups300-45c85324387b35238d056678f8fa8b0f/OpsCenter-rollups300-ka-100672-Data.db): corruption detected, chunk at 101500 of length 26523398.
>
> Few days ago when troubleshooting this I did change the OpsCenter keyspace RF == 2 from 3 since I thought that would help reduce load. Did that cause this corruption?
>
> running 'nodetool scrub OpsCenter rollups300' on that node now
>
> And now I also see this when running nodetool status:
>
> "Note: Non-system keyspaces don't have the same replication settings, effective ownership information is meaningless"
>
> What to do?
>
> I still can't stream to this new node cause of this corruption. Disk space is getting low on these nodes ...
>
>> On Sat, Aug 12, 2017 at 9:51 PM Brian Spindler <br...@gmail.com> wrote:
>> nothing in logs on the node that it was streaming from.
>>
>> however, I think I found the issue on the other node in the C rack:
>>
>> ERROR [STREAM-IN-/10.40.17.114] 2017-08-12 16:48:53,354 StreamSession.java:512 - [Stream #08957970-7f7e-11e7-b2a2-a31e21b877e5] Streaming error occurred
>> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /ephemeral/cassandra/data/...
>>
>> I did a 'cat /var/log/cassandra/system.log|grep Corrupt' and it seems it's a single Index.db file and nothing on the other node.
>>
>> I think nodetool scrub or offline sstablescrub might be in order but with the current load I'm not sure I can take it offline for very long.
>>
>> Thanks again for the help.
>>
>>
>>> On Sat, Aug 12, 2017 at 9:38 PM Jeffrey Jirsa <jj...@gmail.com> wrote:
>>> Compaction is backed up – that may be normal write load (because of the rack imbalance), or it may be a secondary index build. Hard to say for sure. ‘nodetool compactionstats’ if you’re able to provide it. The jstack probably not necessary, streaming is being marked as failed and it’s turning itself off. Not sure why streaming is marked as failing, though, anything on the sending sides?
>>>
>>>
>>>
>>>
>>>
>>> From: Brian Spindler <br...@gmail.com>
>>> Reply-To: <us...@cassandra.apache.org>
>>> Date: Saturday, August 12, 2017 at 6:34 PM
>>> To: <us...@cassandra.apache.org>
>>> Subject: Re: Dropping down replication factor
>>>
>>> Thanks for replying Jeff.
>>>
>>> Responses below.
>>>
>>>> On Sat, Aug 12, 2017 at 8:33 PM Jeff Jirsa <jj...@gmail.com> wrote:
>>>> Answers inline
>>>>
>>>> --
>>>> Jeff Jirsa
>>>>
>>>>
>>>> > On Aug 12, 2017, at 2:58 PM, brian.spindler@gmail.com wrote:
>>>> >
>>>> > Hi folks, hopefully a quick one:
>>>> >
>>>> > We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch. It's all in one region but spread across 3 availability zones. It was nicely balanced with 4 nodes in each.
>>>> >
>>>> > But with a couple of failures and subsequent provisions to the wrong az we now have a cluster with :
>>>> >
>>>> > 5 nodes in az A
>>>> > 5 nodes in az B
>>>> > 2 nodes in az C
>>>> >
>>>> > Not sure why, but when adding a third node in AZ C it fails to stream after getting all the way to completion and no apparent error in logs. I've looked at a couple of bugs referring to scrubbing and possible OOM bugs due to metadata writing at end of streaming (sorry don't have ticket handy). I'm worried I might not be able to do much with these since the disk space usage is high and they are under a lot of load given the small number of them for this rack.
>>>>
>>>> You'll definitely have higher load on az C instances with rf=3 in this ratio
>>>>
>>>> Streaming should still work - are you sure it's not busy doing something? Like building secondary index or similar? jstack thread dump would be useful, or at least nodetool tpstats
>>>>
>>>
>>> Only other thing might be a backup. We do incrementals x1hr and snapshots x24h; they are shipped to s3 then links are cleaned up. The error I get on the node I'm trying to add to rack C is:
>>>
>>> ERROR [main] 2017-08-12 23:54:51,546 CassandraDaemon.java:583 - Exception encountered during startup
>>> java.lang.RuntimeException: Error during boostrap: Stream failed
>>> at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:87) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1166) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:944) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:740) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.service.StorageService.initServer(StorageService.java:617) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391) [apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566) [apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655) [apache-cassandra-2.1.15.jar:2.1.15]
>>> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>>> at org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at com.google.common.util.concurrent.Futures$4.run(Futures.java:1172) ~[guava-16.0.jar:na]
>>> at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297) ~[guava-16.0.jar:na]
>>> at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156) ~[guava-16.0.jar:na]
>>> at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145) ~[guava-16.0.jar:na]
>>> at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202) ~[guava-16.0.jar:na]
>>> at org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:209) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:413) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:700) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:661) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179) ~[apache-cassandra-2.1.15.jar:2.1.15]
>>> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_112]
>>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_112]
>>> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_112]
>>> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_112]
>>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
>>> WARN [StorageServiceShutdownHook] 2017-08-12 23:54:51,582 Gossiper.java:1462 - No local state or state is in silent shutdown, not announcing shutdown
>>> INFO [StorageServiceShutdownHook] 2017-08-12 23:54:51,582 MessagingService.java:734 - Waiting for messaging service to quiesce
>>> INFO [ACCEPT-/10.40.17.114] 2017-08-12 23:54:51,583 MessagingService.java:1020 - MessagingService has terminated the accept() thread
>>>
>>> And I got this on this same node when it was bootstrapping, I ran 'nodetool netstats' just before it shutdown:
>>>
>>> Receiving 377 files, 161928296443 bytes total. Already received 377 files, 161928296443 bytes total
>>>
>>> TPStats on host that was streaming the data to this node:
>>>
>>> Pool Name Active Pending Completed Blocked All time blocked
>>> MutationStage 1 1 4488289014 0 0
>>> ReadStage 0 0 24486526 0 0
>>> RequestResponseStage 0 0 3038847374 0 0
>>> ReadRepairStage 0 0 1601576 0 0
>>> CounterMutationStage 0 0 68403 0 0
>>> MiscStage 0 0 0 0 0
>>> AntiEntropySessions 0 0 0 0 0
>>> HintedHandoff 0 0 18 0 0
>>> GossipStage 0 0 2786892 0 0
>>> CacheCleanupExecutor 0 0 0 0 0
>>> InternalResponseStage 0 0 61115 0 0
>>> CommitLogArchiver 0 0 0 0 0
>>> CompactionExecutor 4 83 304167 0 0
>>> ValidationExecutor 0 0 78249 0 0
>>> MigrationStage 0 0 94201 0 0
>>> AntiEntropyStage 0 0 160505 0 0
>>> PendingRangeCalculator 0 0 30 0 0
>>> Sampler 0 0 0 0 0
>>> MemtableFlushWriter 0 0 71270 0 0
>>> MemtablePostFlush 0 0 175209 0 0
>>> MemtableReclaimMemory 0 0 81222 0 0
>>> Native-Transport-Requests 2 0 1983565628 0 9405444
>>>
>>> Message type Dropped
>>> READ 218
>>> RANGE_SLICE 15
>>> _TRACE 0
>>> MUTATION 2949001
>>> COUNTER_MUTATION 0
>>> BINARY 0
>>> REQUEST_RESPONSE 0
>>> PAGED_RANGE 0
>>> READ_REPAIR 8571
>>>
>>> I can get a jstack if needed.
>>>>
>>>> >
>>>> > Rather than troubleshoot this further, what I was thinking about doing was:
>>>> > - drop the replication factor on our keyspace to two
>>>>
>>>> Repair before you do this, or you'll lose your consistency guarantees
>>>
>>> Given the load on the 2 nodes in rack C I'm hoping a repair will succeed.
>>>
>>>>
>>>> > - hopefully this would reduce load on these two remaining nodes
>>>>
>>>> It should, racks awareness guarantees on replica per rack if rf==num racks, so right now those 2 c machines have 2.5x as much data as the others. This will drop that requirement and drop the load significantly
>>>>
>>>> > - run repairs/cleanup across the cluster
>>>> > - then shoot these two nodes in the 'c' rack
>>>>
>>>> Why shoot the c instances? Why not drop RF and then add 2 more C instances, then increase RF back to 3, run repair, then Decom the extra instances in a and b?
>>>>
>>>>
>>>
>>> Fair point. I was considering staying at RF two but I think with your points below, I should reconsider.
>>>
>>>> > - run repairs/cleanup across the cluster
>>>> >
>>>> > Would this work with minimal/no disruption?
>>>>
>>>> The big risk of running rf=2 is that quorum==all - any gc pause or node restarting will make you lose HA or strong consistency guarantees.
>>>>
>>>> > Should I update their "rack" before hand or after ?
>>>>
>>>> You can't change a node's rack once it's in the cluster, it SHOULD refuse to start if you do that
>>>>
>>>
>>> Got it.
>>>
>>>> > What else am I not thinking about?
>>>> >
>>>> > My main goal atm is to get back to where the cluster is in a clean consistent state that allows nodes to properly bootstrap.
>>>> >
>>>> > Thanks for your help in advance.
>>>> > ---------------------------------------------------------------------
>>>> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>> > For additional commands, e-mail: user-help@cassandra.apache.org
>>>> >
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>>
Re: Dropping down replication factor
Posted by Brian Spindler <br...@gmail.com>.
Hi Jeff, I ran the scrub online and that didn't help. I went ahead and
stopped the node, deleted all the corrupted data files <cf>-<num>-*.db
files and planned on running a repair when it came back online.
Unrelated I believe, now another CF is corrupted!
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/ephemeral/cassandra/data/OpsCenter/rollups300-45c85324387b35238d056678f8fa8b0f/OpsCenter-rollups300-ka-100672-Data.db
Caused by: org.apache.cassandra.io.compress.CorruptBlockException:
(/ephemeral/cassandra/data/OpsCenter/rollups300-45c85324387b35238d056678f8fa8b0f/OpsCenter-rollups300-ka-100672-Data.db):
corruption detected, chunk at 101500 of length 26523398.
Few days ago when troubleshooting this I did change the OpsCenter keyspace
RF == 2 from 3 since I thought that would help reduce load. Did that cause
this corruption?
running *'nodetool scrub OpsCenter rollups300'* on that node now
And now I also see this when running nodetool status:
*"Note: Non-system keyspaces don't have the same replication settings,
effective ownership information is meaningless"*
What to do?
I still can't stream to this new node cause of this corruption. Disk space
is getting low on these nodes ...
On Sat, Aug 12, 2017 at 9:51 PM Brian Spindler <br...@gmail.com>
wrote:
> nothing in logs on the node that it was streaming from.
>
> however, I think I found the issue on the other node in the C rack:
>
> ERROR [STREAM-IN-/10.40.17.114] 2017-08-12 16:48:53,354
> StreamSession.java:512 - [Stream #08957970-7f7e-11e7-b2a2-a31e21b877e5]
> Streaming error occurred
> org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
> /ephemeral/cassandra/data/...
>
> I did a 'cat /var/log/cassandra/system.log|grep Corrupt' and it seems
> it's a single Index.db file and nothing on the other node.
>
> I think nodetool scrub or offline sstablescrub might be in order but with
> the current load I'm not sure I can take it offline for very long.
>
> Thanks again for the help.
>
>
> On Sat, Aug 12, 2017 at 9:38 PM Jeffrey Jirsa <jj...@gmail.com> wrote:
>
>> Compaction is backed up – that may be normal write load (because of the
>> rack imbalance), or it may be a secondary index build. Hard to say for
>> sure. ‘nodetool compactionstats’ if you’re able to provide it. The jstack
>> probably not necessary, streaming is being marked as failed and it’s
>> turning itself off. Not sure why streaming is marked as failing, though,
>> anything on the sending sides?
>>
>>
>>
>>
>>
>> From: Brian Spindler <br...@gmail.com>
>> Reply-To: <us...@cassandra.apache.org>
>> Date: Saturday, August 12, 2017 at 6:34 PM
>> To: <us...@cassandra.apache.org>
>> Subject: Re: Dropping down replication factor
>>
>> Thanks for replying Jeff.
>>
>> Responses below.
>>
>> On Sat, Aug 12, 2017 at 8:33 PM Jeff Jirsa <jj...@gmail.com> wrote:
>>
>>> Answers inline
>>>
>>> --
>>> Jeff Jirsa
>>>
>>>
>>> > On Aug 12, 2017, at 2:58 PM, brian.spindler@gmail.com wrote:
>>> >
>>> > Hi folks, hopefully a quick one:
>>> >
>>> > We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch. It's
>>> all in one region but spread across 3 availability zones. It was nicely
>>> balanced with 4 nodes in each.
>>> >
>>> > But with a couple of failures and subsequent provisions to the wrong
>>> az we now have a cluster with :
>>> >
>>> > 5 nodes in az A
>>> > 5 nodes in az B
>>> > 2 nodes in az C
>>> >
>>> > Not sure why, but when adding a third node in AZ C it fails to stream
>>> after getting all the way to completion and no apparent error in logs.
>>> I've looked at a couple of bugs referring to scrubbing and possible OOM
>>> bugs due to metadata writing at end of streaming (sorry don't have ticket
>>> handy). I'm worried I might not be able to do much with these since the
>>> disk space usage is high and they are under a lot of load given the small
>>> number of them for this rack.
>>>
>>> You'll definitely have higher load on az C instances with rf=3 in this
>>> ratio
>>
>>
>>> Streaming should still work - are you sure it's not busy doing
>>> something? Like building secondary index or similar? jstack thread dump
>>> would be useful, or at least nodetool tpstats
>>>
>>> Only other thing might be a backup. We do incrementals x1hr and
>> snapshots x24h; they are shipped to s3 then links are cleaned up. The
>> error I get on the node I'm trying to add to rack C is:
>>
>> ERROR [main] 2017-08-12 23:54:51,546 CassandraDaemon.java:583 - Exception
>> encountered during startup
>> java.lang.RuntimeException: Error during boostrap: Stream failed
>> at
>> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:87)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1166)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:944)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
>> [apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
>> [apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
>> [apache-cassandra-2.1.15.jar:2.1.15]
>> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
>> at
>> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
>> ~[guava-16.0.jar:na]
>> at
>> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
>> ~[guava-16.0.jar:na]
>> at
>> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
>> ~[guava-16.0.jar:na]
>> at
>> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
>> ~[guava-16.0.jar:na]
>> at
>> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
>> ~[guava-16.0.jar:na]
>> at
>> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:209)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:413)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:700)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:661)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179)
>> ~[apache-cassandra-2.1.15.jar:2.1.15]
>> at
>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
>> ~[na:1.8.0_112]
>> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> ~[na:1.8.0_112]
>> at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> ~[na:1.8.0_112]
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> ~[na:1.8.0_112]
>> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
>> WARN [StorageServiceShutdownHook] 2017-08-12 23:54:51,582
>> Gossiper.java:1462 - No local state or state is in silent shutdown, not
>> announcing shutdown
>> INFO [StorageServiceShutdownHook] 2017-08-12 23:54:51,582
>> MessagingService.java:734 - Waiting for messaging service to quiesce
>> INFO [ACCEPT-/10.40.17.114] 2017-08-12 23:54:51,583
>> MessagingService.java:1020 - MessagingService has terminated the accept()
>> thread
>>
>> And I got this on this same node when it was bootstrapping, I ran
>> 'nodetool netstats' just before it shutdown:
>>
>> Receiving 377 files, 161928296443 bytes total. Already received
>> 377 files, 161928296443 bytes total
>>
>> TPStats on host that was streaming the data to this node:
>>
>> Pool Name Active Pending Completed Blocked
>> All time blocked
>> MutationStage 1 1 4488289014 0
>> 0
>> ReadStage 0 0 24486526 0
>> 0
>> RequestResponseStage 0 0 3038847374
>> <(303)%20884-7374> 0 0
>> ReadRepairStage 0 0 1601576 0
>> 0
>> CounterMutationStage 0 0 68403 0
>> 0
>> MiscStage 0 0 0 0
>> 0
>> AntiEntropySessions 0 0 0 0
>> 0
>> HintedHandoff 0 0 18 0
>> 0
>> GossipStage 0 0 2786892 0
>> 0
>> CacheCleanupExecutor 0 0 0 0
>> 0
>> InternalResponseStage 0 0 61115 0
>> 0
>> CommitLogArchiver 0 0 0 0
>> 0
>> CompactionExecutor 4 83 304167 0
>> 0
>> ValidationExecutor 0 0 78249 0
>> 0
>> MigrationStage 0 0 94201 0
>> 0
>> AntiEntropyStage 0 0 160505 0
>> 0
>> PendingRangeCalculator 0 0 30 0
>> 0
>> Sampler 0 0 0 0
>> 0
>> MemtableFlushWriter 0 0 71270 0
>> 0
>> MemtablePostFlush 0 0 175209 0
>> 0
>> MemtableReclaimMemory 0 0 81222 0
>> 0
>> Native-Transport-Requests 2 0 1983565628 0
>> 9405444
>>
>> Message type Dropped
>> READ 218
>> RANGE_SLICE 15
>> _TRACE 0
>> MUTATION 2949001
>> COUNTER_MUTATION 0
>> BINARY 0
>> REQUEST_RESPONSE 0
>> PAGED_RANGE 0
>> READ_REPAIR 8571
>>
>> I can get a jstack if needed.
>>
>>>
>>> >
>>> > Rather than troubleshoot this further, what I was thinking about doing
>>> was:
>>> > - drop the replication factor on our keyspace to two
>>>
>>> Repair before you do this, or you'll lose your consistency guarantees
>>>
>>
>> Given the load on the 2 nodes in rack C I'm hoping a repair will succeed.
>>
>>
>>> > - hopefully this would reduce load on these two remaining nodes
>>>
>>> It should, racks awareness guarantees on replica per rack if rf==num
>>> racks, so right now those 2 c machines have 2.5x as much data as the
>>> others. This will drop that requirement and drop the load significantly
>>>
>>> > - run repairs/cleanup across the cluster
>>> > - then shoot these two nodes in the 'c' rack
>>>
>>> Why shoot the c instances? Why not drop RF and then add 2 more C
>>> instances, then increase RF back to 3, run repair, then Decom the extra
>>> instances in a and b?
>>>
>>>
>>> Fair point. I was considering staying at RF two but I think with your
>> points below, I should reconsider.
>>
>>
>>> > - run repairs/cleanup across the cluster
>>> >
>>> > Would this work with minimal/no disruption?
>>>
>>> The big risk of running rf=2 is that quorum==all - any gc pause or node
>>> restarting will make you lose HA or strong consistency guarantees.
>>>
>>> > Should I update their "rack" before hand or after ?
>>>
>>> You can't change a node's rack once it's in the cluster, it SHOULD
>>> refuse to start if you do that
>>>
>>> Got it.
>>
>>
>>> > What else am I not thinking about?
>>> >
>>> > My main goal atm is to get back to where the cluster is in a clean
>>> consistent state that allows nodes to properly bootstrap.
>>> >
>>> > Thanks for your help in advance.
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>> > For additional commands, e-mail: user-help@cassandra.apache.org
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>>> For additional commands, e-mail: user-help@cassandra.apache.org
>>>
>>>
Re: Dropping down replication factor
Posted by Brian Spindler <br...@gmail.com>.
nothing in logs on the node that it was streaming from.
however, I think I found the issue on the other node in the C rack:
ERROR [STREAM-IN-/10.40.17.114] 2017-08-12 16:48:53,354
StreamSession.java:512 - [Stream #08957970-7f7e-11e7-b2a2-a31e21b877e5]
Streaming error occurred
org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted:
/ephemeral/cassandra/data/...
I did a 'cat /var/log/cassandra/system.log|grep Corrupt' and it seems it's
a single Index.db file and nothing on the other node.
I think nodetool scrub or offline sstablescrub might be in order but with
the current load I'm not sure I can take it offline for very long.
Thanks again for the help.
On Sat, Aug 12, 2017 at 9:38 PM Jeffrey Jirsa <jj...@gmail.com> wrote:
> Compaction is backed up – that may be normal write load (because of the
> rack imbalance), or it may be a secondary index build. Hard to say for
> sure. ‘nodetool compactionstats’ if you’re able to provide it. The jstack
> probably not necessary, streaming is being marked as failed and it’s
> turning itself off. Not sure why streaming is marked as failing, though,
> anything on the sending sides?
>
>
>
>
>
> From: Brian Spindler <br...@gmail.com>
> Reply-To: <us...@cassandra.apache.org>
> Date: Saturday, August 12, 2017 at 6:34 PM
> To: <us...@cassandra.apache.org>
> Subject: Re: Dropping down replication factor
>
> Thanks for replying Jeff.
>
> Responses below.
>
> On Sat, Aug 12, 2017 at 8:33 PM Jeff Jirsa <jj...@gmail.com> wrote:
>
>> Answers inline
>>
>> --
>> Jeff Jirsa
>>
>>
>> > On Aug 12, 2017, at 2:58 PM, brian.spindler@gmail.com wrote:
>> >
>> > Hi folks, hopefully a quick one:
>> >
>> > We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch. It's
>> all in one region but spread across 3 availability zones. It was nicely
>> balanced with 4 nodes in each.
>> >
>> > But with a couple of failures and subsequent provisions to the wrong az
>> we now have a cluster with :
>> >
>> > 5 nodes in az A
>> > 5 nodes in az B
>> > 2 nodes in az C
>> >
>> > Not sure why, but when adding a third node in AZ C it fails to stream
>> after getting all the way to completion and no apparent error in logs.
>> I've looked at a couple of bugs referring to scrubbing and possible OOM
>> bugs due to metadata writing at end of streaming (sorry don't have ticket
>> handy). I'm worried I might not be able to do much with these since the
>> disk space usage is high and they are under a lot of load given the small
>> number of them for this rack.
>>
>> You'll definitely have higher load on az C instances with rf=3 in this
>> ratio
>
>
>> Streaming should still work - are you sure it's not busy doing something?
>> Like building secondary index or similar? jstack thread dump would be
>> useful, or at least nodetool tpstats
>>
>> Only other thing might be a backup. We do incrementals x1hr and
> snapshots x24h; they are shipped to s3 then links are cleaned up. The
> error I get on the node I'm trying to add to rack C is:
>
> ERROR [main] 2017-08-12 23:54:51,546 CassandraDaemon.java:583 - Exception
> encountered during startup
> java.lang.RuntimeException: Error during boostrap: Stream failed
> at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:87)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1166)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:944)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
> [apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
> [apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
> [apache-cassandra-2.1.15.jar:2.1.15]
> Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
> at
> org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
> ~[guava-16.0.jar:na]
> at
> com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
> ~[guava-16.0.jar:na]
> at
> org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:209)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:413)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:700)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:661)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179)
> ~[apache-cassandra-2.1.15.jar:2.1.15]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_112]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_112]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> ~[na:1.8.0_112]
> at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
> WARN [StorageServiceShutdownHook] 2017-08-12 23:54:51,582
> Gossiper.java:1462 - No local state or state is in silent shutdown, not
> announcing shutdown
> INFO [StorageServiceShutdownHook] 2017-08-12 23:54:51,582
> MessagingService.java:734 - Waiting for messaging service to quiesce
> INFO [ACCEPT-/10.40.17.114] 2017-08-12 23:54:51,583
> MessagingService.java:1020 - MessagingService has terminated the accept()
> thread
>
> And I got this on this same node when it was bootstrapping, I ran
> 'nodetool netstats' just before it shutdown:
>
> Receiving 377 files, 161928296443 bytes total. Already received
> 377 files, 161928296443 bytes total
>
> TPStats on host that was streaming the data to this node:
>
> Pool Name Active Pending Completed Blocked
> All time blocked
> MutationStage 1 1 4488289014 0
> 0
> ReadStage 0 0 24486526 0
> 0
> RequestResponseStage 0 0 3038847374
> <(303)%20884-7374> 0 0
> ReadRepairStage 0 0 1601576 0
> 0
> CounterMutationStage 0 0 68403 0
> 0
> MiscStage 0 0 0 0
> 0
> AntiEntropySessions 0 0 0 0
> 0
> HintedHandoff 0 0 18 0
> 0
> GossipStage 0 0 2786892 0
> 0
> CacheCleanupExecutor 0 0 0 0
> 0
> InternalResponseStage 0 0 61115 0
> 0
> CommitLogArchiver 0 0 0 0
> 0
> CompactionExecutor 4 83 304167 0
> 0
> ValidationExecutor 0 0 78249 0
> 0
> MigrationStage 0 0 94201 0
> 0
> AntiEntropyStage 0 0 160505 0
> 0
> PendingRangeCalculator 0 0 30 0
> 0
> Sampler 0 0 0 0
> 0
> MemtableFlushWriter 0 0 71270 0
> 0
> MemtablePostFlush 0 0 175209 0
> 0
> MemtableReclaimMemory 0 0 81222 0
> 0
> Native-Transport-Requests 2 0 1983565628 0
> 9405444
>
> Message type Dropped
> READ 218
> RANGE_SLICE 15
> _TRACE 0
> MUTATION 2949001
> COUNTER_MUTATION 0
> BINARY 0
> REQUEST_RESPONSE 0
> PAGED_RANGE 0
> READ_REPAIR 8571
>
> I can get a jstack if needed.
>
>>
>> >
>> > Rather than troubleshoot this further, what I was thinking about doing
>> was:
>> > - drop the replication factor on our keyspace to two
>>
>> Repair before you do this, or you'll lose your consistency guarantees
>>
>
> Given the load on the 2 nodes in rack C I'm hoping a repair will succeed.
>
>
>> > - hopefully this would reduce load on these two remaining nodes
>>
>> It should, racks awareness guarantees on replica per rack if rf==num
>> racks, so right now those 2 c machines have 2.5x as much data as the
>> others. This will drop that requirement and drop the load significantly
>>
>> > - run repairs/cleanup across the cluster
>> > - then shoot these two nodes in the 'c' rack
>>
>> Why shoot the c instances? Why not drop RF and then add 2 more C
>> instances, then increase RF back to 3, run repair, then Decom the extra
>> instances in a and b?
>>
>>
>> Fair point. I was considering staying at RF two but I think with your
> points below, I should reconsider.
>
>
>> > - run repairs/cleanup across the cluster
>> >
>> > Would this work with minimal/no disruption?
>>
>> The big risk of running rf=2 is that quorum==all - any gc pause or node
>> restarting will make you lose HA or strong consistency guarantees.
>>
>> > Should I update their "rack" before hand or after ?
>>
>> You can't change a node's rack once it's in the cluster, it SHOULD refuse
>> to start if you do that
>>
>> Got it.
>
>
>> > What else am I not thinking about?
>> >
>> > My main goal atm is to get back to where the cluster is in a clean
>> consistent state that allows nodes to properly bootstrap.
>> >
>> > Thanks for your help in advance.
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> > For additional commands, e-mail: user-help@cassandra.apache.org
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
Re: Dropping down replication factor
Posted by Jeffrey Jirsa <jj...@gmail.com>.
Compaction is backed up that may be normal write load (because of the rack
imbalance), or it may be a secondary index build. Hard to say for sure.
nodetool compactionstats¹ if you¹re able to provide it. The jstack probably
not necessary, streaming is being marked as failed and it¹s turning itself
off. Not sure why streaming is marked as failing, though, anything on the
sending sides?
From: Brian Spindler <br...@gmail.com>
Reply-To: <us...@cassandra.apache.org>
Date: Saturday, August 12, 2017 at 6:34 PM
To: <us...@cassandra.apache.org>
Subject: Re: Dropping down replication factor
Thanks for replying Jeff.
Responses below.
On Sat, Aug 12, 2017 at 8:33 PM Jeff Jirsa <jj...@gmail.com> wrote:
> Answers inline
>
> --
> Jeff Jirsa
>
>
>> > On Aug 12, 2017, at 2:58 PM, brian.spindler@gmail.com wrote:
>> >
>> > Hi folks, hopefully a quick one:
>> >
>> > We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch. It's all
>> in one region but spread across 3 availability zones. It was nicely balanced
>> with 4 nodes in each.
>> >
>> > But with a couple of failures and subsequent provisions to the wrong az we
>> now have a cluster with :
>> >
>> > 5 nodes in az A
>> > 5 nodes in az B
>> > 2 nodes in az C
>> >
>> > Not sure why, but when adding a third node in AZ C it fails to stream after
>> getting all the way to completion and no apparent error in logs. I've looked
>> at a couple of bugs referring to scrubbing and possible OOM bugs due to
>> metadata writing at end of streaming (sorry don't have ticket handy). I'm
>> worried I might not be able to do much with these since the disk space usage
>> is high and they are under a lot of load given the small number of them for
>> this rack.
>
> You'll definitely have higher load on az C instances with rf=3 in this ratio
>
> Streaming should still work - are you sure it's not busy doing something? Like
> building secondary index or similar? jstack thread dump would be useful, or at
> least nodetool tpstats
>
Only other thing might be a backup. We do incrementals x1hr and snapshots
x24h; they are shipped to s3 then links are cleaned up. The error I get on
the node I'm trying to add to rack C is:
ERROR [main] 2017-08-12 23:54:51,546 CassandraDaemon.java:583 - Exception
encountered during startup
java.lang.RuntimeException: Error during boostrap: Stream failed
at
org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:87)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:11
66) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.jav
a:944) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:7
40) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:6
17) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:5
66) [apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
[apache-cassandra-2.1.15.jar:2.1.15]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(S
treamEventJMXNotifier.java:85) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.ex
ecute(MoreExecutors.java:297) ~[guava-16.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionLis
t.java:156) ~[guava-16.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:1
45) ~[guava-16.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture
.java:202) ~[guava-16.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResult
Future.java:209) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(Stre
amResultFuture.java:185) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java
:413) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.ja
va:700) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.jav
a:661) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(St
reamReceiveTask.java:179) ~[apache-cassandra-2.1.15.jar:2.1.15]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_112]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:11
42) ~[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:6
17) ~[na:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
WARN [StorageServiceShutdownHook] 2017-08-12 23:54:51,582
Gossiper.java:1462 - No local state or state is in silent shutdown, not
announcing shutdown
INFO [StorageServiceShutdownHook] 2017-08-12 23:54:51,582
MessagingService.java:734 - Waiting for messaging service to quiesce
INFO [ACCEPT-/10.40.17.114 <http://10.40.17.114> ] 2017-08-12 23:54:51,583
MessagingService.java:1020 - MessagingService has terminated the accept()
thread
And I got this on this same node when it was bootstrapping, I ran 'nodetool
netstats' just before it shutdown:
Receiving 377 files, 161928296443 bytes total. Already received 377
files, 161928296443 bytes total
TPStats on host that was streaming the data to this node:
Pool Name Active Pending Completed Blocked All
time blocked
MutationStage 1 1 4488289014 0
0
ReadStage 0 0 24486526 0
0
RequestResponseStage 0 0 3038847374
<tel:(303)%20884-7374> 0 0
ReadRepairStage 0 0 1601576 0
0
CounterMutationStage 0 0 68403 0
0
MiscStage 0 0 0 0
0
AntiEntropySessions 0 0 0 0
0
HintedHandoff 0 0 18 0
0
GossipStage 0 0 2786892 0
0
CacheCleanupExecutor 0 0 0 0
0
InternalResponseStage 0 0 61115 0
0
CommitLogArchiver 0 0 0 0
0
CompactionExecutor 4 83 304167 0
0
ValidationExecutor 0 0 78249 0
0
MigrationStage 0 0 94201 0
0
AntiEntropyStage 0 0 160505 0
0
PendingRangeCalculator 0 0 30 0
0
Sampler 0 0 0 0
0
MemtableFlushWriter 0 0 71270 0
0
MemtablePostFlush 0 0 175209 0
0
MemtableReclaimMemory 0 0 81222 0
0
Native-Transport-Requests 2 0 1983565628 0
9405444
Message type Dropped
READ 218
RANGE_SLICE 15
_TRACE 0
MUTATION 2949001
COUNTER_MUTATION 0
BINARY 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 8571
I can get a jstack if needed.
>
>> >
>> > Rather than troubleshoot this further, what I was thinking about doing was:
>> > - drop the replication factor on our keyspace to two
>
> Repair before you do this, or you'll lose your consistency guarantees
Given the load on the 2 nodes in rack C I'm hoping a repair will succeed.
>
>> > - hopefully this would reduce load on these two remaining nodes
>
> It should, racks awareness guarantees on replica per rack if rf==num racks, so
> right now those 2 c machines have 2.5x as much data as the others. This will
> drop that requirement and drop the load significantly
>
>> > - run repairs/cleanup across the cluster
>> > - then shoot these two nodes in the 'c' rack
>
> Why shoot the c instances? Why not drop RF and then add 2 more C instances,
> then increase RF back to 3, run repair, then Decom the extra instances in a
> and b?
>
>
Fair point. I was considering staying at RF two but I think with your
points below, I should reconsider.
>> > - run repairs/cleanup across the cluster
>> >
>> > Would this work with minimal/no disruption?
>
> The big risk of running rf=2 is that quorum==all - any gc pause or node
> restarting will make you lose HA or strong consistency guarantees.
>
>> > Should I update their "rack" before hand or after ?
>
> You can't change a node's rack once it's in the cluster, it SHOULD refuse to
> start if you do that
>
Got it.
>> > What else am I not thinking about?
>> >
>> > My main goal atm is to get back to where the cluster is in a clean
>> consistent state that allows nodes to properly bootstrap.
>> >
>> > Thanks for your help in advance.
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> > For additional commands, e-mail: user-help@cassandra.apache.org
>> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
Re: Dropping down replication factor
Posted by Brian Spindler <br...@gmail.com>.
Thanks for replying Jeff.
Responses below.
On Sat, Aug 12, 2017 at 8:33 PM Jeff Jirsa <jj...@gmail.com> wrote:
> Answers inline
>
> --
> Jeff Jirsa
>
>
> > On Aug 12, 2017, at 2:58 PM, brian.spindler@gmail.com wrote:
> >
> > Hi folks, hopefully a quick one:
> >
> > We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch. It's
> all in one region but spread across 3 availability zones. It was nicely
> balanced with 4 nodes in each.
> >
> > But with a couple of failures and subsequent provisions to the wrong az
> we now have a cluster with :
> >
> > 5 nodes in az A
> > 5 nodes in az B
> > 2 nodes in az C
> >
> > Not sure why, but when adding a third node in AZ C it fails to stream
> after getting all the way to completion and no apparent error in logs.
> I've looked at a couple of bugs referring to scrubbing and possible OOM
> bugs due to metadata writing at end of streaming (sorry don't have ticket
> handy). I'm worried I might not be able to do much with these since the
> disk space usage is high and they are under a lot of load given the small
> number of them for this rack.
>
> You'll definitely have higher load on az C instances with rf=3 in this
> ratio
> Streaming should still work - are you sure it's not busy doing something?
> Like building secondary index or similar? jstack thread dump would be
> useful, or at least nodetool tpstats
>
> Only other thing might be a backup. We do incrementals x1hr and snapshots
x24h; they are shipped to s3 then links are cleaned up. The error I get on
the node I'm trying to add to rack C is:
ERROR [main] 2017-08-12 23:54:51,546 CassandraDaemon.java:583 - Exception
encountered during startup
java.lang.RuntimeException: Error during boostrap: Stream failed
at
org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:87)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1166)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:944)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:740)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:617)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:391)
[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:566)
[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:655)
[apache-cassandra-2.1.15.jar:2.1.15]
Caused by: org.apache.cassandra.streaming.StreamException: Stream failed
at
org.apache.cassandra.streaming.management.StreamEventJMXNotifier.onFailure(StreamEventJMXNotifier.java:85)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
com.google.common.util.concurrent.Futures$4.run(Futures.java:1172)
~[guava-16.0.jar:na]
at
com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
~[guava-16.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
~[guava-16.0.jar:na]
at
com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
~[guava-16.0.jar:na]
at
com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
~[guava-16.0.jar:na]
at
org.apache.cassandra.streaming.StreamResultFuture.maybeComplete(StreamResultFuture.java:209)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.streaming.StreamResultFuture.handleSessionComplete(StreamResultFuture.java:185)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.streaming.StreamSession.closeSession(StreamSession.java:413)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.streaming.StreamSession.maybeCompleted(StreamSession.java:700)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.streaming.StreamSession.taskCompleted(StreamSession.java:661)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
org.apache.cassandra.streaming.StreamReceiveTask$OnCompletionRunnable.run(StreamReceiveTask.java:179)
~[apache-cassandra-2.1.15.jar:2.1.15]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_112]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_112]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
~[na:1.8.0_112]
at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_112]
WARN [StorageServiceShutdownHook] 2017-08-12 23:54:51,582
Gossiper.java:1462 - No local state or state is in silent shutdown, not
announcing shutdown
INFO [StorageServiceShutdownHook] 2017-08-12 23:54:51,582
MessagingService.java:734 - Waiting for messaging service to quiesce
INFO [ACCEPT-/10.40.17.114] 2017-08-12 23:54:51,583
MessagingService.java:1020 - MessagingService has terminated the accept()
thread
And I got this on this same node when it was bootstrapping, I ran 'nodetool
netstats' just before it shutdown:
Receiving 377 files, 161928296443 bytes total. Already received 377
files, 161928296443 bytes total
TPStats on host that was streaming the data to this node:
Pool Name Active Pending Completed Blocked All
time blocked
MutationStage 1 1 4488289014 0
0
ReadStage 0 0 24486526 0
0
RequestResponseStage 0 0 3038847374
<(303)%20884-7374> 0 0
ReadRepairStage 0 0 1601576 0
0
CounterMutationStage 0 0 68403 0
0
MiscStage 0 0 0 0
0
AntiEntropySessions 0 0 0 0
0
HintedHandoff 0 0 18 0
0
GossipStage 0 0 2786892 0
0
CacheCleanupExecutor 0 0 0 0
0
InternalResponseStage 0 0 61115 0
0
CommitLogArchiver 0 0 0 0
0
CompactionExecutor 4 83 304167 0
0
ValidationExecutor 0 0 78249 0
0
MigrationStage 0 0 94201 0
0
AntiEntropyStage 0 0 160505 0
0
PendingRangeCalculator 0 0 30 0
0
Sampler 0 0 0 0
0
MemtableFlushWriter 0 0 71270 0
0
MemtablePostFlush 0 0 175209 0
0
MemtableReclaimMemory 0 0 81222 0
0
Native-Transport-Requests 2 0 1983565628 0
9405444
Message type Dropped
READ 218
RANGE_SLICE 15
_TRACE 0
MUTATION 2949001
COUNTER_MUTATION 0
BINARY 0
REQUEST_RESPONSE 0
PAGED_RANGE 0
READ_REPAIR 8571
I can get a jstack if needed.
>
> >
> > Rather than troubleshoot this further, what I was thinking about doing
> was:
> > - drop the replication factor on our keyspace to two
>
> Repair before you do this, or you'll lose your consistency guarantees
>
Given the load on the 2 nodes in rack C I'm hoping a repair will succeed.
> > - hopefully this would reduce load on these two remaining nodes
>
> It should, racks awareness guarantees on replica per rack if rf==num
> racks, so right now those 2 c machines have 2.5x as much data as the
> others. This will drop that requirement and drop the load significantly
>
> > - run repairs/cleanup across the cluster
> > - then shoot these two nodes in the 'c' rack
>
> Why shoot the c instances? Why not drop RF and then add 2 more C
> instances, then increase RF back to 3, run repair, then Decom the extra
> instances in a and b?
>
>
> Fair point. I was considering staying at RF two but I think with your
points below, I should reconsider.
> > - run repairs/cleanup across the cluster
> >
> > Would this work with minimal/no disruption?
>
> The big risk of running rf=2 is that quorum==all - any gc pause or node
> restarting will make you lose HA or strong consistency guarantees.
>
> > Should I update their "rack" before hand or after ?
>
> You can't change a node's rack once it's in the cluster, it SHOULD refuse
> to start if you do that
>
> Got it.
> > What else am I not thinking about?
> >
> > My main goal atm is to get back to where the cluster is in a clean
> consistent state that allows nodes to properly bootstrap.
> >
> > Thanks for your help in advance.
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> > For additional commands, e-mail: user-help@cassandra.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
Re: Dropping down replication factor
Posted by Jeff Jirsa <jj...@gmail.com>.
Answers inline
--
Jeff Jirsa
> On Aug 12, 2017, at 2:58 PM, brian.spindler@gmail.com wrote:
>
> Hi folks, hopefully a quick one:
>
> We are running a 12 node cluster (2.1.15) in AWS with Ec2Snitch. It's all in one region but spread across 3 availability zones. It was nicely balanced with 4 nodes in each.
>
> But with a couple of failures and subsequent provisions to the wrong az we now have a cluster with :
>
> 5 nodes in az A
> 5 nodes in az B
> 2 nodes in az C
>
> Not sure why, but when adding a third node in AZ C it fails to stream after getting all the way to completion and no apparent error in logs. I've looked at a couple of bugs referring to scrubbing and possible OOM bugs due to metadata writing at end of streaming (sorry don't have ticket handy). I'm worried I might not be able to do much with these since the disk space usage is high and they are under a lot of load given the small number of them for this rack.
You'll definitely have higher load on az C instances with rf=3 in this ratio
Streaming should still work - are you sure it's not busy doing something? Like building secondary index or similar? jstack thread dump would be useful, or at least nodetool tpstats
>
> Rather than troubleshoot this further, what I was thinking about doing was:
> - drop the replication factor on our keyspace to two
Repair before you do this, or you'll lose your consistency guarantees
> - hopefully this would reduce load on these two remaining nodes
It should, racks awareness guarantees on replica per rack if rf==num racks, so right now those 2 c machines have 2.5x as much data as the others. This will drop that requirement and drop the load significantly
> - run repairs/cleanup across the cluster
> - then shoot these two nodes in the 'c' rack
Why shoot the c instances? Why not drop RF and then add 2 more C instances, then increase RF back to 3, run repair, then Decom the extra instances in a and b?
> - run repairs/cleanup across the cluster
>
> Would this work with minimal/no disruption?
The big risk of running rf=2 is that quorum==all - any gc pause or node restarting will make you lose HA or strong consistency guarantees.
> Should I update their "rack" before hand or after ?
You can't change a node's rack once it's in the cluster, it SHOULD refuse to start if you do that
> What else am I not thinking about?
>
> My main goal atm is to get back to where the cluster is in a clean consistent state that allows nodes to properly bootstrap.
>
> Thanks for your help in advance.
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org