You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Parag Patel <pp...@clearpoolgroup.com> on 2014/07/30 04:15:59 UTC
bootstrapping new nodes on 1.2.12
Hi,
It's taking a while to boostrap a 13th node into a 12 node cluster. The average node size is about 1.7TB. At the beginning of today we were close to .9TB on the new node and 12 hours later we're at 1.1TB. I figured it would have finished by now because when I was looking on OpsCenter, there were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look again now and those same nodes haven't progressed all day. Instead I see 9 more transfers (some of which are progressing).
1) Would anyone be able to help me interrupt this information from OpsCenter?
2) Is there anything I can do to speed this up?
Thanks,
Parag
RE: bootstrapping new nodes on 1.2.12
Posted by Parag Patel <pp...@clearpoolgroup.com>.
My understanding of a 9 second GC seems to be very off based on the gossip logs. Correct me if im wrong, but the “handshaking version” is just a log for it to attempt to connect to the other nodes?
Manual FGC
2:01:02 - Node1 full GC
2:01:25 - Node2 detects node1 DOWN
2:01:27 - Node2 handshaking version with Node1
2:01:32 - Node2 handshaking version with Node1 because failed previously
2:01:37 - Node2 handshaking version with Node1 because failed previously
2:01:39 - Node2 detects Node1 UP
Production FGC
9:30:45 - Node1 full gc
9:30:47 - Node2 detects Node1 DOWN
9:30:55 - handshaking version with Node1
9:31:00 - handshaking version with Node1 because failed previously
9:31:05 - handshaking version with Node1 because failed previously
9:31:10 - handshaking version with Node1 because failed previously
9:31:15 - handshaking version with Node1 because failed previously
9:31:20 - handshaking version with Node1 because failed previously
9:31:37 - Node2 – detects Node1 UP
From: Mark Reddy [mailto:mark.reddy@boxever.com]
Sent: Wednesday, July 30, 2014 9:41 AM
To: user@cassandra.apache.org
Subject: Re: bootstrapping new nodes on 1.2.12
Our Full GC’s take about 9 seconds. If we were to increase the phi_convict_threshold to not take a node offline for a 9 second unavailability, what negative side effects can there be?
When you observe these GC's do you also see the node being marked down and then back up ~9 seconds later? GC's can often happen and have no effect on gossip marking a node as down, in which case the streaming session will remain intact. The side effect of long GC's is increased latency from that node during that period.
Mark
On Wed, Jul 30, 2014 at 2:24 PM, Parag Patel <pp...@clearpoolgroup.com>> wrote:
As to why we do it, we need to reevaluate because the GC optimizations we’ve made recently probably don’t require it anymore. However, prior to our optimizations we observed a benefit at our peak time. When we force a GC, we don’t remove it from the ring. This seems like a fundamental flaw in our approach. Thanks for pointing this out. For the purposes of bootstrapping, we will disable the manual GC’s to make sure we don’t interrupt the joining process. However, one unpredictable problem can always remain – a Full GC happens causing the node to go offline and causing the bootstrap to fail. To solve this, we’ll try increasing the phi_convict_threshold.
Our Full GC’s take about 9 seconds. If we were to increase the phi_convict_threshold to not take a node offline for a 9 second unavailability, what negative side effects can there be?
Parag
From: Mark Reddy [mailto:mark.reddy@boxever.com<ma...@boxever.com>]
Sent: Wednesday, July 30, 2014 9:06 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: bootstrapping new nodes on 1.2.12
HI Parag,
I see this output my log many times over for 2 nodes. We have a cron entry across all clusters that force a full GC at 2 AM. node1 is due to Full GC that was scheduled (I can disable this). Node2 was due to a Full GC that occurred during our peak operation (these happen occasionally, we’ve been working to reduce them).
Firstly, why are you forcing a GC? Do you have sufficient evidence that Cassandra is not managing the heap in the way in which your application requires?
Also how are you accomplishing this full GC? Are you removing the node from the ring, forcing a GC and then adding it back in? Or are you forcing a GC while it is in the ring?
1) Will any node leaving the cluster while streaming force us to bootsrap all over again? If so, is this addressed in future versions?
If the node that is leaving the ring is streaming data to the bootstrapping node, yes, this will break the streaming session and no further streams will be started from that node. To my knowledge, there is nothing in newer / future versions that will prevent this.
2) We have too much data to migrate to run on non-production hours. How do we make it such that full GC’s don’t impact bootstrapping? Should we increase phi_convict_threshold ?
Again I'll need some more information around these manual GC's. But yes, increasing the phi value would reduce the chance of a node in the ring being marked down during a heavy gc cycle.
Mark
On Wed, Jul 30, 2014 at 1:41 PM, Parag Patel <pp...@clearpoolgroup.com>> wrote:
Mark,
I see this output my log many times over for 2 nodes. We have a cron entry across all clusters that force a full GC at 2 AM. node1 is due to Full GC that was scheduled (I can disable this). Node2 was due to a Full GC that occurred during our peak operation (these happen occasionally, we’ve been working to reduce them). Few Questions
1) Will any node leaving the cluster while streaming force us to bootsrap all over again? If so, is this addressed in future versions?
2) We have too much data to migrate to run on non-production hours. How do we make it such that full GC’s don’t impact bootstrapping? Should we increase phi_convict_threshold ?
Parag
From: Mark Reddy [mailto:mark.reddy@boxever.com<ma...@boxever.com>]
Sent: Wednesday, July 30, 2014 7:58 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: bootstrapping new nodes on 1.2.12
Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before?
This could indicate that the bootstrap process is hung due to a failed streaming session. Can you check your logs for the following line:
AbstractStreamSession.java (line 110) Stream failed because /xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active in background, but further streams won't be started)
If that is the case you will need to wipe the node and begin the bootstrapping process again
Mark
On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel <pp...@clearpoolgroup.com>> wrote:
Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before?
Side question – does a new node stream from other nodes in any particular order? Perhaps this is a coincidence, but if I were to sort my hostnames in alphabetical order, it’s currently streaming from the last 2.
From: Mark Reddy [mailto:mark.reddy@boxever.com<ma...@boxever.com>]
Sent: Wednesday, July 30, 2014 4:42 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: bootstrapping new nodes on 1.2.12
Hi Parag,
1) Would anyone be able to help me interrupt this information from OpsCenter?
At a high level bootstrapping a new node has two phases, streaming and secondary index builds. I believe OpsCenter will only report active streams, the pending stream will be listed as such in OpsCenter as well. In OpsCenter rather than looking at the Data Size check the used space on the Storage Capacity pie chart, this will show how much data is on disk but not necessarily live on the node yet.
Personally I would check 'nodetool netstats' to see what streams are remaining, this will list all active / pending stream and what files are to be streamed, at the moment you might just be streaming some very large files and once complete you will see a dramatic increase in data size.
If streaming is complete and you use secondary indexes, check 'nodetool compcationstats' for any secondary index builds that may be taking place.
2) Is there anything I can do to speed this up?
If you have the capacity you could increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml
If you don't have the capacity you could add more nodes to spread the data so you stream less in future.
Finally you could upgrade to 2.0.x as it contains a complete refactor of streaming and should make your streaming sessions more robust and transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286
Mark
On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel <pp...@clearpoolgroup.com>> wrote:
Hi,
It’s taking a while to boostrap a 13th node into a 12 node cluster. The average node size is about 1.7TB. At the beginning of today we were close to .9TB on the new node and 12 hours later we’re at 1.1TB. I figured it would have finished by now because when I was looking on OpsCenter, there were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look again now and those same nodes haven’t progressed all day. Instead I see 9 more transfers (some of which are progressing).
1) Would anyone be able to help me interrupt this information from OpsCenter?
2) Is there anything I can do to speed this up?
Thanks,
Parag
Re: bootstrapping new nodes on 1.2.12
Posted by Mark Reddy <ma...@boxever.com>.
>
> Our Full GC’s take about 9 seconds. If we were to increase the
> phi_convict_threshold to not take a node offline for a 9 second
> unavailability, what negative side effects can there be?
When you observe these GC's do you also see the node being marked down and
then back up ~9 seconds later? GC's can often happen and have no effect on
gossip marking a node as down, in which case the streaming session will
remain intact. The side effect of long GC's is increased latency from that
node during that period.
Mark
On Wed, Jul 30, 2014 at 2:24 PM, Parag Patel <pp...@clearpoolgroup.com>
wrote:
> As to why we do it, we need to reevaluate because the GC optimizations
> we’ve made recently probably don’t require it anymore. However, prior to
> our optimizations we observed a benefit at our peak time. When we force a
> GC, we don’t remove it from the ring. This seems like a fundamental flaw
> in our approach. Thanks for pointing this out. For the purposes of
> bootstrapping, we will disable the manual GC’s to make sure we don’t
> interrupt the joining process. However, one unpredictable problem can
> always remain – a Full GC happens causing the node to go offline and
> causing the bootstrap to fail. To solve this, we’ll try increasing the
> phi_convict_threshold.
>
>
>
> Our Full GC’s take about 9 seconds. If we were to increase the
> phi_convict_threshold to not take a node offline for a 9 second
> unavailability, what negative side effects can there be?
>
>
>
> Parag
>
>
>
>
>
> *From:* Mark Reddy [mailto:mark.reddy@boxever.com]
> *Sent:* Wednesday, July 30, 2014 9:06 AM
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: bootstrapping new nodes on 1.2.12
>
>
>
> HI Parag,
>
>
>
> I see this output my log many times over for 2 nodes. We have a cron
> entry across all clusters that force a full GC at 2 AM. node1 is due to
> Full GC that was scheduled (I can disable this). Node2 was due to a Full
> GC that occurred during our peak operation (these happen occasionally,
> we’ve been working to reduce them).
>
>
>
> Firstly, why are you forcing a GC? Do you have sufficient evidence that
> Cassandra is not managing the heap in the way in which your application
> requires?
>
>
>
> Also how are you accomplishing this full GC? Are you removing the node
> from the ring, forcing a GC and then adding it back in? Or are you forcing
> a GC while it is in the ring?
>
>
>
> 1) Will any node leaving the cluster while streaming force us to
> bootsrap all over again? If so, is this addressed in future versions?
>
>
>
> If the node that is leaving the ring is streaming data to the
> bootstrapping node, yes, this will break the streaming session and no further
> streams will be started from that node. To my knowledge, there is nothing
> in newer / future versions that will prevent this.
>
>
>
>
>
> 2) We have too much data to migrate to run on non-production hours.
> How do we make it such that full GC’s don’t impact bootstrapping? Should
> we increase phi_convict_threshold ?
>
>
>
> Again I'll need some more information around these manual GC's. But yes,
> increasing the phi value would reduce the chance of a node in the ring
> being marked down during a heavy gc cycle.
>
>
>
>
>
> Mark
>
>
>
> On Wed, Jul 30, 2014 at 1:41 PM, Parag Patel <pp...@clearpoolgroup.com>
> wrote:
>
> Mark,
>
>
>
> I see this output my log many times over for 2 nodes. We have a cron
> entry across all clusters that force a full GC at 2 AM. node1 is due to
> Full GC that was scheduled (I can disable this). Node2 was due to a Full
> GC that occurred during our peak operation (these happen occasionally,
> we’ve been working to reduce them). Few Questions
>
>
>
> 1) Will any node leaving the cluster while streaming force us to
> bootsrap all over again? If so, is this addressed in future versions?
>
> 2) We have too much data to migrate to run on non-production hours.
> How do we make it such that full GC’s don’t impact bootstrapping? Should
> we increase phi_convict_threshold ?
>
>
>
> Parag
>
>
>
>
>
> *From:* Mark Reddy [mailto:mark.reddy@boxever.com]
> *Sent:* Wednesday, July 30, 2014 7:58 AM
>
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: bootstrapping new nodes on 1.2.12
>
>
>
> Thanks for the detailed response. I checked ‘nodetool netstats’ and I see
> there are pending streams, all of which are stuck at 0%. I was expecting
> to see at least one output that was more than 0%. Have you seen this
> before?
>
>
>
> This could indicate that the bootstrap process is hung due to a failed
> streaming session. Can you check your logs for the following line:
>
>
>
> AbstractStreamSession.java (line 110) Stream failed because
> /xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active
> in background, but further streams won't be started)
>
>
>
> If that is the case you will need to wipe the node and begin the
> bootstrapping process again
>
>
>
>
>
> Mark
>
>
>
>
>
> On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel <pp...@clearpoolgroup.com>
> wrote:
>
> Thanks for the detailed response. I checked ‘nodetool netstats’ and I
> see there are pending streams, all of which are stuck at 0%. I was
> expecting to see at least one output that was more than 0%. Have you seen
> this before?
>
>
>
> Side question – does a new node stream from other nodes in any particular
> order? Perhaps this is a coincidence, but if I were to sort my hostnames
> in alphabetical order, it’s currently streaming from the last 2.
>
>
>
> *From:* Mark Reddy [mailto:mark.reddy@boxever.com]
> *Sent:* Wednesday, July 30, 2014 4:42 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: bootstrapping new nodes on 1.2.12
>
>
>
> Hi Parag,
>
>
>
> 1) Would anyone be able to help me interrupt this information from
> OpsCenter?
>
>
>
> At a high level bootstrapping a new node has two phases, streaming and
> secondary index builds. I believe OpsCenter will only report active
> streams, the pending stream will be listed as such in OpsCenter as well. In
> OpsCenter rather than looking at the Data Size check the used space on
> the Storage Capacity pie chart, this will show how much data is on disk but
> not necessarily live on the node yet.
>
>
>
> Personally I would check 'nodetool netstats' to see what streams are
> remaining, this will list all active / pending stream and what files are to
> be streamed, at the moment you might just be streaming some very large
> files and once complete you will see a dramatic increase in data size.
>
>
>
> If streaming is complete and you use secondary indexes, check 'nodetool
> compcationstats' for any secondary index builds that may be taking place.
>
>
>
>
>
> 2) Is there anything I can do to speed this up?
>
>
>
> If you have the capacity you could
> increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml
>
>
>
> If you don't have the capacity you could add more nodes to spread the data
> so you stream less in future.
>
>
>
> Finally you could upgrade to 2.0.x as it contains a complete refactor of
> streaming and should make your streaming sessions more robust and
> transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286
>
>
>
>
>
> Mark
>
>
>
> On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel <pp...@clearpoolgroup.com>
> wrote:
>
> Hi,
>
>
>
> It’s taking a while to boostrap a 13th node into a 12 node cluster. The
> average node size is about 1.7TB. At the beginning of today we were close
> to .9TB on the new node and 12 hours later we’re at 1.1TB. I figured it
> would have finished by now because when I was looking on OpsCenter, there
> were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look
> again now and those same nodes haven’t progressed all day. Instead I see 9
> more transfers (some of which are progressing).
>
>
>
> 1) Would anyone be able to help me interrupt this information from
> OpsCenter?
>
> 2) Is there anything I can do to speed this up?
>
>
>
> Thanks,
>
> Parag
>
>
>
>
>
>
>
>
>
RE: bootstrapping new nodes on 1.2.12
Posted by Parag Patel <pp...@clearpoolgroup.com>.
As to why we do it, we need to reevaluate because the GC optimizations we’ve made recently probably don’t require it anymore. However, prior to our optimizations we observed a benefit at our peak time. When we force a GC, we don’t remove it from the ring. This seems like a fundamental flaw in our approach. Thanks for pointing this out. For the purposes of bootstrapping, we will disable the manual GC’s to make sure we don’t interrupt the joining process. However, one unpredictable problem can always remain – a Full GC happens causing the node to go offline and causing the bootstrap to fail. To solve this, we’ll try increasing the phi_convict_threshold.
Our Full GC’s take about 9 seconds. If we were to increase the phi_convict_threshold to not take a node offline for a 9 second unavailability, what negative side effects can there be?
Parag
From: Mark Reddy [mailto:mark.reddy@boxever.com]
Sent: Wednesday, July 30, 2014 9:06 AM
To: user@cassandra.apache.org
Subject: Re: bootstrapping new nodes on 1.2.12
HI Parag,
I see this output my log many times over for 2 nodes. We have a cron entry across all clusters that force a full GC at 2 AM. node1 is due to Full GC that was scheduled (I can disable this). Node2 was due to a Full GC that occurred during our peak operation (these happen occasionally, we’ve been working to reduce them).
Firstly, why are you forcing a GC? Do you have sufficient evidence that Cassandra is not managing the heap in the way in which your application requires?
Also how are you accomplishing this full GC? Are you removing the node from the ring, forcing a GC and then adding it back in? Or are you forcing a GC while it is in the ring?
1) Will any node leaving the cluster while streaming force us to bootsrap all over again? If so, is this addressed in future versions?
If the node that is leaving the ring is streaming data to the bootstrapping node, yes, this will break the streaming session and no further streams will be started from that node. To my knowledge, there is nothing in newer / future versions that will prevent this.
2) We have too much data to migrate to run on non-production hours. How do we make it such that full GC’s don’t impact bootstrapping? Should we increase phi_convict_threshold ?
Again I'll need some more information around these manual GC's. But yes, increasing the phi value would reduce the chance of a node in the ring being marked down during a heavy gc cycle.
Mark
On Wed, Jul 30, 2014 at 1:41 PM, Parag Patel <pp...@clearpoolgroup.com>> wrote:
Mark,
I see this output my log many times over for 2 nodes. We have a cron entry across all clusters that force a full GC at 2 AM. node1 is due to Full GC that was scheduled (I can disable this). Node2 was due to a Full GC that occurred during our peak operation (these happen occasionally, we’ve been working to reduce them). Few Questions
1) Will any node leaving the cluster while streaming force us to bootsrap all over again? If so, is this addressed in future versions?
2) We have too much data to migrate to run on non-production hours. How do we make it such that full GC’s don’t impact bootstrapping? Should we increase phi_convict_threshold ?
Parag
From: Mark Reddy [mailto:mark.reddy@boxever.com<ma...@boxever.com>]
Sent: Wednesday, July 30, 2014 7:58 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: bootstrapping new nodes on 1.2.12
Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before?
This could indicate that the bootstrap process is hung due to a failed streaming session. Can you check your logs for the following line:
AbstractStreamSession.java (line 110) Stream failed because /xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active in background, but further streams won't be started)
If that is the case you will need to wipe the node and begin the bootstrapping process again
Mark
On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel <pp...@clearpoolgroup.com>> wrote:
Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before?
Side question – does a new node stream from other nodes in any particular order? Perhaps this is a coincidence, but if I were to sort my hostnames in alphabetical order, it’s currently streaming from the last 2.
From: Mark Reddy [mailto:mark.reddy@boxever.com<ma...@boxever.com>]
Sent: Wednesday, July 30, 2014 4:42 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: bootstrapping new nodes on 1.2.12
Hi Parag,
1) Would anyone be able to help me interrupt this information from OpsCenter?
At a high level bootstrapping a new node has two phases, streaming and secondary index builds. I believe OpsCenter will only report active streams, the pending stream will be listed as such in OpsCenter as well. In OpsCenter rather than looking at the Data Size check the used space on the Storage Capacity pie chart, this will show how much data is on disk but not necessarily live on the node yet.
Personally I would check 'nodetool netstats' to see what streams are remaining, this will list all active / pending stream and what files are to be streamed, at the moment you might just be streaming some very large files and once complete you will see a dramatic increase in data size.
If streaming is complete and you use secondary indexes, check 'nodetool compcationstats' for any secondary index builds that may be taking place.
2) Is there anything I can do to speed this up?
If you have the capacity you could increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml
If you don't have the capacity you could add more nodes to spread the data so you stream less in future.
Finally you could upgrade to 2.0.x as it contains a complete refactor of streaming and should make your streaming sessions more robust and transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286
Mark
On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel <pp...@clearpoolgroup.com>> wrote:
Hi,
It’s taking a while to boostrap a 13th node into a 12 node cluster. The average node size is about 1.7TB. At the beginning of today we were close to .9TB on the new node and 12 hours later we’re at 1.1TB. I figured it would have finished by now because when I was looking on OpsCenter, there were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look again now and those same nodes haven’t progressed all day. Instead I see 9 more transfers (some of which are progressing).
1) Would anyone be able to help me interrupt this information from OpsCenter?
2) Is there anything I can do to speed this up?
Thanks,
Parag
Re: bootstrapping new nodes on 1.2.12
Posted by Mark Reddy <ma...@boxever.com>.
HI Parag,
I see this output my log many times over for 2 nodes. We have a cron entry
> across all clusters that force a full GC at 2 AM. node1 is due to Full
> GC that was scheduled (I can disable this). Node2 was due to a Full GC
> that occurred during our peak operation (these happen occasionally, we’ve
> been working to reduce them).
Firstly, why are you forcing a GC? Do you have sufficient evidence that
Cassandra is not managing the heap in the way in which your application
requires?
Also how are you accomplishing this full GC? Are you removing the node from
the ring, forcing a GC and then adding it back in? Or are you forcing a GC
while it is in the ring?
1) Will any node leaving the cluster while streaming force us to
> bootsrap all over again? If so, is this addressed in future versions?
If the node that is leaving the ring is streaming data to the
bootstrapping node, yes, this will break the streaming session and no further
streams will be started from that node. To my knowledge, there is nothing
in newer / future versions that will prevent this.
2) We have too much data to migrate to run on non-production hours.
> How do we make it such that full GC’s don’t impact bootstrapping? Should
> we increase phi_convict_threshold ?
Again I'll need some more information around these manual GC's. But yes,
increasing the phi value would reduce the chance of a node in the ring
being marked down during a heavy gc cycle.
Mark
On Wed, Jul 30, 2014 at 1:41 PM, Parag Patel <pp...@clearpoolgroup.com>
wrote:
> Mark,
>
>
>
> I see this output my log many times over for 2 nodes. We have a cron
> entry across all clusters that force a full GC at 2 AM. node1 is due to
> Full GC that was scheduled (I can disable this). Node2 was due to a Full
> GC that occurred during our peak operation (these happen occasionally,
> we’ve been working to reduce them). Few Questions
>
>
>
> 1) Will any node leaving the cluster while streaming force us to
> bootsrap all over again? If so, is this addressed in future versions?
>
> 2) We have too much data to migrate to run on non-production hours.
> How do we make it such that full GC’s don’t impact bootstrapping? Should
> we increase phi_convict_threshold ?
>
>
>
> Parag
>
>
>
>
>
> *From:* Mark Reddy [mailto:mark.reddy@boxever.com]
> *Sent:* Wednesday, July 30, 2014 7:58 AM
>
> *To:* user@cassandra.apache.org
> *Subject:* Re: bootstrapping new nodes on 1.2.12
>
>
>
> Thanks for the detailed response. I checked ‘nodetool netstats’ and I see
> there are pending streams, all of which are stuck at 0%. I was expecting
> to see at least one output that was more than 0%. Have you seen this
> before?
>
>
>
> This could indicate that the bootstrap process is hung due to a failed
> streaming session. Can you check your logs for the following line:
>
>
>
> AbstractStreamSession.java (line 110) Stream failed because
> /xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active
> in background, but further streams won't be started)
>
>
>
> If that is the case you will need to wipe the node and begin the
> bootstrapping process again
>
>
>
>
>
> Mark
>
>
>
>
>
> On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel <pp...@clearpoolgroup.com>
> wrote:
>
> Thanks for the detailed response. I checked ‘nodetool netstats’ and I
> see there are pending streams, all of which are stuck at 0%. I was
> expecting to see at least one output that was more than 0%. Have you seen
> this before?
>
>
>
> Side question – does a new node stream from other nodes in any particular
> order? Perhaps this is a coincidence, but if I were to sort my hostnames
> in alphabetical order, it’s currently streaming from the last 2.
>
>
>
> *From:* Mark Reddy [mailto:mark.reddy@boxever.com]
> *Sent:* Wednesday, July 30, 2014 4:42 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: bootstrapping new nodes on 1.2.12
>
>
>
> Hi Parag,
>
>
>
> 1) Would anyone be able to help me interrupt this information from
> OpsCenter?
>
>
>
> At a high level bootstrapping a new node has two phases, streaming and
> secondary index builds. I believe OpsCenter will only report active
> streams, the pending stream will be listed as such in OpsCenter as well. In
> OpsCenter rather than looking at the Data Size check the used space on
> the Storage Capacity pie chart, this will show how much data is on disk but
> not necessarily live on the node yet.
>
>
>
> Personally I would check 'nodetool netstats' to see what streams are
> remaining, this will list all active / pending stream and what files are to
> be streamed, at the moment you might just be streaming some very large
> files and once complete you will see a dramatic increase in data size.
>
>
>
> If streaming is complete and you use secondary indexes, check 'nodetool
> compcationstats' for any secondary index builds that may be taking place.
>
>
>
>
>
> 2) Is there anything I can do to speed this up?
>
>
>
> If you have the capacity you could
> increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml
>
>
>
> If you don't have the capacity you could add more nodes to spread the data
> so you stream less in future.
>
>
>
> Finally you could upgrade to 2.0.x as it contains a complete refactor of
> streaming and should make your streaming sessions more robust and
> transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286
>
>
>
>
>
> Mark
>
>
>
> On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel <pp...@clearpoolgroup.com>
> wrote:
>
> Hi,
>
>
>
> It’s taking a while to boostrap a 13th node into a 12 node cluster. The
> average node size is about 1.7TB. At the beginning of today we were close
> to .9TB on the new node and 12 hours later we’re at 1.1TB. I figured it
> would have finished by now because when I was looking on OpsCenter, there
> were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look
> again now and those same nodes haven’t progressed all day. Instead I see 9
> more transfers (some of which are progressing).
>
>
>
> 1) Would anyone be able to help me interrupt this information from
> OpsCenter?
>
> 2) Is there anything I can do to speed this up?
>
>
>
> Thanks,
>
> Parag
>
>
>
>
>
>
>
RE: bootstrapping new nodes on 1.2.12
Posted by Parag Patel <pp...@clearpoolgroup.com>.
Mark,
I see this output my log many times over for 2 nodes. We have a cron entry across all clusters that force a full GC at 2 AM. node1 is due to Full GC that was scheduled (I can disable this). Node2 was due to a Full GC that occurred during our peak operation (these happen occasionally, we’ve been working to reduce them). Few Questions
1) Will any node leaving the cluster while streaming force us to bootsrap all over again? If so, is this addressed in future versions?
2) We have too much data to migrate to run on non-production hours. How do we make it such that full GC’s don’t impact bootstrapping? Should we increase phi_convict_threshold ?
Parag
From: Mark Reddy [mailto:mark.reddy@boxever.com]
Sent: Wednesday, July 30, 2014 7:58 AM
To: user@cassandra.apache.org
Subject: Re: bootstrapping new nodes on 1.2.12
Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before?
This could indicate that the bootstrap process is hung due to a failed streaming session. Can you check your logs for the following line:
AbstractStreamSession.java (line 110) Stream failed because /xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active in background, but further streams won't be started)
If that is the case you will need to wipe the node and begin the bootstrapping process again
Mark
On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel <pp...@clearpoolgroup.com>> wrote:
Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before?
Side question – does a new node stream from other nodes in any particular order? Perhaps this is a coincidence, but if I were to sort my hostnames in alphabetical order, it’s currently streaming from the last 2.
From: Mark Reddy [mailto:mark.reddy@boxever.com<ma...@boxever.com>]
Sent: Wednesday, July 30, 2014 4:42 AM
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: bootstrapping new nodes on 1.2.12
Hi Parag,
1) Would anyone be able to help me interrupt this information from OpsCenter?
At a high level bootstrapping a new node has two phases, streaming and secondary index builds. I believe OpsCenter will only report active streams, the pending stream will be listed as such in OpsCenter as well. In OpsCenter rather than looking at the Data Size check the used space on the Storage Capacity pie chart, this will show how much data is on disk but not necessarily live on the node yet.
Personally I would check 'nodetool netstats' to see what streams are remaining, this will list all active / pending stream and what files are to be streamed, at the moment you might just be streaming some very large files and once complete you will see a dramatic increase in data size.
If streaming is complete and you use secondary indexes, check 'nodetool compcationstats' for any secondary index builds that may be taking place.
2) Is there anything I can do to speed this up?
If you have the capacity you could increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml
If you don't have the capacity you could add more nodes to spread the data so you stream less in future.
Finally you could upgrade to 2.0.x as it contains a complete refactor of streaming and should make your streaming sessions more robust and transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286
Mark
On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel <pp...@clearpoolgroup.com>> wrote:
Hi,
It’s taking a while to boostrap a 13th node into a 12 node cluster. The average node size is about 1.7TB. At the beginning of today we were close to .9TB on the new node and 12 hours later we’re at 1.1TB. I figured it would have finished by now because when I was looking on OpsCenter, there were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look again now and those same nodes haven’t progressed all day. Instead I see 9 more transfers (some of which are progressing).
1) Would anyone be able to help me interrupt this information from OpsCenter?
2) Is there anything I can do to speed this up?
Thanks,
Parag
Re: bootstrapping new nodes on 1.2.12
Posted by Mark Reddy <ma...@boxever.com>.
>
> Thanks for the detailed response. I checked ‘nodetool netstats’ and I see
> there are pending streams, all of which are stuck at 0%. I was expecting
> to see at least one output that was more than 0%. Have you seen this
> before?
This could indicate that the bootstrap process is hung due to a failed
streaming session. Can you check your logs for the following line:
AbstractStreamSession.java (line 110) Stream failed because
/xxx.xxx.xxx.xxx died or was restarted/removed (streams may still be active
in background, but further streams won't be started)
If that is the case you will need to wipe the node and begin the
bootstrapping process again
Mark
On Wed, Jul 30, 2014 at 12:03 PM, Parag Patel <pp...@clearpoolgroup.com>
wrote:
> Thanks for the detailed response. I checked ‘nodetool netstats’ and I
> see there are pending streams, all of which are stuck at 0%. I was
> expecting to see at least one output that was more than 0%. Have you seen
> this before?
>
>
>
> Side question – does a new node stream from other nodes in any particular
> order? Perhaps this is a coincidence, but if I were to sort my hostnames
> in alphabetical order, it’s currently streaming from the last 2.
>
>
>
> *From:* Mark Reddy [mailto:mark.reddy@boxever.com]
> *Sent:* Wednesday, July 30, 2014 4:42 AM
> *To:* user@cassandra.apache.org
> *Subject:* Re: bootstrapping new nodes on 1.2.12
>
>
>
> Hi Parag,
>
>
>
> 1) Would anyone be able to help me interrupt this information from
> OpsCenter?
>
>
>
> At a high level bootstrapping a new node has two phases, streaming and
> secondary index builds. I believe OpsCenter will only report active
> streams, the pending stream will be listed as such in OpsCenter as well. In
> OpsCenter rather than looking at the Data Size check the used space on
> the Storage Capacity pie chart, this will show how much data is on disk but
> not necessarily live on the node yet.
>
>
>
> Personally I would check 'nodetool netstats' to see what streams are
> remaining, this will list all active / pending stream and what files are to
> be streamed, at the moment you might just be streaming some very large
> files and once complete you will see a dramatic increase in data size.
>
>
>
> If streaming is complete and you use secondary indexes, check 'nodetool
> compcationstats' for any secondary index builds that may be taking place.
>
>
>
>
>
> 2) Is there anything I can do to speed this up?
>
>
>
> If you have the capacity you could
> increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml
>
>
>
> If you don't have the capacity you could add more nodes to spread the data
> so you stream less in future.
>
>
>
> Finally you could upgrade to 2.0.x as it contains a complete refactor of
> streaming and should make your streaming sessions more robust and
> transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286
>
>
>
>
>
> Mark
>
>
>
> On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel <pp...@clearpoolgroup.com>
> wrote:
>
> Hi,
>
>
>
> It’s taking a while to boostrap a 13th node into a 12 node cluster. The
> average node size is about 1.7TB. At the beginning of today we were close
> to .9TB on the new node and 12 hours later we’re at 1.1TB. I figured it
> would have finished by now because when I was looking on OpsCenter, there
> were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look
> again now and those same nodes haven’t progressed all day. Instead I see 9
> more transfers (some of which are progressing).
>
>
>
> 1) Would anyone be able to help me interrupt this information from
> OpsCenter?
>
> 2) Is there anything I can do to speed this up?
>
>
>
> Thanks,
>
> Parag
>
>
>
>
>
RE: bootstrapping new nodes on 1.2.12
Posted by Parag Patel <pp...@clearpoolgroup.com>.
Thanks for the detailed response. I checked ‘nodetool netstats’ and I see there are pending streams, all of which are stuck at 0%. I was expecting to see at least one output that was more than 0%. Have you seen this before?
Side question – does a new node stream from other nodes in any particular order? Perhaps this is a coincidence, but if I were to sort my hostnames in alphabetical order, it’s currently streaming from the last 2.
From: Mark Reddy [mailto:mark.reddy@boxever.com]
Sent: Wednesday, July 30, 2014 4:42 AM
To: user@cassandra.apache.org
Subject: Re: bootstrapping new nodes on 1.2.12
Hi Parag,
1) Would anyone be able to help me interrupt this information from OpsCenter?
At a high level bootstrapping a new node has two phases, streaming and secondary index builds. I believe OpsCenter will only report active streams, the pending stream will be listed as such in OpsCenter as well. In OpsCenter rather than looking at the Data Size check the used space on the Storage Capacity pie chart, this will show how much data is on disk but not necessarily live on the node yet.
Personally I would check 'nodetool netstats' to see what streams are remaining, this will list all active / pending stream and what files are to be streamed, at the moment you might just be streaming some very large files and once complete you will see a dramatic increase in data size.
If streaming is complete and you use secondary indexes, check 'nodetool compcationstats' for any secondary index builds that may be taking place.
2) Is there anything I can do to speed this up?
If you have the capacity you could increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml
If you don't have the capacity you could add more nodes to spread the data so you stream less in future.
Finally you could upgrade to 2.0.x as it contains a complete refactor of streaming and should make your streaming sessions more robust and transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286
Mark
On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel <pp...@clearpoolgroup.com>> wrote:
Hi,
It’s taking a while to boostrap a 13th node into a 12 node cluster. The average node size is about 1.7TB. At the beginning of today we were close to .9TB on the new node and 12 hours later we’re at 1.1TB. I figured it would have finished by now because when I was looking on OpsCenter, there were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look again now and those same nodes haven’t progressed all day. Instead I see 9 more transfers (some of which are progressing).
1) Would anyone be able to help me interrupt this information from OpsCenter?
2) Is there anything I can do to speed this up?
Thanks,
Parag
Re: bootstrapping new nodes on 1.2.12
Posted by Mark Reddy <ma...@boxever.com>.
Hi Parag,
1) Would anyone be able to help me interrupt this information from
> OpsCenter?
At a high level bootstrapping a new node has two phases, streaming and
secondary index builds. I believe OpsCenter will only report active
streams, the pending stream will be listed as such in OpsCenter as well. In
OpsCenter rather than looking at the Data Size check the used space on
the Storage Capacity pie chart, this will show how much data is on disk but
not necessarily live on the node yet.
Personally I would check 'nodetool netstats' to see what streams are
remaining, this will list all active / pending stream and what files are to
be streamed, at the moment you might just be streaming some very large
files and once complete you will see a dramatic increase in data size.
If streaming is complete and you use secondary indexes, check 'nodetool
compcationstats' for any secondary index builds that may be taking place.
2) Is there anything I can do to speed this up?
If you have the capacity you could
increase stream_throughput_outbound_megabits_per_sec in your cassandra.yaml
If you don't have the capacity you could add more nodes to spread the data
so you stream less in future.
Finally you could upgrade to 2.0.x as it contains a complete refactor of
streaming and should make your streaming sessions more robust and
transparent: https://issues.apache.org/jira/browse/CASSANDRA-5286
Mark
On Wed, Jul 30, 2014 at 3:15 AM, Parag Patel <pp...@clearpoolgroup.com>
wrote:
> Hi,
>
>
>
> It’s taking a while to boostrap a 13th node into a 12 node cluster. The
> average node size is about 1.7TB. At the beginning of today we were close
> to .9TB on the new node and 12 hours later we’re at 1.1TB. I figured it
> would have finished by now because when I was looking on OpsCenter, there
> were 2 transfers remaining. 1 was at 0% and the other was at 2%. I look
> again now and those same nodes haven’t progressed all day. Instead I see 9
> more transfers (some of which are progressing).
>
>
>
> 1) Would anyone be able to help me interrupt this information from
> OpsCenter?
>
> 2) Is there anything I can do to speed this up?
>
>
>
> Thanks,
>
> Parag
>
>
>