You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Chris Goffinet (Created) (JIRA)" <ji...@apache.org> on 2011/11/11 06:00:51 UTC
[jira] [Created] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Support bringing up a new datacenter to existing cluster without repair
-----------------------------------------------------------------------
Key: CASSANDRA-3483
URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
Project: Cassandra
Issue Type: Bug
Affects Versions: 1.0.2
Reporter: Chris Goffinet
Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Chris Goffinet (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154898#comment-13154898 ]
Chris Goffinet commented on CASSANDRA-3483:
-------------------------------------------
Jonathan,
You are correct. Sylvain's proposal does satisfy this ticket. It doesn't solve the case of (2) where if you want to add lots of nodes in an existing DC, and you know the tokens they should be at, and want to join them all at once.
Our use case is, we actually add 60-100 nodes in one big capacity add. We would like to avoid the 120 second per node time frame. It's not a deal breaker though. We actually realized though if we are adding that many nodes to our cluster, with a large cluster already, we need to rebalance heavily anyway. Peter is almost done with the 'rebuild' patch, I'm assigning him to this ticket.
Our next big focus is improving the rebalancing of a cluster. We have a very large cluster and after adding 100 nodes every month or so, this becomes painful. Almost all of our nodes have over 600GB+ each. We have an application that will require us to be rebalancing at all times to reduce our hot spots.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13154502#comment-13154502 ]
Jonathan Ellis commented on CASSANDRA-3483:
-------------------------------------------
So, am I understanding correctly that we're talking about two different scenarios?
- Add new DC without repair
- Add many nodes to existing DC without RING_DELAY in between
I think Sylvain's proposal addresses the first nicely. So what I need help with is understanding what problem you're trying to solve with the second part. Dealing with overlapping ranges in node movement basically requires a rewrite of that subsystem (CASSANDRA-2434). But I suspect there is a "good enough" solution that we could find if I understood better what your pain point is here.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195448#comment-13195448 ]
Peter Schuller commented on CASSANDRA-3483:
-------------------------------------------
(It also contains the addition of a brace from CASSANDRA-3806; this is intentional to avoid pain.)
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3483:
--------------------------------------
Attachment: CASSANDRA-3483-1.0.txt
Attached is a version rebased against 1.0 (and tested).
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196090#comment-13196090 ]
Sylvain Lebresne commented on CASSANDRA-3483:
---------------------------------------------
bq. It addresses CASSANDRA-3807 by supporting fetch "consistency levels" (though only ONE is currently usable without patching)
I think a first issue is that current bootstrap does not fail if no node is alive for a given range, which arguably it should. I'm good with doing that, though it would be worth backporting to 1.0 too so it may be worth splitting that to a separate patch (or rather just create one for the fix in 1.0).
However, that does not solve the problem of bootstrap possibly breaking the consistency contract. The problem being that if we transfer a range from a node that happens to be lacking behind in term of consistency, and we end up replacing a node that was not lacking behind, we could break some consistency contracts. To fix that, I really only see only one solution right off the bat (which doesn't mean there isn't other): it is to ensure that for each range, we transfer it from (at least) the node we will replace for this range.
I believe the FetchConsistencyLevel of this patch is making an attempt to fix this by allowing to fetch from more than one node. While it does make it less likely to break consistency, unless we fetch from all nodes (and thus the one we'll replace), we cannot be sure we won't break the consistency level for people that say write at CL.ALL and read at CL.ONE. Overall, I fully agree this is a problem that we should fix someone, but I'm not sure the FetchConsistencyLevel is the right solution and even if it is it's a complicated enough problem that it's worth it's own ticket. I would agree that the problem with rebuild is a little bit different, but since anyway the patch introduce FCL without using it, let's keep that for later if that's ok.
bq. There is still some duplication between Bootstrapper.bootstrap() and StorageService.rebuild() in that both do the dance of iteration over tables to construct the final map. I am not really feeling that abstracting away that is a good idea to include in this ticket
I think it is, at least for a good chunk of it. It's not very complicated, it clearly improves code readability and since the patch already refactor that code I don't see a good reason to push that to later, especially if we agree it's worthwhile.
Attaching a v3 that 1) remove FetchConsistencyLevel for the reasons above and 2) move most of the details of creating the multimaps in RangeStreamer.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Fix For: 1.1
>
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt, CASSANDRA-3483-trunk-refactored-v2.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196264#comment-13196264 ]
Peter Schuller commented on CASSANDRA-3483:
-------------------------------------------
For the record I never intended to fix the general problem of bootstrapping never violating consistency. But in retrospect it's obvious how my choice of naming would make it sound like I did :) I agree it's a problem for its own ticket.
Thanks!
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Fix For: 1.1
>
> Attachments: 3483-cleanup.txt, 3483-v3.patch, CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt, CASSANDRA-3483-trunk-refactored-v2.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Chris Goffinet (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151854#comment-13151854 ]
Chris Goffinet commented on CASSANDRA-3483:
-------------------------------------------
Sorry, I had to re-read what Sylvain said for it to 'click'. So the process he proposes is as follows with RF of 3:
1. strategy options dc2:0
2. bring up new nodes in dc2 with auto_bootstrap off and token set
3. set strategy options dc2:3
4. run 'rebuild' on each node in dc2
this would handle the writes part.
i was kinda hoping though that we could modify the gossip state because I could very well see this playing into the case where you weren't adding DCs but wanted to add lots of nodes (60-100 like we do currently) ... and wanted to have them all added to existing DC.. having the bootstrap defined that way, would allow us to bootstrap nodes as we please in existing DCs, bring them to ring all at once to have a consistent state without taking on traffic until they changed states. Where as this proposal wouldn't be able to satisfy that use case.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3483:
--------------------------------------
Attachment: CASSANDRA-3483-trunk-rebase2.txt
I do. I'm sorry for the delay, this has been nagging me for quite some time. It's not forgotten, I have just been inundated with urgent stuff to do.
I'm attaching a fresh rebase against current trunk and I hope to submit an improved version later tonight (keyword being "hope").
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168924#comment-13168924 ]
Sylvain Lebresne commented on CASSANDRA-3483:
---------------------------------------------
I haven't applied the patch yet, it needs rebase and preferably against trunk since that is the likely target for this, but a few comments.
We could have more reuse of code between Boostrapper ant the rebuild command. Typically:
* RangeStreamer.getAllRangeWithSourcesFor does essentially the same thing that Boostrapper.getRangesWithSources, so it would be nice to do some reuse.
* In rebuild, we essentially have the code of Boostrapper.getWorkMap, again would be nice to do some code reuse.
I think we should move all of those in RangeStreamer and ultimately Boostrapper.boostrap() should be just one call to rebuild with the right arguments (mostly the correct tokenMetada instance and the "myRange" collection).
A few nits:
* rebuild code could be simplified slightly by using StorageService.getLocalRanges()
* rebuild doesn't fully respect the code style.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Chris Goffinet (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13148280#comment-13148280 ]
Chris Goffinet commented on CASSANDRA-3483:
-------------------------------------------
Some discussion from irc:
{noformat}
23:43 < goffinet> has datastax ever had a customer add a new datacenter to an existing cluster? No docs or info on web suggest anyone has done this before
23:44 < driftx> yeah
23:44 < goffinet> how is it done? we are running a case where if i modify strategy options before adding nodes, writes will fail since no endpoints for DC have been added
23:44 < goffinet> we were expecting this might work because we want to bootstrap the new DC to the existing cluster
23:44 < goffinet> take on writes + stream data with RF factor
23:45 < driftx> general best practice is (jbellis can correct if I'm outdated) add the dc at rf:0, add the nodes/update snitch, repair
23:45 < driftx> err, update rf, repair
23:46 < goffinet> yeah mind if i open up a jira? that seems extreme to make the cluster do that .. ?
23:46 < goffinet> or is repair smart enough to just stream ranges instead of AES?
23:46 < driftx> 'instead of AES?' that's what repair is, but if just streams ranges
23:46 < driftx> s/if/it/
23:47 < goffinet> right but AES builds merkle tree, scans through all data ?
23:47 < goffinet> isn't bootstrap a different operation?
23:47 < goffinet> when streaming just sstables
23:47 < driftx> yeah, it is
23:47 < goffinet> yeah thats more heavy. dont understand why we couldnt use that instead
23:47 < goffinet> like bootstrap
23:48 < stuhood> now that i think about it, it doesn't really make sense that a CL.ONE write fails if a DC isn't available
23:48 < stuhood> independent of the bootstrap case, that sounds like the real issue
23:49 < stuhood> goffinet: ^
23:50 < driftx> hmm, yeah that doesn't
23:50 < driftx> but the problem with bootstrapping a dc is the first node you bootstrap gets everything
23:50 < goffinet> stuhood: yeah. it was complaining about not enough endpoints
23:50 < goffinet> driftx: why is that? if you are doubling the cluster, and assign the tokens manually ?
23:51 < driftx> still have to do them 2 mins apart, and they're probably going to be part of the same replica set which I think is troublesome too
23:51 < goffinet> driftx: maybe we can make repair a bit more intelligent? if no data exists on the node .. just stream the ranges instead of using AES
23:52 < driftx> problem is we're pushing AES to do the entire replica set (which is nearly does now)
23:52 < stuhood> goffinet: it shouldn't be as heavyweight as you're thinking
23:53 < goffinet> stuhood: but we have a way currently that is less heavy
23:53 < goffinet> i dont understand why we couldnt use that method
23:53 < stuhood> not implemented =)
23:53 < goffinet> don't cut corners :)
23:53 < stuhood> human time vs cpu time =P
23:54 < driftx> you could almost do something like #3452 and then have a jmx call to say 'ok, finish'
23:54 < CassBotJr> https://issues.apache.org/jira/browse/CASSANDRA-3452 : Create an 'infinite bootstrap' mode for sampling live traffic
23:54 < driftx> except the first one that tries is going to have every node pound it with all the writes
23:54 < goffinet> driftx: ill make a jira ticket so we can discuss there, it doesn't seem like it would be too much trouble to support this use case
23:54 < goffinet> we'd be happy to write the patch after some input
23:55 < driftx> trickier than it sounds I'll bet, but sgtm
23:57 < stuhood> alternatively, is now the right time to add back group bootstrap?
23:58 < stuhood> so you'd 1) add the dc to the strategy, 2) do a group bootstrap of the entire dc
23:58 < stuhood> would also have to fix the CL.ONE problem though.
23:59 < goffinet> how did group bootstrap work again?
23:59 < driftx> #2434 is relevant
23:59 < CassBotJr> https://issues.apache.org/jira/browse/CASSANDRA-2434 : range movements can violate consistency
--- Day changed Fri Nov 11 2011
00:00 < stuhood> goffinet: bootstrapping many nodes at once without the 2 minute wait
00:01 < goffinet> why was it removed?
00:01 < stuhood> used zookeeper
00:01 < goffinet> oh.
00:01 < stuhood> but come to think of it, removing the 2 minute wait would seem to be relatively easy
00:02 < goffinet> stuhood, i thought the 2 minute wait was just waiting for ring state to settle?
00:02 < goffinet> before it streamed from nodes
00:02 < stuhood> goffinet: yea: you could form a "group" bootstrap by inverting things and waiting until you -hadn't- seen a new node in 2-10 minutes before you chose a token and started bootstrapping
00:03 < stuhood> so, not terribly simple, but.
00:04 < stuhood> you'd basically have a bunch of nodes sitting around waiting until no new nodes started, and then they have to deterministically choose tokens.
00:05 < goffinet> yes
00:05 < stuhood> well, alternatively, you wouldn't need a new way to deterministically choose tokens
00:05 < stuhood> (easier)
00:05 < stuhood> no… scratch that. you would need a way
00:05 < stuhood> for this DC case, all of the nodes are entering an empty ring
00:06 < stuhood> so the group would need to choose something balanced
00:06 < goffinet> empty ring?
00:06 < stuhood> yea, essentially… there are no tokens in that dc
00:06 < goffinet> but we were going to provide the tokens manually?
00:06 < goffinet> were you thinking of making it automatic?
00:07 < stuhood> yea. fixing bootstrapping groups of nodes would make automatic safe again
00:08 < stuhood> so… whatever state a node is in when it is sitting and waiting for enough information to choose a token, it should just stay that way and watch what other nodes enter that state
00:08 < goffinet> so i have a question about the 120 second window you have to wait..
00:09 < stuhood> mm
00:09 < driftx> hmm, what if they started up at rf:0 but stayed in some dead state (hibernate might work) without doing anything until you changed the rf, then actually bootstrapped?
00:09 < goffinet> so imagine i startup all the nodes in DC2 at same time, does join_ring=false not grab gossip info at all? I was thinking it would be good if we could just start gossip on all nodes, but until operator says 'go' then i could bootstrap them all at same time
00:09 < goffinet> since i would only have to wait at most 120 seconds before kicking them all off
00:10 < stuhood> driftx: yea, that could work too… but you'd still need to choose tokens. (also, the rf=0 thing shouldn't be necessary, right? that's the CL.ONE bug)
00:11 < driftx> well, you really want to choose tokens anyway
00:11 < stuhood> goffinet: it does get gossip… i think that's basically equivalent to the pre-join state
00:11 < driftx> I guess you don't need rf=0 if all the nodes are in hibernate
00:12 < goffinet> yeah i think you do need hibernate in this case, because if i set tokens upfront, i want all nodes to know about ATL ones too
00:12 < goffinet> before i kick off bootstrap
00:12 < stuhood> driftx: i'm confused… what is the difference between rf=0 and not being there?
00:12 < stuhood> is that a workaround for the CL.ONE bug?
00:13 < driftx> you know there's a dc with rf:0, can add one with impacting anything
00:13 < driftx> err, without
00:14 ?? boaz__ (0819c319@gateway/web/freenode/ip.8.25.195.25) has joined #cassandra-dev
00:14 < stuhood> so what was the point of adding it? that's why i'm confused...
00:14 < goffinet> im fine with rf:0, its so you can add the nodes to the cluster before calling repair
00:14 < goffinet> before you add nodes
00:15 < driftx> because the dc is in the schema
00:15 < driftx> so you need it there to have nodes be in it
00:15 < stuhood> ah
00:16 < goffinet> driftx: any reason why we couldnt just fix that? so dc2:3 wont throw an error if nodes are down?
00:16 < goffinet> that way you would needed to do two steps
00:16 < goffinet> dc2:0, add nodes, dc2:3
00:16 < goffinet> wouldn't*
00:16 < driftx> I don't understand, you can already do that
00:17 < driftx> you just have to repair afterwards
00:17 < goffinet> it throws an error currently? if you set dc2:3 and no nodes exist for dc2
00:17 < goffinet> we'll double check on that
00:18 < goffinet> for writes
00:18 < driftx> oh, it does
00:19 < driftx> but only for writes
00:19 < goffinet> yeah
00:19 < goffinet> so thats fine, thats fixable
00:19 < goffinet> im just curious about a) how can we bootstrap nodes without 120s delays between N nodes b) stream from DC1 without AES
00:21 < stuhood> goffinet: if you figure out a, i don't think b is necessary?
00:22 < stuhood> assuming they are aware of the other joining nodes, and can all join the same range
00:22 < stuhood> that would be the keystone for some kind of group bootstrap
00:23 < goffinet> let me test out join_ring, because im curious. if join_ring=false still gossips but doesnt offically join.. it would be nice if node 2 in DC2 knew about that node too somehow?
00:23 < driftx> that's why I proposed cheating, add them all as non-members, then ask them to bootstrap
00:23 < goffinet> because then .. i could just run a command on each node at same time
00:23 < goffinet> since they all know about each other in a hibernate state
00:23 < goffinet> driftx: yes i like that
00:24 < driftx> private void joinTokenRing(int delay) throws IOException, org.apache.cassandra.config.ConfigurationException
00:24 < driftx> {
00:24 < driftx> logger_.info("Starting up server gossip");
00:24 < driftx> they don't use gossip with join_ring off
00:24 < stuhood> but will that actually allow them to all join the same range?
00:24 < goffinet> okay cool, yeah we would need to make it join in that special state then
00:25 < stuhood> i think there is an edgecase here… if multiple nodes are joining the same range, and one of them fails, then should they all fail?
00:25 < driftx> no, it basically saves you server startup time that is not ring-related :)
00:25 < goffinet> stuhood, they all know the tokens ahead of time?
00:25 < goffinet> they just need to know the current global state of things
00:25 < stuhood> goffinet: right, but if they are streaming the range that they will be responsible for...
00:26 ?? mw1 (~Adium@8.25.195.29) has quit (Quit: Leaving.)
00:26 < stuhood> Joining nodes don't stick around if they fail
00:26 < goffinet> they shouldnt be allowed to do that until they joined ?
00:26 < stuhood> nah, you stream while you are joining… unless you are talking about repair
00:26 < goffinet> stuhood: was that removed? i thought u had to still remove the node
00:26 < goffinet> using the new options in 1.0
00:26 < stuhood> don't know about 1.0
00:27 < driftx> no, a failed non-member is just a fat client and disappears
00:27 < goffinet> but i thought there was a timeout for fat client ?
00:27 < goffinet> is it 30s or something?
00:27 < driftx> yes
00:28 < goffinet> so nodes that arent fat clients, why might we remove them ? if we didnt..
00:28 < goffinet> and let the operator do it
00:28 < goffinet> or have a larger timeout
00:28 < goffinet> might make this a non-issue
00:28 < driftx> what does a larger timeout/keeping them around buy you?
00:29 < goffinet> because if they go away, and i bootstrap after they failed, wont my view of ring be skewed?
00:29 < stuhood> driftx: i guess in this case, the node would resume bootstrapping from where it left off
00:29 < driftx> it would've missed writes in the meantime and require a repair afterwards anyway
00:29 < stuhood> sorry… "resume" in the sense of "start over", but yea
00:31 < stuhood> that would be a pretty big change, but it might make sense
00:31 < goffinet> stuhood: what would you change
00:31 < stuhood> what you said, about nodes in joining staying in joining
00:31 < stuhood> so if the machine restarts, it begins joining at the same position again
00:33 < goffinet> if we supported that + letting nodes gossip in hibernate, would allow us to add capacity at operator control
{noformat}
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151933#comment-13151933 ]
Sylvain Lebresne commented on CASSANDRA-3483:
---------------------------------------------
So you're proposing to add support for bootstrapping multiple nodes together. I'm not against that, it would be nice and that would give you 90% of what this ticket is about (you'd have to add the ability to multi-boostrap *and* add a DC/augment the replication faction at the same time). But it is orders of magnitude more complicated than what I'm suggesting. Which is not a problem in itself given it's a broader solution, but it means we'll have multi-node boostrap at best for 1.1, while I'm pretty sure I can get the 'rebuild' command wrote in like an hour (and I see no reason why it couldn't be put in the 1.0 series).
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151503#comment-13151503 ]
Brandon Williams commented on CASSANDRA-3483:
---------------------------------------------
We would still need to put the nodes into a 'bootstrap' state to get incoming writes forwarded to them, otherwise you have to repair in the end anyway.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-3483) Support bringing up
a new datacenter to existing cluster without repair
Posted by "Chris Goffinet (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151854#comment-13151854 ]
Chris Goffinet edited comment on CASSANDRA-3483 at 11/17/11 6:31 AM:
---------------------------------------------------------------------
Sorry, I had to re-read what Sylvain said for it to 'click'. So the process he proposes is as follows with RF of 3:
1. strategy options dc2:0
2. bring up new nodes in dc2 with auto_bootstrap off and token set
3. set strategy options dc2:3
4. run 'rebuild' on each node in dc2
this would handle the writes part.
i was kinda hoping though that we could modify the gossip state because I could very well see this playing into the case where you weren't adding DCs but wanted to add lots of nodes (60-100 like we do currently) ... and wanted to have them all added to existing DC.. having the bootstrap defined that way, would allow us to bootstrap nodes as we please in existing DCs, bring them all to the ring at once to have a consistent state without taking on traffic until they transitioned states (joining/normal). Where as this proposal wouldn't be able to satisfy that use case.
was (Author: lenn0x):
Sorry, I had to re-read what Sylvain said for it to 'click'. So the process he proposes is as follows with RF of 3:
1. strategy options dc2:0
2. bring up new nodes in dc2 with auto_bootstrap off and token set
3. set strategy options dc2:3
4. run 'rebuild' on each node in dc2
this would handle the writes part.
i was kinda hoping though that we could modify the gossip state because I could very well see this playing into the case where you weren't adding DCs but wanted to add lots of nodes (60-100 like we do currently) ... and wanted to have them all added to existing DC.. having the bootstrap defined that way, would allow us to bootstrap nodes as we please in existing DCs, bring them to ring all at once to have a consistent state without taking on traffic until they changed states. Where as this proposal wouldn't be able to satisfy that use case.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3483:
--------------------------------------
Attachment: CASSANDRA-3483-trunk-noredesign.txt
Attaching version rebased to trunk but not yet re-factored.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Chris Goffinet (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Chris Goffinet reassigned CASSANDRA-3483:
-----------------------------------------
Assignee: Peter Schuller
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13155592#comment-13155592 ]
Peter Schuller commented on CASSANDRA-3483:
-------------------------------------------
We ended up going for the simpler rebuild patch as Chris hinted at. I'll quote myself:
{quote}
I've been looking at this some more.
Here's the proposal so far:
We introduce a GHOST state, in which a node receives neither reads nor writes. This allows us to bring in a group of nodes in the ring without suffering any ill effects. It is completely invisible to reads and writes, and will never count towards e.g. consistency level.
Once all ghosts are ready, we can start bootstrapping, taking ghost nodes into accounts for purposes of determining which range we are responsible for, but streaming only from non-ghost nodes. This accomplishes the goal of not transferring more data than necessary.
In order to avoid a bootstrapping node from taking writes for more than it's eventual share, we'd have to make the write endpoints be aware of ghost nodes. This is dosable, but not critical since we're bisecting a range that was previously handled by a single node anyway so the traffic would be managable. It would just be cleaner to not have to cleanup afterwards.
Once we transition from bootstrapping to being "up", we have a bigger problem however. If the read paths and write paths are only aware of non-ghost nodes, the read/write paths would think that these nodes had more ownership than they really do.
So we must really be taking into account the other nodes in the read/write path as well - but only when determining ownership of a completely bootstrapped node that was previously a ghost. That means we must distinguish between a "normally up" node and one that's been bootstraped from ghost state (call it "ghost strapped").
This suddenly gets complex. The process of group bootstrap would then be:
Add a bunch of nodes in GHOST state
Bootstrap all of them, each of them going into GHOSTSTRAPPED state
Once all are GHOSTSTRAPPED, we can safely transntion from GHOSTSTRAPPED to normal/up
Is there a simpler solution?
{quote}
After some additional discussion we felt this was adding to much complexity and potential edge cases/bugs that it became more cost-effective to just go with the simple rebuild for our immediate needs, hoping to address the problem of adding lots of nodes to a DC separately in some other way.
A patch is forthcoming soon.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Chris Goffinet (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151784#comment-13151784 ]
Chris Goffinet commented on CASSANDRA-3483:
-------------------------------------------
Yeah as Brandon mentioned, we would still want to go into the bootstrap state to get those writes. This would also allow us to add capacity in the same way, if we manually pick tokens (auto bootstrap is kinda worthless IMO) to existing DC as well. We can just fire off the bootstrap command from nodetool as needed.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Sylvain Lebresne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-3483:
----------------------------------------
Attachment: 3483-v3.patch
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Fix For: 1.1
>
> Attachments: 3483-v3.patch, CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt, CASSANDRA-3483-trunk-refactored-v2.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13195449#comment-13195449 ]
Peter Schuller commented on CASSANDRA-3483:
-------------------------------------------
I borked the unit test, will address that too.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3483:
--------------------------------------
Attachment: CASSANDRA-3483-trunk-refactored-v1.txt
{{CASSANDRA\-3483\-trunk\-refactored\-v1.txt}} addresses the duplication between BootStrapper and RangeStreamer.
Next patch will address rebuild/getworkmap duplication.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Chris Goffinet (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152496#comment-13152496 ]
Chris Goffinet commented on CASSANDRA-3483:
-------------------------------------------
Sylvain, it looks like a state 'HIBERNATE' exists in GOSSIP already based on recovering a dead node in 1.0. Do you have a preference on the name of the new state if we attempted adding multiple nodes patch? WAITING? STANDBY?
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3483:
--------------------------------------
Reviewer: slebresne
Affects Version/s: (was: 1.0.2)
Fix Version/s: 1.1
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Fix For: 1.1
>
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt, CASSANDRA-3483-trunk-refactored-v2.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168976#comment-13168976 ]
Peter Schuller commented on CASSANDRA-3483:
-------------------------------------------
I'll get it rebased once it's otherwise okay.
As for re-use: I had intermediate versions that tried to do this, but ever time I ended up realizing that it was exploding in verbosity at the point where I was using the abstraction so it didn't actually help. However, I think there were a few changes towards the end after which I didn't re-evaluate.
I'll look at it again and see what I can do.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Jeremiah Jordan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162857#comment-13162857 ]
Jeremiah Jordan commented on CASSANDRA-3483:
--------------------------------------------
Once this option is in, is this the procedure for running rebuild (with 4 changed to 'rebuild dc1')?
{quote}
1. strategy options dc2:0
2. bring up new nodes in dc2 with auto_bootstrap off and token set
3. set strategy options dc2:3
4. run 'rebuild' on each node in dc2
{quote}
Do we need to stagger issuing the rebuild commands or can they be run all at once?
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13151499#comment-13151499 ]
Sylvain Lebresne commented on CASSANDRA-3483:
---------------------------------------------
I think it wouldn't be crazy and actually very (very) simple to add a new nodetool command (rebuild?) that would basically have the node asks the other replicas to stream all there data to him (for the correct ranges obviously). In other words, a command that force the streaming part of bootstrap without all the join ring part. Or another way to say is to have the streaming part of a repair but without the validation part.
The method to add a new DC would be the same as today except that repair would be replaced by this new operation.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3483:
--------------------------------------
Attachment: CASSANDRA-3483-0.8-prelim.txt
Here is a patch rebased against 0.8 for cursory review. I do not expect this to go into 0.8, and in fact I have not tested this patch other than build against vanilla 0.8 (the original patch is tested, but against our internal 0.8).
If there are no concerns with the overall implementation, I'll submit a rebased version for 1.0/trunk.
There are two components of the change:
* Breaking out the streaming part of BootStrapper into a separate RangeStreamer. Change BootStrapper to use that.
* Implement the rebuild command on top of RangeStreamer.
There are two ways to invoke rebuild:
{code}
nodetool rebuild
nodetool rebuild nameofdc
{code}
The first form streams from nearest endpoints, while the latter streams from nearest endpoints in the specified data center.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Sylvain Lebresne (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196128#comment-13196128 ]
Sylvain Lebresne commented on CASSANDRA-3483:
---------------------------------------------
bq. This is CASSANDRA-2434 isn't it?
it is.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Fix For: 1.1
>
> Attachments: 3483-v3.patch, CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt, CASSANDRA-3483-trunk-refactored-v2.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Jonathan Ellis (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-3483:
--------------------------------------
Attachment: 3483-cleanup.txt
cleanup patch addressing mostly typos and style. only substantial code change was to RangeStreamer.getRangeFetchMap. Also, moved OperationType.REBUILD to the end of the enum to make sure we don't break anything depending on ordinal.
+1 from me otherwise.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Fix For: 1.1
>
> Attachments: 3483-cleanup.txt, 3483-v3.patch, CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt, CASSANDRA-3483-trunk-refactored-v2.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13162971#comment-13162971 ]
Peter Schuller commented on CASSANDRA-3483:
-------------------------------------------
Yes, that looks good.
And yes, you can run rebuilds concurrently as long as you're comfortable with the amount of bandwidth you'll be pushing and the load you'll be putting on the source nodes.
However, if you expect to see reasonable performance and streaming at full speed to all nodes, you also need CASSANDRA-3494.
Regardless: I strongly recommend testing this with your exact version of Cassandra before trying it for real.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13152634#comment-13152634 ]
Peter Schuller commented on CASSANDRA-3483:
-------------------------------------------
I was thinking of GHOST too after I realized we'll need a state where the node receives neither writes nor reads (more details here later).
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Brandon Williams (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160223#comment-13160223 ]
Brandon Williams commented on CASSANDRA-3483:
---------------------------------------------
bq. If there are no concerns with the overall implementation, I'll submit a rebased version for 1.0/trunk.
This looks good to me, I like the RangeStreamer abstraction.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Peter Schuller updated CASSANDRA-3483:
--------------------------------------
Attachment: CASSANDRA-3483-trunk-refactored-v2.txt
{{CASSANDRA\-3483\-trunk\-refactored\-v2.txt}} I believe addresses the concerns, plus makes other improvements. I'm much more happy with this one.
It addresses CASSANDRA-3807 by supporting fetch "consistency levels" (though only ONE is currently usable without patching), and the filtering of hosts is abstracted out.
There is still some duplication between {{Bootstrapper.bootstrap()}} and {{StorageService.rebuild()}} in that both do the dance of iteration over tables to construct the final map. I am not really feeling that abstracting away that is a good idea to include in this ticket, though I think it's worthwhile doing at some point separately.
The unit test is fixed; my adjustment of it was wrong because I wasn't picking pending ranges (in the test).
I've tested both rebuild and bootstrap in a 3 node cluster.
I've added some more logging than what is typically the case; there have been several cases where I wished streaming was logged in more detail at INFO, particularly when bootstrapping or rebuilding. I think it's worthwhile to get that in while at it.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt, CASSANDRA-3483-trunk-refactored-v2.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Peter Schuller (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13168975#comment-13168975 ]
Peter Schuller commented on CASSANDRA-3483:
-------------------------------------------
I'll get it rebased once it's otherwise okay.
As for re-use: I had intermediate versions that tried to do this, but ever time I ended up realizing that it was exploding in verbosity at the point where I was using the abstraction so it didn't actually help. However, I think there were a few changes towards the end after which I didn't re-evaluate.
I'll look at it again and see what I can do.
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 1.0.2
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Attachments: CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3483) Support bringing up a new
datacenter to existing cluster without repair
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13196127#comment-13196127 ]
Jonathan Ellis commented on CASSANDRA-3483:
-------------------------------------------
bq. Overall, I fully agree this is a problem that we should fix someone, but I'm not sure the FetchConsistencyLevel is the right solution and even if it is it's a complicated enough problem that it's worth it's own
This is CASSANDRA-2434 isn't it?
> Support bringing up a new datacenter to existing cluster without repair
> -----------------------------------------------------------------------
>
> Key: CASSANDRA-3483
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3483
> Project: Cassandra
> Issue Type: Bug
> Reporter: Chris Goffinet
> Assignee: Peter Schuller
> Fix For: 1.1
>
> Attachments: 3483-v3.patch, CASSANDRA-3483-0.8-prelim.txt, CASSANDRA-3483-1.0.txt, CASSANDRA-3483-trunk-noredesign.txt, CASSANDRA-3483-trunk-rebase2.txt, CASSANDRA-3483-trunk-refactored-v1.txt, CASSANDRA-3483-trunk-refactored-v2.txt
>
>
> Was talking to Brandon in irc, and we ran into a case where we want to bring up a new DC to an existing cluster. He suggested from jbellis the way to do it currently was set strategy options of dc2:0, then add the nodes. After the nodes are up, change the RF of dc2, and run repair.
> I'd like to avoid a repair as it runs AES and is a bit more intense than how bootstrap works currently by just streaming ranges from the SSTables. Would it be possible to improve this functionality (adding a new DC to existing cluster) than the proposed method? We'd be happy to do a patch if we got some input on the best way to go about it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira