You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by cwhi <ch...@gmail.com> on 2013/12/15 23:43:23 UTC
How can you move a shard from one SolrCloud node to another?
Let's say I want to rebalance a SolrCloud collection. I call SPLITSHARD to
split an existing shard, and then I'd like to move one of the subshards to a
new machine so the index is more balanced. Can this be done? If not, how
do you rebalance an existing SolrCloud collection?
--
View this message in context: http://lucene.472066.n3.nabble.com/How-can-you-move-a-shard-from-one-SolrCloud-node-to-another-tp4106815.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: How can you move a shard from one SolrCloud node to another?
Posted by Mark Miller <ma...@gmail.com>.
On Dec 16, 2013, at 12:58 PM, Tim Potter <ti...@lucidworks.com> wrote:
> The only trick here is you need to set the name of the core, which from what I can tell can be arbitrary
Yup.
> but I chose to use the same naming standard as the other cores
Which is likely a good idea - those names are used mainly for oversharding (you might have more than one core on a node from the same collection and shard). Probably better to use a naming scheme that is friendly to this in the long run, and the default naming scheme is. Also less confusing to match the default naming scheme.
- Mark
RE: How can you move a shard from one SolrCloud node to another?
Posted by Tim Potter <ti...@lucidworks.com>.
Hi Yago,
When you create a new core (via API or Web UI), you specify the collection name and shard id, in my example "cloud" and "shard1_0". When the core initializes in SolrCloud mode, it recognizes that the collection exists and adds itself as a replica to the shard. Then the main replica recovery process kicks in; try PeerSync, realize too far out of date, try snapshot replication from leader. The following core API command led to the same result as using the UI:
curl -v "http://localhost:8986/solr/admin/cores?action=CREATE&collection=cloud&shard=shard1_0&name=cloud_shard1_0_replica3"
The only trick here is you need to set the name of the core, which from what I can tell can be arbitrary but I chose to use the same naming standard as the other cores
Cheers,
Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com
________________________________________
From: Yago Riveiro <ya...@gmail.com>
Sent: Monday, December 16, 2013 9:32 AM
To: solr-user@lucene.apache.org
Subject: Re: How can you move a shard from one SolrCloud node to another?
Tim,
Can you explain how the replication snapshot is done using the coreAdminAPI?
--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Monday, December 16, 2013 at 4:23 PM, Tim Potter wrote:
> Hi Chris,
>
> The easiest approach is to just create a new core on the new machine that references the collection and shard you want to migrate. For example, say you split shard1 of a collection named "cloud", which results in having: shard1_0 and shard1_1. Now let's say you want to migrate shard 1_0 over to the new machine.
>
> First, fire off a q=*:*&distrib=false query to the shard you're migrating so that you know how many docs it has (which will be used to verify the migration was clean below).
>
> Next, bring up the new machine in cloud mode (-zkHost=?) and then go to the admin console on that server. Nav to the core admin page and create a new core, specifying the collection and shard1_0 in the form; note: the form leads you to believe you need to create the directory on the local system but you actually don't need to worry about doing that as the config will get pulled from ZK and the directory will get created on the fly (at least that's what happened in my env using branch_4x).
>
> When the new core initializes, it will use good ol' snapshot replication to pull the index from the leader. Verify the new core is happy by executing the q=*:*&distrib=false query again. Once you're satisfied, you can unload the core you migrated.
>
> Btw ... you can do all this with the core admin API instead of the Web UI if you want to script it.
>
> Cheers,
>
> Timothy Potter
> Sr. Software Engineer, LucidWorks
> www.lucidworks.com (http://www.lucidworks.com)
>
> ________________________________________
> From: cwhi <chris.whiten@gmail.com (mailto:chris.whiten@gmail.com)>
> Sent: Sunday, December 15, 2013 3:43 PM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: How can you move a shard from one SolrCloud node to another?
>
> Let's say I want to rebalance a SolrCloud collection. I call SPLITSHARD to
> split an existing shard, and then I'd like to move one of the subshards to a
> new machine so the index is more balanced. Can this be done? If not, how
> do you rebalance an existing SolrCloud collection?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/How-can-you-move-a-shard-from-one-SolrCloud-node-to-another-tp4106815.html
> Sent from the Solr - User mailing list archive at Nabble.com (http://Nabble.com).
>
>
Re: How can you move a shard from one SolrCloud node to
another?
Posted by Yago Riveiro <ya...@gmail.com>.
Tim,
Can you explain how the replication snapshot is done using the coreAdminAPI?
--
Yago Riveiro
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)
On Monday, December 16, 2013 at 4:23 PM, Tim Potter wrote:
> Hi Chris,
>
> The easiest approach is to just create a new core on the new machine that references the collection and shard you want to migrate. For example, say you split shard1 of a collection named "cloud", which results in having: shard1_0 and shard1_1. Now let's say you want to migrate shard 1_0 over to the new machine.
>
> First, fire off a q=*:*&distrib=false query to the shard you're migrating so that you know how many docs it has (which will be used to verify the migration was clean below).
>
> Next, bring up the new machine in cloud mode (-zkHost=?) and then go to the admin console on that server. Nav to the core admin page and create a new core, specifying the collection and shard1_0 in the form; note: the form leads you to believe you need to create the directory on the local system but you actually don't need to worry about doing that as the config will get pulled from ZK and the directory will get created on the fly (at least that's what happened in my env using branch_4x).
>
> When the new core initializes, it will use good ol' snapshot replication to pull the index from the leader. Verify the new core is happy by executing the q=*:*&distrib=false query again. Once you're satisfied, you can unload the core you migrated.
>
> Btw ... you can do all this with the core admin API instead of the Web UI if you want to script it.
>
> Cheers,
>
> Timothy Potter
> Sr. Software Engineer, LucidWorks
> www.lucidworks.com (http://www.lucidworks.com)
>
> ________________________________________
> From: cwhi <chris.whiten@gmail.com (mailto:chris.whiten@gmail.com)>
> Sent: Sunday, December 15, 2013 3:43 PM
> To: solr-user@lucene.apache.org (mailto:solr-user@lucene.apache.org)
> Subject: How can you move a shard from one SolrCloud node to another?
>
> Let's say I want to rebalance a SolrCloud collection. I call SPLITSHARD to
> split an existing shard, and then I'd like to move one of the subshards to a
> new machine so the index is more balanced. Can this be done? If not, how
> do you rebalance an existing SolrCloud collection?
>
>
>
> --
> View this message in context: http://lucene.472066.n3.nabble.com/How-can-you-move-a-shard-from-one-SolrCloud-node-to-another-tp4106815.html
> Sent from the Solr - User mailing list archive at Nabble.com (http://Nabble.com).
>
>
RE: How can you move a shard from one SolrCloud node to another?
Posted by Tim Potter <ti...@lucidworks.com>.
Hi Chris,
The easiest approach is to just create a new core on the new machine that references the collection and shard you want to migrate. For example, say you split shard1 of a collection named "cloud", which results in having: shard1_0 and shard1_1. Now let's say you want to migrate shard 1_0 over to the new machine.
First, fire off a q=*:*&distrib=false query to the shard you're migrating so that you know how many docs it has (which will be used to verify the migration was clean below).
Next, bring up the new machine in cloud mode (-zkHost=?) and then go to the admin console on that server. Nav to the core admin page and create a new core, specifying the collection and shard1_0 in the form; note: the form leads you to believe you need to create the directory on the local system but you actually don't need to worry about doing that as the config will get pulled from ZK and the directory will get created on the fly (at least that's what happened in my env using branch_4x).
When the new core initializes, it will use good ol' snapshot replication to pull the index from the leader. Verify the new core is happy by executing the q=*:*&distrib=false query again. Once you're satisfied, you can unload the core you migrated.
Btw ... you can do all this with the core admin API instead of the Web UI if you want to script it.
Cheers,
Timothy Potter
Sr. Software Engineer, LucidWorks
www.lucidworks.com
________________________________________
From: cwhi <ch...@gmail.com>
Sent: Sunday, December 15, 2013 3:43 PM
To: solr-user@lucene.apache.org
Subject: How can you move a shard from one SolrCloud node to another?
Let's say I want to rebalance a SolrCloud collection. I call SPLITSHARD to
split an existing shard, and then I'd like to move one of the subshards to a
new machine so the index is more balanced. Can this be done? If not, how
do you rebalance an existing SolrCloud collection?
--
View this message in context: http://lucene.472066.n3.nabble.com/How-can-you-move-a-shard-from-one-SolrCloud-node-to-another-tp4106815.html
Sent from the Solr - User mailing list archive at Nabble.com.