You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by "Branham, Jeremy [HR]" <Je...@sprint.com> on 2014/06/14 21:29:17 UTC
SOLR Cloud Rebuild core
We are looking to move from legacy master/slave configuration to the cloud configuration.
In the past we have handled rebuilding cores by using a 'live' core and a core for performing the rebuild on.
When a rebuild is complete, we swap the rebuilt core with the live core.
Is this still a good way to do offline rebuilding when using cloud?
Full re-indexing on our largest index only takes 25 min.
Thanks!
Jeremy Branham
________________________________
This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.
Re: SOLR Cloud Rebuild core
Posted by Greg Walters <gr...@answers.com>.
Plus one to Shawn's method below. I'm using that method in production right now and on occasion there's only a small blip of queued queries while the alias is in the process of swapping.
Thanks,
Greg
On Jun 15, 2014, at 10:36 AM, Shawn Heisey <so...@elyograg.org> wrote:
> On 6/14/2014 1:29 PM, Branham, Jeremy [HR] wrote:
>> We are looking to move from legacy master/slave configuration to the cloud configuration.
>>
>> In the past we have handled rebuilding cores by using a 'live' core and a core for performing the rebuild on.
>> When a rebuild is complete, we swap the rebuilt core with the live core.
>>
>> Is this still a good way to do offline rebuilding when using cloud?
>>
>> Full re-indexing on our largest index only takes 25 min.
>
> Walter has already replied, but I have one thing to add:
>
> "Swapping" does not exist for SolrCloud, but there is a workaround. It
> is accomplished in SolrCloud through collection aliasing. What you do
> is create two entire collections. If your application will use a
> collection named "frobozz" then you might create "frob_0" and "frob_1"
> as the actual collection names. You have an alias from frobozz to
> frob_0 ... when you want to rebuild, you clear frob_1 and build the new
> collection there. Then when it's done, you change the alias to point at
> frob_1.
>
> Note that I am saying collection -- not core. If your collections are
> distributed, you will not be able to rebuild just one shard, you'll have
> to rebuild the entire thing.
>
> Aliases can point at multiple collections, but if you want them to work
> for index updates as well as queries, then you need to point them at a
> single collection.
>
> Thanks,
> Shawn
>
Re: SOLR Cloud Rebuild core
Posted by Shawn Heisey <so...@elyograg.org>.
On 6/14/2014 1:29 PM, Branham, Jeremy [HR] wrote:
> We are looking to move from legacy master/slave configuration to the cloud configuration.
>
> In the past we have handled rebuilding cores by using a 'live' core and a core for performing the rebuild on.
> When a rebuild is complete, we swap the rebuilt core with the live core.
>
> Is this still a good way to do offline rebuilding when using cloud?
>
> Full re-indexing on our largest index only takes 25 min.
Walter has already replied, but I have one thing to add:
"Swapping" does not exist for SolrCloud, but there is a workaround. It
is accomplished in SolrCloud through collection aliasing. What you do
is create two entire collections. If your application will use a
collection named "frobozz" then you might create "frob_0" and "frob_1"
as the actual collection names. You have an alias from frobozz to
frob_0 ... when you want to rebuild, you clear frob_1 and build the new
collection there. Then when it's done, you change the alias to point at
frob_1.
Note that I am saying collection -- not core. If your collections are
distributed, you will not be able to rebuild just one shard, you'll have
to rebuild the entire thing.
Aliases can point at multiple collections, but if you want them to work
for index updates as well as queries, then you need to point them at a
single collection.
Thanks,
Shawn
RE: SOLR Cloud Rebuild core
Posted by rulinma <ru...@gmail.com>.
I also confuse to this long time.
--
View this message in context: http://lucene.472066.n3.nabble.com/SOLR-Cloud-Rebuild-core-tp4141869p4142194.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: SOLR Cloud Rebuild core
Posted by "Branham, Jeremy [HR]" <Je...@sprint.com>.
Perfect.
Rather than rolling back a single transaction, we could just rollback everything since the last commit.
Thanks for the insight.
Jeremy Branham
-----Original Message-----
From: Walter Underwood [mailto:wunder@wunderwood.org]
Sent: Saturday, June 14, 2014 3:03 PM
To: solr-user@lucene.apache.org
Subject: Re: SOLR Cloud Rebuild core
Exactly. Do one commit at the end. I do this for indexes with 4 million or more documents. Works fine.
I don't know of a way to flush the queued add-document commands. Solr does not have the concept of a single transaction that can be dropped.
wunder
On Jun 14, 2014, at 12:57 PM, "Branham, Jeremy [HR]" <Je...@sprint.com> wrote:
> Thanks Walter -
>
> We have a few different use cases where there is no good way [currently] to uniquely identify a document.
> We also have some cores where real-time freshness is key.
> I'd rather move to cloud than have 2 different instances.
>
> Since some cores will need to have all documents deleted during the rebuild, there a way to do complete rebuild without disturbing queries to the core?
> Maybe postponing the commit to the end of the rebuild process?
>
> I'm thinking the commit would then delete all existing document add all the new documents.
> Is this flawed?
>
> Assuming that would work - if the reindexing fails could we bail on the commit and still have a sane core?
>
> Jeremy Branham
>
> -----Original Message-----
> From: Walter Underwood [mailto:wunder@wunderwood.org]
> Sent: Saturday, June 14, 2014 2:44 PM
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR Cloud Rebuild core
>
> If you don't need near real-time index freshness, why are you implementing Solr Cloud? It is harder to set up and adds functionality you are not using. Solr Cloud is designed for a fully live index, not offline indexing.
>
> Also, I don't understand why people do this complicated offline build and swapping stuff. That is precisely what replication already does for you and it is built-in. You build an index on one system and swap it in over the network.
>
> wunder
>
> On Jun 14, 2014, at 12:29 PM, "Branham, Jeremy [HR]" <Je...@sprint.com> wrote:
>
>> We are looking to move from legacy master/slave configuration to the cloud configuration.
>>
>> In the past we have handled rebuilding cores by using a 'live' core and a core for performing the rebuild on.
>> When a rebuild is complete, we swap the rebuilt core with the live core.
>>
>> Is this still a good way to do offline rebuilding when using cloud?
>>
>> Full re-indexing on our largest index only takes 25 min.
>>
>> Thanks!
>>
>>
>> Jeremy Branham
>>
>
>
>
>
>
> ________________________________
>
> This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.
>
--
Walter Underwood
wunder@wunderwood.org
________________________________
This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.
Re: SOLR Cloud Rebuild core
Posted by Walter Underwood <wu...@wunderwood.org>.
Exactly. Do one commit at the end. I do this for indexes with 4 million or more documents. Works fine.
I don't know of a way to flush the queued add-document commands. Solr does not have the concept of a single transaction that can be dropped.
wunder
On Jun 14, 2014, at 12:57 PM, "Branham, Jeremy [HR]" <Je...@sprint.com> wrote:
> Thanks Walter -
>
> We have a few different use cases where there is no good way [currently] to uniquely identify a document.
> We also have some cores where real-time freshness is key.
> I'd rather move to cloud than have 2 different instances.
>
> Since some cores will need to have all documents deleted during the rebuild, there a way to do complete rebuild without disturbing queries to the core?
> Maybe postponing the commit to the end of the rebuild process?
>
> I'm thinking the commit would then delete all existing document add all the new documents.
> Is this flawed?
>
> Assuming that would work - if the reindexing fails could we bail on the commit and still have a sane core?
>
> Jeremy Branham
>
> -----Original Message-----
> From: Walter Underwood [mailto:wunder@wunderwood.org]
> Sent: Saturday, June 14, 2014 2:44 PM
> To: solr-user@lucene.apache.org
> Subject: Re: SOLR Cloud Rebuild core
>
> If you don't need near real-time index freshness, why are you implementing Solr Cloud? It is harder to set up and adds functionality you are not using. Solr Cloud is designed for a fully live index, not offline indexing.
>
> Also, I don't understand why people do this complicated offline build and swapping stuff. That is precisely what replication already does for you and it is built-in. You build an index on one system and swap it in over the network.
>
> wunder
>
> On Jun 14, 2014, at 12:29 PM, "Branham, Jeremy [HR]" <Je...@sprint.com> wrote:
>
>> We are looking to move from legacy master/slave configuration to the cloud configuration.
>>
>> In the past we have handled rebuilding cores by using a 'live' core and a core for performing the rebuild on.
>> When a rebuild is complete, we swap the rebuilt core with the live core.
>>
>> Is this still a good way to do offline rebuilding when using cloud?
>>
>> Full re-indexing on our largest index only takes 25 min.
>>
>> Thanks!
>>
>>
>> Jeremy Branham
>>
>
>
>
>
>
> ________________________________
>
> This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.
>
--
Walter Underwood
wunder@wunderwood.org
RE: SOLR Cloud Rebuild core
Posted by "Branham, Jeremy [HR]" <Je...@sprint.com>.
Thanks Walter -
We have a few different use cases where there is no good way [currently] to uniquely identify a document.
We also have some cores where real-time freshness is key.
I'd rather move to cloud than have 2 different instances.
Since some cores will need to have all documents deleted during the rebuild, there a way to do complete rebuild without disturbing queries to the core?
Maybe postponing the commit to the end of the rebuild process?
I'm thinking the commit would then delete all existing document add all the new documents.
Is this flawed?
Assuming that would work - if the reindexing fails could we bail on the commit and still have a sane core?
Jeremy Branham
-----Original Message-----
From: Walter Underwood [mailto:wunder@wunderwood.org]
Sent: Saturday, June 14, 2014 2:44 PM
To: solr-user@lucene.apache.org
Subject: Re: SOLR Cloud Rebuild core
If you don't need near real-time index freshness, why are you implementing Solr Cloud? It is harder to set up and adds functionality you are not using. Solr Cloud is designed for a fully live index, not offline indexing.
Also, I don't understand why people do this complicated offline build and swapping stuff. That is precisely what replication already does for you and it is built-in. You build an index on one system and swap it in over the network.
wunder
On Jun 14, 2014, at 12:29 PM, "Branham, Jeremy [HR]" <Je...@sprint.com> wrote:
> We are looking to move from legacy master/slave configuration to the cloud configuration.
>
> In the past we have handled rebuilding cores by using a 'live' core and a core for performing the rebuild on.
> When a rebuild is complete, we swap the rebuilt core with the live core.
>
> Is this still a good way to do offline rebuilding when using cloud?
>
> Full re-indexing on our largest index only takes 25 min.
>
> Thanks!
>
>
> Jeremy Branham
>
________________________________
This e-mail may contain Sprint proprietary information intended for the sole use of the recipient(s). Any use by others is prohibited. If you are not the intended recipient, please contact the sender and delete all copies of the message.
Re: SOLR Cloud Rebuild core
Posted by Walter Underwood <wu...@wunderwood.org>.
If you don't need near real-time index freshness, why are you implementing Solr Cloud? It is harder to set up and adds functionality you are not using. Solr Cloud is designed for a fully live index, not offline indexing.
Also, I don't understand why people do this complicated offline build and swapping stuff. That is precisely what replication already does for you and it is built-in. You build an index on one system and swap it in over the network.
wunder
On Jun 14, 2014, at 12:29 PM, "Branham, Jeremy [HR]" <Je...@sprint.com> wrote:
> We are looking to move from legacy master/slave configuration to the cloud configuration.
>
> In the past we have handled rebuilding cores by using a 'live' core and a core for performing the rebuild on.
> When a rebuild is complete, we swap the rebuilt core with the live core.
>
> Is this still a good way to do offline rebuilding when using cloud?
>
> Full re-indexing on our largest index only takes 25 min.
>
> Thanks!
>
>
> Jeremy Branham
>