You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by da...@ontrenet.com on 2009/06/30 19:09:34 UTC
Combining Distributed Shards
Hi,
I read the distributed search/indexing wiki page but wondered if there
is a best-practice for re-combining distributed shards back into a
single local index. I have many computers generated indexes that cost
money when running, but want to merge all the indexes so I can terminate
the servers.
Has anyone does this successfully?
thanks for any tips!
Darren
Re: Combining Distributed Shards
Posted by da...@ontrenet.com.
>
> On Jun 30, 2009, at 1:51 PM, darren@ontrenet.com wrote:
>> It seems the merge index request (admin or lucene tool) expects the
>> indexes to already be local.
>>
>> Maybe in the future I can specify the URL to the remote indexes for
>> Solr
>> to merge. For now, I will find a way maybe using rsync or scp or
>> something. If anyone has done it this way with success, that'd be
>> helpful
>> to know.
>
> There could certainly be some tricks played by remotely controlling
> Solrs to create a new core, replicate from another machine, then do a
> merge. Hmmmm!
I see where you're going with the 'Hmmmm!'.....off to research it!
Re: Combining Distributed Shards
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jun 30, 2009, at 2:02 PM, Walter Underwood wrote:
> In Ultraseek, each collection (core) has a gathering method
> associated with
> it, for example, a web spider. There were two special collection
> types, a
> mirror collection, which copied an index from a different server,
> and a
> merge collection, which combined other collections.
Wheels were made to be reinvented :) One of these days we'll build a
better one, or die trying.
Erik
Re: Combining Distributed Shards
Posted by Walter Underwood <wu...@netflix.com>.
In Ultraseek, each collection (core) has a gathering method associated with
it, for example, a web spider. There were two special collection types, a
mirror collection, which copied an index from a different server, and a
merge collection, which combined other collections.
See the "Creating Collections" chapter in the on line docs:
http://www.ultraseek.com/support/docs/UltraseekAdmin/wwhelp/wwhimpl/js/html/
wwhelp.htm
wunder
On 6/30/09 10:55 AM, "Erik Hatcher" <er...@ehatchersolutions.com> wrote:
>
> On Jun 30, 2009, at 1:51 PM, darren@ontrenet.com wrote:
>> It seems the merge index request (admin or lucene tool) expects the
>> indexes to already be local.
>>
>> Maybe in the future I can specify the URL to the remote indexes for
>> Solr
>> to merge. For now, I will find a way maybe using rsync or scp or
>> something. If anyone has done it this way with success, that'd be
>> helpful
>> to know.
>
> There could certainly be some tricks played by remotely controlling
> Solrs to create a new core, replicate from another machine, then do a
> merge. Hmmmm!
>
> Erik
>
Re: Combining Distributed Shards
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jun 30, 2009, at 1:51 PM, darren@ontrenet.com wrote:
> It seems the merge index request (admin or lucene tool) expects the
> indexes to already be local.
>
> Maybe in the future I can specify the URL to the remote indexes for
> Solr
> to merge. For now, I will find a way maybe using rsync or scp or
> something. If anyone has done it this way with success, that'd be
> helpful
> to know.
There could certainly be some tricks played by remotely controlling
Solrs to create a new core, replicate from another machine, then do a
merge. Hmmmm!
Erik
Re: Combining Distributed Shards
Posted by da...@ontrenet.com.
Thank you. That's what I need I think, but the indexes are created on
other computers.
It seems the merge index request (admin or lucene tool) expects the
indexes to already be local.
Maybe in the future I can specify the URL to the remote indexes for Solr
to merge. For now, I will find a way maybe using rsync or scp or
something. If anyone has done it this way with success, that'd be helpful
to know.
> On Tue, Jun 30, 2009 at 10:39 PM, <da...@ontrenet.com> wrote:
>
>> I read the distributed search/indexing wiki page but wondered if there
>> is a best-practice for re-combining distributed shards back into a
>> single local index. I have many computers generated indexes that cost
>> money when running, but want to merge all the indexes so I can terminate
>> the servers.
>
>
> Some information is available at
> http://wiki.apache.org/solr/MergingSolrIndexes
> --
> Regards,
> Shalin Shekhar Mangar.
>
Re: Combining Distributed Shards
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Tue, Jun 30, 2009 at 10:39 PM, <da...@ontrenet.com> wrote:
> I read the distributed search/indexing wiki page but wondered if there
> is a best-practice for re-combining distributed shards back into a
> single local index. I have many computers generated indexes that cost
> money when running, but want to merge all the indexes so I can terminate
> the servers.
Some information is available at
http://wiki.apache.org/solr/MergingSolrIndexes
--
Regards,
Shalin Shekhar Mangar.