You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Giovanni Fernandez-Kincade <gf...@capitaliq.com> on 2009/11/24 23:50:04 UTC
Index Splitter
Hi,
I've heard about a tool that can be used to split Lucene indexes, for cases where you want to break up a large index into shards. Do you know where I can find it? Any observations/recommendations about its use?
This seems promising but I'm not sure if there is anything more mature out there:
http://blog.foofactory.fi/2008/01/regenerating-equally-sized-shards-from.html
Thanks,
Gio.
Re: Index Splitter
Posted by Andrzej Bialecki <ab...@getopt.org>.
Koji Sekiguchi wrote:
> Giovanni Fernandez-Kincade wrote:
>> You can't really use this if you have an optimized index, right?
>>
>>
> For optimized index, I think you can use MultiPassIndexSplitter.
Correct - MultiPassIndexSplitter can handle any index - optimized or
not, with or without deletions, etc. The cost for this flexibility is
that it needs to read index files multiple times (hence "multi-pass").
--
Best regards,
Andrzej Bialecki <><
___. ___ ___ ___ _ _ __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com
Re: Index Splitter
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
Giovanni Fernandez-Kincade wrote:
> You can't really use this if you have an optimized index, right?
>
>
For optimized index, I think you can use MultiPassIndexSplitter.
Koji
--
http://www.rondhuit.com/en/
RE: Index Splitter
Posted by Giovanni Fernandez-Kincade <gf...@capitaliq.com>.
You can't really use this if you have an optimized index, right?
-----Original Message-----
From: Koji Sekiguchi [mailto:koji@r.email.ne.jp]
Sent: Tuesday, November 24, 2009 6:57 PM
To: solr-user@lucene.apache.org
Subject: Re: Index Splitter
Giovanni Fernandez-Kincade wrote:
> Hi,
> I've heard about a tool that can be used to split Lucene indexes, for cases where you want to break up a large index into shards. Do you know where I can find it? Any observations/recommendations about its use?
>
> This seems promising but I'm not sure if there is anything more mature out there:
> http://blog.foofactory.fi/2008/01/regenerating-equally-sized-shards-from.html
>
> Thanks,
> Gio.
>
>
There are IndexSplitter and MultiPassIndexSplitter tools in 3.0.
https://issues.apache.org/jira/browse/LUCENE-1959
I'd written an article about them before:
http://lucene.jugem.jp/?eid=344
It is Japanese but I think you can read out how to use them from command
lines...
Koji
--
http://www.rondhuit.com/en/
Re: Index Splitter
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
Giovanni Fernandez-Kincade wrote:
> Hi,
> I've heard about a tool that can be used to split Lucene indexes, for cases where you want to break up a large index into shards. Do you know where I can find it? Any observations/recommendations about its use?
>
> This seems promising but I'm not sure if there is anything more mature out there:
> http://blog.foofactory.fi/2008/01/regenerating-equally-sized-shards-from.html
>
> Thanks,
> Gio.
>
>
There are IndexSplitter and MultiPassIndexSplitter tools in 3.0.
https://issues.apache.org/jira/browse/LUCENE-1959
I'd written an article about them before:
http://lucene.jugem.jp/?eid=344
It is Japanese but I think you can read out how to use them from command
lines...
Koji
--
http://www.rondhuit.com/en/