Query:Adding all docs at once or creating smaller indexes and merge

[Tarandeep Singh <ta...@gmail.com>]

Hi,

>From efficiency point of view, what will be more efficient-

Creating a single big index (big enough for one machine) by adding all
documents in it at once

or

Creating smaller indexes and then merge them to make one bigger index?

And if there is a performance penalty, then any rough estimate how much?

Thanks,
Tarandeep

Re: Query:Adding all docs at once or creating smaller indexes and merge

[Ian Lea <ia...@gmail.com>]

Hi


My guess is that one big index would be more efficient since the total
IO read and write load would be less.  The big reason for creating
smaller intermediate indexes is that you could spread their creation
over multiple jobs/disks/servers.

There is lots of good advice in
http://wiki.apache.org/lucene-java/ImproveIndexingSpeed

As to a performance penalty, there are too many variables to hazard a
guess.  Try it out in your application on your hardware.


--
Ian.


On Thu, Jun 4, 2009 at 10:10 PM, Tarandeep Singh<ta...@gmail.com> wrote:
> Hi,
>
> From efficiency point of view, what will be more efficient-
>
> Creating a single big index (big enough for one machine) by adding all
> documents in it at once
>
> or
>
> Creating smaller indexes and then merge them to make one bigger index?
>
> And if there is a performance penalty, then any rough estimate how much?
>
> Thanks,
> Tarandeep
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org