use index, big or small?

[Yang <te...@gmail.com>]

I have an index containing all students, now I want to do an index
search inside an Apache Hadoop mapper,
i.e.

for each (record from mapper input reader) {
    output = lucene.search("name:"+ record.name  + " OR " + " id:" +
record.id );
    emit(output)
}


my question is whether I should shard the index (across terms, not
splitting the same postings list for one term) or simply replicate it.
the index for the entire dataset is not too big, so it can fig into
my local disk, and I can copy it to every node in the cluster, and let
them sit there all the time, so no copy overhead is incurred.
the only argument in favor of sharding is that a smaller index might
be faster.  but since index search is only O(lg(n)) time, maybe this
time saving is very small.

so will sharding be worth the effort?

thanks
yang

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Re: use index, big or small?

[Ian Lea <ia...@gmail.com>]

Impossible to say - how big is big?  How fast is fast?  I'd start with
the simplest option and if it's fast enough, stop.


--
Ian.


On Sat, May 5, 2012 at 12:47 AM, Yang <te...@gmail.com> wrote:
> I have an index containing all students, now I want to do an index
> search inside an Apache Hadoop mapper,
> i.e.
>
> for each (record from mapper input reader) {
>    output = lucene.search("name:"+ record.name  + " OR " + " id:" +
> record.id );
>    emit(output)
> }
>
>
> my question is whether I should shard the index (across terms, not
> splitting the same postings list for one term) or simply replicate it.
> the index for the entire dataset is not too big, so it can fig into
> my local disk, and I can copy it to every node in the cluster, and let
> them sit there all the time, so no copy overhead is incurred.
> the only argument in favor of sharding is that a smaller index might
> be faster.  but since index search is only O(lg(n)) time, maybe this
> time saving is very small.
>
> so will sharding be worth the effort?
>
> thanks
> yang
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org