Bloom filters and optimized vs. unoptimized indices

[Otis Gospodnetic <ot...@gmail.com>]

Hi,

I was looking at
http://lucene.apache.org/core/4_2_1/codecs/org/apache/lucene/codecs/bloom/BloomFilteringPostingsFormat.html
and this piece of text:
"
A PostingsFormat useful for low doc-frequency fields such as primary
keys. Bloom filters are maintained in a ".blm" file which offers
"fast-fail" for reads in segments known to have no record of the key.
"

Is this implying that if you are doing PK lookups AND you have a large
index (i.e. slow queries) it may actually be better to keep the index
unoptimized, so whole index segments can be skipped?

Thanks,
Otis
--
SOLR Performance Monitoring - http://sematext.com/spm/index.html

Re: Bloom filters and optimized vs. unoptimized indices

[Michael McCandless <lu...@mikemccandless.com>]

Be sure to test the bloom postings format on your own use case ... in
my tests (heavy PK lookups) it was slower.

But to answer your question: I would expect a single segment index to
have much faster PK lookups than a multi-segment one, with and without
the bloom postings format, but bloom may make the many-segment case
faster (just be sure to test it yourself).


Mike McCandless

http://blog.mikemccandless.com


On Tue, Apr 30, 2013 at 1:05 AM, Otis Gospodnetic
<ot...@gmail.com> wrote:
> Hi,
>
> I was looking at
> http://lucene.apache.org/core/4_2_1/codecs/org/apache/lucene/codecs/bloom/BloomFilteringPostingsFormat.html
> and this piece of text:
> "
> A PostingsFormat useful for low doc-frequency fields such as primary
> keys. Bloom filters are maintained in a ".blm" file which offers
> "fast-fail" for reads in segments known to have no record of the key.
> "
>
> Is this implying that if you are doing PK lookups AND you have a large
> index (i.e. slow queries) it may actually be better to keep the index
> unoptimized, so whole index segments can be skipped?
>
> Thanks,
> Otis
> --
> SOLR Performance Monitoring - http://sematext.com/spm/index.html