Does a Lucene Filter reduce the search space of the underlying Query?

[Gevorg Hari <ge...@gmail.com>]

Hello!

I'd like to have Filters reduce the search space of a Lucene Query. My
experiments with DefaultSimilarity and different kind of filtering show
that this does not happens. Specifically, I noticed that while calculating
tf*idf the values that contribute to the idf (numDocs and docFreq) do not
change even in the presence of filters.

Are parameters like numDocs and docFreq calculated at index creation or are
those retrieved before applying the filter? Is there a way to have the
query being executed after the filter and with values that reflect the
reduced search space?

SO question here:
http://stackoverflow.com/questions/19882257/java-lucene-filters-seem-not-to-alter-the-query-search-space-as-expected-by-me

Gist shportcut here: https://gist.github.com/gevorghari/7438048

Thanks a lot!

Re: Does a Lucene Filter reduce the search space of the underlying Query?

[Erick Erickson <er...@gmail.com>]

Not  quite sure what you're after here. numDocs and docFreq are
index-wide numbers, they're not re-calculated on a per-query basis.

AFAIK, filters have nothing at all to do with these numbers.

Why do you care? What is it that you'd like to behave differently and
why would that be good? Or did I misunderstand your question entirely?

I'm sure others will jump in if I'm wrong here....

Erick


On Tue, Nov 12, 2013 at 6:29 PM, Gevorg Hari <ge...@gmail.com> wrote:

> Hello!
>
> I'd like to have Filters reduce the search space of a Lucene Query. My
> experiments with DefaultSimilarity and different kind of filtering show
> that this does not happens. Specifically, I noticed that while calculating
> tf*idf the values that contribute to the idf (numDocs and docFreq) do not
> change even in the presence of filters.
>
> Are parameters like numDocs and docFreq calculated at index creation or are
> those retrieved before applying the filter? Is there a way to have the
> query being executed after the filter and with values that reflect the
> reduced search space?
>
> SO question here:
>
> http://stackoverflow.com/questions/19882257/java-lucene-filters-seem-not-to-alter-the-query-search-space-as-expected-by-me
>
> Gist shportcut here: https://gist.github.com/gevorghari/7438048
>
> Thanks a lot!
>