FieldCacheRangeFilter Vs NumericRangeQuery

[Romi <ro...@gmail.com>]

Hi,
   I am using Lucene3.0 for my project. I used FieldCacheRangeFilter (Static
method newLongRange()) with NumericRangeQuery in one implementation of
dateRangeQuery search and only NumericRangeQuery for another implementation
of dateRangeQuery search.
I found 17525 documents via both implementation but search time via
FieldCacheRangeFilter was 172milliseconds while via NumericRangeQuery search
it was 62milliseconds.

I got confused because i was expecting that FieldCacheRangeFilter do search
faster.
Why it happened please explain.

-----
Romi
-- 
View this message in context: http://lucene.472066.n3.nabble.com/FieldCacheRangeFilter-Vs-NumericRangeQuery-tp2038419p2038419.html
Sent from the Lucene - General mailing list archive at Nabble.com.

RE: FieldCacheRangeFilter Vs NumericRangeQuery

[Uwe Schindler <uw...@thetaphi.de>]

> Hi,
>    I run search using FieldChacheFilter and NumericRangeFilter and i am
> getting the similar results. Searchtime is same in both the cases. My
index
> size is 10000000 documents.
> My point is :
> 
> 1. When search is performed using FieldCacheFilter then it caches the
first
> query search result and so next query searches becomes fast. But its not
the
> case with NumericRangeFilter then Why i am getting the same search time in
> both implementations.

It does not cache the filter result unless you use CachingWrapperFilter. It
caches only the actual values for each document in the FieldCache but that
has nothing to do with Filters at all. But it must still review all values
for all documents in the FieldCache to filter the documents. This is faster
for lots of documents matching the query, but for very sparse result sets
this is too expensive. NumericRangeFilter works different, but for it to
work, you have to use a non-infinite precisionStep during indexing with
NumericField, so the index gets bigger. If you only use
FieldCacheRangeFilter, you can index with precisionStep=Integer.MAX_VALUE
(see docs).

I posted you this article, it explains all of this: http://s.apache.org/tk

> 2. I want to know in which case i should use filters. I am performing
searches
> on DateRanges. User just enters a date to search the required document,
will
> it b benificial to use filter in this case.
> As far as i know filters should use when u want to apply some constraints(
eg.
> hide documents which are of "high security") , but in my case too seaching
is
> very fast in case of filter. Should i really use filter?

Since recent versions of Lucene, there is no difference for MultiTermQueries
like NumericRangeQuery, TermRangeQuery,... and replacing them by a filter.
The pro for filters is just for the case where you want to cache them (e.g.
NumericRangeFilter, FieldCacheRangeFilter,...) for later reuse. But for
user-entered arbitrary queries this is not needed and Filters are as fast as
Queries, sometimes the Queries are even faster (this has internal reasons
because of a missing optimization in IndexSearcher).

Uwe

RE: FieldCacheRangeFilter Vs NumericRangeQuery

[Romi <ro...@gmail.com>]

Hi,
   I run search using FieldChacheFilter and NumericRangeFilter and i am
getting the similar results. Searchtime is same in both the cases. My index
size is 10000000 documents.
My point is :

1. When search is performed using FieldCacheFilter then it caches the first
query search result and so next query searches becomes fast. But its not the
case with NumericRangeFilter then Why i am getting the same search time in
both implementations.

2. I want to know in which case i should use filters. I am performing
searches on DateRanges. User just enters a date to search the required
document, will it b benificial to use filter in this case.
As far as i know filters should use when u want to apply some constraints(
eg. hide documents which are of "high security") , but in my case too
seaching is very fast in case of filter. Should i really use filter?

-----
Romi
-- 
View this message in context: http://lucene.472066.n3.nabble.com/FieldCacheRangeFilter-Vs-NumericRangeQuery-tp2038419p2057294.html
Sent from the Lucene - General mailing list archive at Nabble.com.

RE: FieldCacheRangeFilter Vs NumericRangeQuery

[Uwe Schindler <uw...@thetaphi.de>]

Hi Romi,

Depending on how many documents are returned, FieldCacheRangeFilter (may)
outperform NumericRangeQuery/Filter. But for sparse results,
NumericRangeQuery performs much better. Look here: http://s.apache.org/tk

Additionally take into account that the FieldCache must be built before the
first query (warming) and FC only works correct with exactly one value /
document. Missing values or more than one value break FieldCache usage for
range filtering. FieldCacheRangeFilter is good, if you use it on fields you
are already need for sorting and are in FieldCache already.

Please note: Bothe NumericRangeFilter and FieldCacheRangeFilter can be used
with CachingWrapperFilter. In this case, NumericRangeFilter is faster most
times (when already cached).

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Romi [mailto:romijain3186@gmail.com]
> Sent: Wednesday, December 08, 2010 7:45 AM
> To: general@lucene.apache.org
> Subject: FieldCacheRangeFilter Vs NumericRangeQuery
> 
> 
> Hi,
>    I am using Lucene3.0 for my project. I used FieldCacheRangeFilter
(Static
> method newLongRange()) with NumericRangeQuery in one implementation
> of dateRangeQuery search and only NumericRangeQuery for another
> implementation of dateRangeQuery search.
> I found 17525 documents via both implementation but search time via
> FieldCacheRangeFilter was 172milliseconds while via NumericRangeQuery
> search it was 62milliseconds.
> 
> I got confused because i was expecting that FieldCacheRangeFilter do
search
> faster.
> Why it happened please explain.
> 
> -----
> Romi
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/FieldCacheRangeFilter-Vs-
> NumericRangeQuery-tp2038419p2038419.html
> Sent from the Lucene - General mailing list archive at Nabble.com.