You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Hoss <ho...@fucit.org> on 2004/11/23 03:25:24 UTC
Numeric Range Restrictions: Queries vs Filters
(NOTE: numbers in [] indicate Footnotes)
I'm rather new to Lucene (and this list), so if I'm grossly
misunderstanding things, forgive me.
One of my main needs as I investigate Search technologies is to restrict
results based on Ranges of numeric values. Looking over the archives of
this list, it seems that lots of people have run into problems dealing
with this. In particular, whenever someone asks a question about "Numeric
Ranges" the question seem to always involve one (or more) of the
following:
(a) Lexical sorting puts 11 in the range "1 TO 5"
(b) Dates (or Dates and Times)
(c) BooleanQuery$TooManyClauses Exceptions
(d) Should I use a filter?
(a) is a solved problem as long as you use a formatter like
LongField.java[1]
(b) is really nothing more then a special case of dealing with generic
numeric values. While there are certainly special purposes solutions that
sometimes apply to dealing with Date ranges, any good solution for dealing
with raw numeric ranges can be applied to Dates (and Times)
(c) is a situation that seems to come up a lot because of the way
RangeQuery works. The rewrite method walks all of the Terms in the index
starting with "lowerTerm" and builds up BooleanQuery containing a separate
TermQuery for every Term found, until it reaches the upperTerm. This
causes a range search of "0001 TO 1000" to generate a BooleanQuery with N
clauses, where N is the quantity of unique values in the field which are
lexically greater then 0001 and lexically less then 1000. depending on
the nature of your data, this might be 0 BooleanClauses, or it might be
1000 BooleanClauses; but the list is built before the search is ever even
executed.
At first, this may seem really strange -- I know I was certainly confused
-- but there is a very good reason for it: Ultimately RangeQuery still
provides you with a meaningful score for each document, based on the
frequency (and quantity) of terms that document has in the range [2]. In
order to do that, it has to expand itself, but what if you don't care if
your Range restriction impacts the Score? [3]
Which brings us to...
(c) Filtering. Filters in general make a lot of sense to me. They are a
way to specify (at query time) that only a certain subset of the index
should be considered for results. The Filter class has a very straight
forward API that seems very easy to subclass to get the behavior I want.
The Query API on the other hand ... I freely admit, that I can't make
heads or tails out of it. I don't even know where I would begin to try
and write a new subclass of Query if I wanted to.
I would think that most people who want to do a "numeric range
restriction" on their data, probably don't care about the Scoring benefits
of RangeQuery. Looking at the code base, the way DateFilter works seems
like it provides an ideal solution to any sort of Range restriction (not
just Dates) that *should* be more efficient then using RangeQuery when
dealing with an unbounded value set. (Both approaches need to iterate over
all of the terms in the specified field using TermEnum, but RangeQuery has
to build up an set of BooleanQuery objects for each matching term, and
then each of those queries have to help score the documents -- DateFilter
on the other hand only has to maintain a single BitSet of documents that
it finds as it iterates)
But I was surprised then to see the following quote from "Erik Hatcher" in
the archives:
"In fact, DateFilter by itself is practically of no use, I think." [4]
...Erik goes on to suggest that given "a set of canned date ranges", it
doesn't really matter if you use a RangeQuery or a DateFilter -- as long
as you cache them to reuse them (with something like CachingWrappingFilter
or QueryFilter). I'm hoping that he might elaborate on that comment?
As a test, I wrote a "RangeFilter" which borrows heavily from DateFilter
to both convince myself it could work, and to do a comparison between it
and RangeQuery. [5] Based on my limited tests, using a Filter to restrict
to a Range is a lot faster then using RangeQuery -- independent of
caching.
The attachment contains my RangeFilter, a unit test that demonstrates it,
and a Benchmarking unit test that does a side-by-side comparison with
RangeQuery [6]. If developers feel that this class is useful, then by all
means roll it into the code base. (90% of it is cut/pasted from
DateFilter/RangeQuery anyway)
Comments? ... Questions? ... Answers?
Footnotes:
[1] It seems to me this class is extremely useful, does anyone know
if there's a particular reason it hasn't been added to the main Lucene
codebase?
http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg04790.html
[2] Take a look at RangeQueryScoreDemo.java in the attachment, which
produces output something like this...
Range Search for: 'apple' TO 'dog'
0.40924072 ... bed dog emu
0.38014847 ... DOG
0.2825246 ... cat
0.17657787 ... apple emu
0.12671615 ... dog
[3] According to the list archives "Matt Quail" mentioned in May that
he was working on a "QuickRangeQuery" class that wouldn't have the
BooleanQuery limitation, at the expense of always scoring "1.0",
but I haven't seen any mention of anything like it since. Is Matt
still an active list member? Matt, is this something you're still
pursuing?
http://nagoya.apache.org/eyebrowse/ReadMsg?msgId=1659395
[4] http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg07015.html
[5] The only major difference between my RangeFilter and DateFilter is
that RangeFilter supports options for inclusion/exclusion
(individually for the low/high terms I might add). But for the
purposes of a benchmark, doing the same thing with DateFilter would
have worked fine.
[6] If you have ant, see "ant -projecthelp"; otherwise, read the top
of build.xml
--
-------------------------------------------------------------------
"Oh, you're a tricky one." Chris M Hostetter
-- Trisha Weir hossman@rescomp.berkeley.edu
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Hoss <ho...@fucit.org>.
Of course, not only did I manage to forget to include the attachment, but
when I sent a reply with the code, mail.apache.org rejected it because it
was a ZIP file.
So let's see how mail.apache.or feels about 6 seperate text files.
: Date: Mon, 22 Nov 2004 18:25:24 -0800 (PST)
: Subject: Numeric Range Restrictions: Queries vs Filters
:
: (NOTE: numbers in [] indicate Footnotes)
:
: I'm rather new to Lucene (and this list), so if I'm grossly
: misunderstanding things, forgive me.
:
: One of my main needs as I investigate Search technologies is to restrict
: results based on Ranges of numeric values. Looking over the archives of
: this list, it seems that lots of people have run into problems dealing
: with this. In particular, whenever someone asks a question about "Numeric
: Ranges" the question seem to always involve one (or more) of the
: following:
:
: (a) Lexical sorting puts 11 in the range "1 TO 5"
: (b) Dates (or Dates and Times)
: (c) BooleanQuery$TooManyClauses Exceptions
: (d) Should I use a filter?
:
: (a) is a solved problem as long as you use a formatter like
: LongField.java[1]
:
: (b) is really nothing more then a special case of dealing with generic
: numeric values. While there are certainly special purposes solutions that
: sometimes apply to dealing with Date ranges, any good solution for dealing
: with raw numeric ranges can be applied to Dates (and Times)
:
: (c) is a situation that seems to come up a lot because of the way
: RangeQuery works. The rewrite method walks all of the Terms in the index
: starting with "lowerTerm" and builds up BooleanQuery containing a separate
: TermQuery for every Term found, until it reaches the upperTerm. This
: causes a range search of "0001 TO 1000" to generate a BooleanQuery with N
: clauses, where N is the quantity of unique values in the field which are
: lexically greater then 0001 and lexically less then 1000. depending on
: the nature of your data, this might be 0 BooleanClauses, or it might be
: 1000 BooleanClauses; but the list is built before the search is ever even
: executed.
:
: At first, this may seem really strange -- I know I was certainly confused
: -- but there is a very good reason for it: Ultimately RangeQuery still
: provides you with a meaningful score for each document, based on the
: frequency (and quantity) of terms that document has in the range [2]. In
: order to do that, it has to expand itself, but what if you don't care if
: your Range restriction impacts the Score? [3]
:
: Which brings us to...
:
: (c) Filtering. Filters in general make a lot of sense to me. They are a
: way to specify (at query time) that only a certain subset of the index
: should be considered for results. The Filter class has a very straight
: forward API that seems very easy to subclass to get the behavior I want.
: The Query API on the other hand ... I freely admit, that I can't make
: heads or tails out of it. I don't even know where I would begin to try
: and write a new subclass of Query if I wanted to.
:
: I would think that most people who want to do a "numeric range
: restriction" on their data, probably don't care about the Scoring benefits
: of RangeQuery. Looking at the code base, the way DateFilter works seems
: like it provides an ideal solution to any sort of Range restriction (not
: just Dates) that *should* be more efficient then using RangeQuery when
: dealing with an unbounded value set. (Both approaches need to iterate over
: all of the terms in the specified field using TermEnum, but RangeQuery has
: to build up an set of BooleanQuery objects for each matching term, and
: then each of those queries have to help score the documents -- DateFilter
: on the other hand only has to maintain a single BitSet of documents that
: it finds as it iterates)
:
: But I was surprised then to see the following quote from "Erik Hatcher" in
: the archives:
:
: "In fact, DateFilter by itself is practically of no use, I think." [4]
:
: ...Erik goes on to suggest that given "a set of canned date ranges", it
: doesn't really matter if you use a RangeQuery or a DateFilter -- as long
: as you cache them to reuse them (with something like CachingWrappingFilter
: or QueryFilter). I'm hoping that he might elaborate on that comment?
:
: As a test, I wrote a "RangeFilter" which borrows heavily from DateFilter
: to both convince myself it could work, and to do a comparison between it
: and RangeQuery. [5] Based on my limited tests, using a Filter to restrict
: to a Range is a lot faster then using RangeQuery -- independent of
: caching.
:
: The attachment contains my RangeFilter, a unit test that demonstrates it,
: and a Benchmarking unit test that does a side-by-side comparison with
: RangeQuery [6]. If developers feel that this class is useful, then by all
: means roll it into the code base. (90% of it is cut/pasted from
: DateFilter/RangeQuery anyway)
:
:
: Comments? ... Questions? ... Answers?
:
:
:
: Footnotes:
:
: [1] It seems to me this class is extremely useful, does anyone know
: if there's a particular reason it hasn't been added to the main Lucene
: codebase?
: http://www.mail-archive.com/lucene-dev@jakarta.apache.org/msg04790.html
:
: [2] Take a look at RangeQueryScoreDemo.java in the attachment, which
: produces output something like this...
: Range Search for: 'apple' TO 'dog'
: 0.40924072 ... bed dog emu
: 0.38014847 ... DOG
: 0.2825246 ... cat
: 0.17657787 ... apple emu
: 0.12671615 ... dog
:
: [3] According to the list archives "Matt Quail" mentioned in May that
: he was working on a "QuickRangeQuery" class that wouldn't have the
: BooleanQuery limitation, at the expense of always scoring "1.0",
: but I haven't seen any mention of anything like it since. Is Matt
: still an active list member? Matt, is this something you're still
: pursuing?
: http://nagoya.apache.org/eyebrowse/ReadMsg?msgId=1659395
:
: [4] http://www.mail-archive.com/lucene-user@jakarta.apache.org/msg07015.html
:
: [5] The only major difference between my RangeFilter and DateFilter is
: that RangeFilter supports options for inclusion/exclusion
: (individually for the low/high terms I might add). But for the
: purposes of a benchmark, doing the same thing with DateFilter would
: have worked fine.
:
: [6] If you have ant, see "ant -projecthelp"; otherwise, read the top
: of build.xml
:
:
:
: --
:
: -------------------------------------------------------------------
: "Oh, you're a tricky one." Chris M Hostetter
: -- Trisha Weir hossman@rescomp.berkeley.edu
:
:
--
-------------------------------------------------------------------
"Oh, you're a tricky one." Chris M Hostetter
-- Trisha Weir hossman@rescomp.berkeley.edu
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Chris Hostetter <ho...@fucit.org>.
: Note that I said FilteredQuery, not QueryFilter.
Doh .. right sorry, I confused myself by thinking you were still refering
to your comments 2004-03-29 comparing DateFilter with RangeQuery wrapped
in a QueryFilter.
: I debate (with myself) on whether add-ons that can be done with other
: code is worth adding to Lucene's core. In this case the utility
: methods are so commonly needed that it makes sense. But it could be
In particular, having a class of utilities like that in the code base is
usefull, because now the javadocs for classes like RangeQuery and
RangeFilter can refrence them as being neccessary important to ensure that
ranges work the way you expect ... and hopefully fewer people will be
confused in the future.
: I think there needs to be some discussion on what other utility methods
: should be added. For example, most of the numerics I index are
: positive integers and using a zero-padded is sufficient. I'd rather
: have clearly recognizable numbers in my fields than some strange
: contortion that requires a conversion process to see.
I'm of two minds, on one hand I think there's no big harm in providing
every concievable utility function known to man so people have their
choice of representation. On the other hand, I think it would be nice if
Lucene had a much simpler API for dealing with "non-strings" that just did
"the right thing" based on simple expectations -- without the user having
to ask themselves: "Will i ever need negative numbers? Will I ever need
numbers bigger then 1000?" or to later remember that they padded tis field
to 5 digits and that field to 7 digits.
Having clearly recognized values is something that can (should?) be easily
accomplished by indexing the contorted but lexically sortable value, and
storing the more readable value...
Document d = /* some doc */;
Long l = /* some value */;
Field f1 = Field.UnIndexed("field", l.toString());
Field f2 = Field.UnStored("field", NumerTools.longToString(l));
d.add(f1);
d.add(f2);
(I'm not imagining things right? that should work, correct?)
What would really be sweet, Is if Lucene had an API that
transparently dealt with all of the major primitive types, both at
indexing time and at query time, so that users ddn't have to pay any
attention to the stringification, or when to Index a different value
then they store...
Field f = Field.Long("field", l); /* indexes one string, stores the other */
d.add(f);
...
Query q = new RangeQuery("field", l1, l2); /* knows to use the contorted string */
...
String s = hits.doc(i).getValue("field"); /* returns pretty string */
Long l = hits.doc(i).getValue("field"); /* returns orriginal Long */
--
-------------------------------------------------------------------
"Oh, you're a tricky one." Chris M Hostetter
-- Trisha Weir hossman@rescomp.berkeley.edu
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Nov 23, 2004, at 3:41 PM, Erik Hatcher wrote:
> On Nov 23, 2004, at 2:16 PM, Chris Hostetter wrote:
>> First: Is there any reason Matt Quail's "LongField" class hasn't been
>> added to CVS (or has it and I'm just not seeing it?)
>
> Laziness is the only reason, at least on my part. I think adding it
> is a great thing. I'll look into it.
I'm feeling particularly commit-y today. I dug up Matt Quail's
original LongField contribution in e-mail and adapted it to a new
NumberTools class. I committed it along with the tests he contributed
also.
I think there needs to be some discussion on what other utility methods
should be added. For example, most of the numerics I index are
positive integers and using a zero-padded is sufficient. I'd rather
have clearly recognizable numbers in my fields than some strange
contortion that requires a conversion process to see.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Nov 23, 2004, at 2:16 PM, Chris Hostetter wrote:
> : I did a little code cleanup, Chris, renaming some RangeFilter
> variables
> : and correcting typos in the Javadocs. Let me know if everything
> looks
> : ok.
>
> Wow ... that was fast. Things look fine to me (typo's in javadocs are
> my
> specialty) but now I wish I'd included more tests
We can always add more tests. Anytime.
> First: Is there any reason Matt Quail's "LongField" class hasn't been
> added to CVS (or has it and I'm just not seeing it?)
Laziness is the only reason, at least on my part. I think adding it is
a great thing. I'll look into it.
> I haven't tested it extensively, but strikes me as being a crucial
> utility
> for people who want to do any serious sorting or filtering of numeric
> values.
I debate (with myself) on whether add-ons that can be done with other
code is worth adding to Lucene's core. In this case the utility
methods are so commonly needed that it makes sense. But it could be
argued also that there are are classes in Lucene that are not central
to its operation.
> Although I would suggest a few minor tweaks:
> a) Rename to something like NumberTools (to be consistent with the
> new
> DateTools and because...)
Agreed.
> b) Add some one line convinience methods like intToString and
> floatToString and doubleToString ala:
> return longToString(Double.doubleToLongBits(d));
No objects to having convenience methods - though I need to look at
what the LongField code is providing before commenting in detail.
> : And now with FilteredQuery you can have the best of both worlds :)
>
> See, this is what I'm not getting: what is the advantage of the second
> world? :) ... in what situations would using...
>
> s.search(q1, new QueryFilter(new RangeQuery(t1,t2,true));
>
> ...be a better choice then...
>
> s.search(q1, new
> RangeFilter(t1.field(),t1.text(),t2.text(),true,true);
Note that I said FilteredQuery, not QueryFilter.
Certainly RangeFilter is cleaner than using a QueryFilter(RangeFilter)
combination - that's why we added it. :)
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Yonik Seeley <ys...@yahoo.com>.
Hmmm, scratch that. I explained the tradeoff of a
filter vs a range query - not between the different
types of filters you talk about.
--- Yonik Seeley <ys...@yahoo.com> wrote:
> I think it depends on the query. If the query (q1)
> covers a large number of documents and the fiter
> covers a very small number, then using a RangeFilter
> will probably be slower than a RangeQuery.
>
> -Yonik
>
>
> > See, this is what I'm not getting: what is the
> > advantage of the second
> > world? :) ... in what situations would using...
> >
> > s.search(q1, new QueryFilter(new
> > RangeQuery(t1,t2,true));
> >
> > ...be a better choice then...
> >
> > s.search(q1, new
> >
>
RangeFilter(t1.field(),t1.text(),t2.text(),true,true);
>
>
>
> __________________________________________________
> Do You Yahoo!?
> Tired of spam? Yahoo! Mail has the best spam
> protection around
> http://mail.yahoo.com
>
>
---------------------------------------------------------------------
> To unsubscribe, e-mail:
> lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
>
>
__________________________________
Do you Yahoo!?
Meet the all-new My Yahoo! - Try it today!
http://my.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Yonik Seeley <ys...@yahoo.com>.
I think it depends on the query. If the query (q1)
covers a large number of documents and the fiter
covers a very small number, then using a RangeFilter
will probably be slower than a RangeQuery.
-Yonik
> See, this is what I'm not getting: what is the
> advantage of the second
> world? :) ... in what situations would using...
>
> s.search(q1, new QueryFilter(new
> RangeQuery(t1,t2,true));
>
> ...be a better choice then...
>
> s.search(q1, new
>
RangeFilter(t1.field(),t1.text(),t2.text(),true,true);
__________________________________________________
Do You Yahoo!?
Tired of spam? Yahoo! Mail has the best spam protection around
http://mail.yahoo.com
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Chris Hostetter <ho...@fucit.org>.
: Done. I deprecated DateField and DateFilter, and added the RangeFilter
: class contributed by Chris.
:
: I did a little code cleanup, Chris, renaming some RangeFilter variables
: and correcting typos in the Javadocs. Let me know if everything looks
: ok.
Wow ... that was fast. Things look fine to me (typo's in javadocs are my
specialty) but now I wish I'd included more tests
I still feel a little confused about two things though...
First: Is there any reason Matt Quail's "LongField" class hasn't been
added to CVS (or has it and I'm just not seeing it?)
I haven't tested it extensively, but strikes me as being a crucial utility
for people who want to do any serious sorting or filtering of numeric
values.
Although I would suggest a few minor tweaks:
a) Rename to something like NumberTools (to be consistent with the new
DateTools and because...)
b) Add some one line convinience methods like intToString and
floatToString and doubleToString ala:
return longToString(Double.doubleToLongBits(d));
Second...
: RangeQuery wrapped inside a QueryFilter is more specifically what I
: said. I'm not a fan of DateField and how the built-in date support in
: Lucene works, so this is why I don't like DateFilter personally.
:
: Your RangeFilter, however, is nicely done and well worth deprecating
: DateFilter for.
[...]
: > and RangeQuery. [5] Based on my limited tests, using a Filter to
: > restrict
: > to a Range is a lot faster then using RangeQuery -- independent of
: > caching.
:
: And now with FilteredQuery you can have the best of both worlds :)
See, this is what I'm not getting: what is the advantage of the second
world? :) ... in what situations would using...
s.search(q1, new QueryFilter(new RangeQuery(t1,t2,true));
...be a better choice then...
s.search(q1, new RangeFilter(t1.field(),t1.text(),t2.text(),true,true);
?
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Nov 22, 2004, at 9:25 PM, Hoss wrote:
> I'm rather new to Lucene (and this list), so if I'm grossly
> misunderstanding things, forgive me.
You're spot on!
> But I was surprised then to see the following quote from "Erik
> Hatcher" in
> the archives:
>
> "In fact, DateFilter by itself is practically of no use, I think."
> [4]
>
> ...Erik goes on to suggest that given "a set of canned date ranges", it
> doesn't really matter if you use a RangeQuery or a DateFilter -- as
> long
> as you cache them to reuse them (with something like
> CachingWrappingFilter
> or QueryFilter). I'm hoping that he might elaborate on that comment?
RangeQuery wrapped inside a QueryFilter is more specifically what I
said. I'm not a fan of DateField and how the built-in date support in
Lucene works, so this is why I don't like DateFilter personally.
Your RangeFilter, however, is nicely done and well worth deprecating
DateFilter for.
> As a test, I wrote a "RangeFilter" which borrows heavily from
> DateFilter
> to both convince myself it could work, and to do a comparison between
> it
> and RangeQuery. [5] Based on my limited tests, using a Filter to
> restrict
> to a Range is a lot faster then using RangeQuery -- independent of
> caching.
And now with FilteredQuery you can have the best of both worlds :)
Thanks for your detailed code, tests, and contribution. We'll fold it
in.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Morus Walter <mo...@tanto.de>.
Hoss writes:
>
> (c) Filtering. Filters in general make a lot of sense to me. They are a
> way to specify (at query time) that only a certain subset of the index
> should be considered for results. The Filter class has a very straight
> forward API that seems very easy to subclass to get the behavior I want.
> The Query API on the other hand ... I freely admit, that I can't make
> heads or tails out of it. I don't even know where I would begin to try
> and write a new subclass of Query if I wanted to.
>
> I would think that most people who want to do a "numeric range
> restriction" on their data, probably don't care about the Scoring benefits
> of RangeQuery. Looking at the code base, the way DateFilter works seems
> like it provides an ideal solution to any sort of Range restriction (not
> just Dates) that *should* be more efficient then using RangeQuery when
> dealing with an unbounded value set. (Both approaches need to iterate over
> all of the terms in the specified field using TermEnum, but RangeQuery has
> to build up an set of BooleanQuery objects for each matching term, and
> then each of those queries have to help score the documents -- DateFilter
> on the other hand only has to maintain a single BitSet of documents that
> it finds as it iterates)
>
IMO there's another option, at least as long as the number of your documents
isn't too high.
Sorting already creates a list of all field values for some field that
will be used during the search for sorting.
Nothing prevents you from using that aproach for search restriction also.
The advantage is, that you can create that list once and use it for different
ranges until the index is changed whereas a filter can only represent
one range.
The disadvantate is, that you have to keep one value for each document in
memory instead of one bit in a filter.
I did that (before the sort code was introduced) for date queries in order
to be able to sort and restrict searches on dates.
But I haven't thought about how a general API for such a solution might look
like so far.
Of course it depends on a number of questions, which way is preferable.
How often is the index modified, are range queries usually done for the
same or different ranges, how many documents are indexed and so on.
Morus
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Paul Elschot <pa...@xs4all.nl>.
Chris,
On Tuesday 23 November 2004 03:25, Hoss wrote:
> (NOTE: numbers in [] indicate Footnotes)
>
> I'm rather new to Lucene (and this list), so if I'm grossly
> misunderstanding things, forgive me.
>
> One of my main needs as I investigate Search technologies is to restrict
> results based on Ranges of numeric values. Looking over the archives of
> this list, it seems that lots of people have run into problems dealing
> with this. In particular, whenever someone asks a question about "Numeric
> Ranges" the question seem to always involve one (or more) of the
> following:
>
> (a) Lexical sorting puts 11 in the range "1 TO 5"
> (b) Dates (or Dates and Times)
> (c) BooleanQuery$TooManyClauses Exceptions
> (d) Should I use a filter?
FWIW, the javadoc of the development version of
BooleanQuery.maxClauseCount reads:
The maximum number of clauses permitted. Default value is 1024. Use the
org.apache.lucene.maxClauseCount system property to override.
TermQuery clauses are generated from for example prefix queries and
fuzzy queries. Each TermQuery needs some buffer space during search,
so this parameter indirectly controls the maximum buffer requirements for
query search. Normally the buffers are allocated by the JVM. When using
for example MMapDirectory the buffering is left to the operating system.
MMapDirectory uses memory mapped files for the index.
It would be useful to also provide a reference to filters (DateFilter)
and to LongField in case it is added to the code base.
...
> The Query API on the other hand ... I freely admit, that I can't make
> heads or tails out of it. I don't even know where I would begin to try
> and write a new subclass of Query if I wanted to.
In a nutshell:
A Query either rewrites to another Query, or it provides a Weight.
A Weight first does normalisation and then provides a Scorer
to be used during search.
RangeQuery is a good example:
A RangeQuery rewrites to a BooleanQuery over TermQuery's
for the matching terms.
A BooleanQuery provides a BooleanScorer via its Weight.
A TermQuery provides a TermScorer via its Weight.
Regards,
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Nov 23, 2004, at 10:01 AM, Praveen Peddi wrote:
> Chris's RangeFilter does not cache anything where as QueryFilter does
> caching. Is it better to add the caching funtionality to RangeFilter
> also? or does it not make any difference?
Caching is a different _aspect_. Filtering and caching are not related
and should not be intimately tied, in my opinion. The solution is to
use the CachingWrapperFilter to wrap a RangeFilter when caching is
desired.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Praveen Peddi <pp...@contextmedia.com>.
Chris's RangeFilter does not cache anything where as QueryFilter does
caching. Is it better to add the caching funtionality to RangeFilter also?
or does it not make any difference?
Praveen
----- Original Message -----
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Tuesday, November 23, 2004 9:19 AM
Subject: Re: Numeric Range Restrictions: Queries vs Filters
> On Nov 23, 2004, at 4:18 AM, Doug Cutting wrote:
>> Hoss wrote:
>>> The attachment contains my RangeFilter, a unit test that demonstrates
>>> it,
>>> and a Benchmarking unit test that does a side-by-side comparison with
>>> RangeQuery [6]. If developers feel that this class is useful, then by
>>> all
>>> means roll it into the code base. (90% of it is cut/pasted from
>>> DateFilter/RangeQuery anyway)
>>
>> +1
>>
>> DateFilter could be deprecated, and replaced with the more generally and
>> appropriately named RangeFilter. Should we also deprecate DateField, in
>> preference for DateTools?
>
> Done. I deprecated DateField and DateFilter, and added the RangeFilter
> class contributed by Chris.
>
> I did a little code cleanup, Chris, renaming some RangeFilter variables
> and correcting typos in the Javadocs. Let me know if everything looks ok.
>
> Erik
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Nov 23, 2004, at 4:18 AM, Doug Cutting wrote:
> Hoss wrote:
>> The attachment contains my RangeFilter, a unit test that demonstrates
>> it,
>> and a Benchmarking unit test that does a side-by-side comparison with
>> RangeQuery [6]. If developers feel that this class is useful, then
>> by all
>> means roll it into the code base. (90% of it is cut/pasted from
>> DateFilter/RangeQuery anyway)
>
> +1
>
> DateFilter could be deprecated, and replaced with the more generally
> and appropriately named RangeFilter. Should we also deprecate
> DateField, in preference for DateTools?
Done. I deprecated DateField and DateFilter, and added the RangeFilter
class contributed by Chris.
I did a little code cleanup, Chris, renaming some RangeFilter variables
and correcting typos in the Javadocs. Let me know if everything looks
ok.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: Numeric Range Restrictions: Queries vs Filters
Posted by Doug Cutting <cu...@apache.org>.
Hoss wrote:
> The attachment contains my RangeFilter, a unit test that demonstrates it,
> and a Benchmarking unit test that does a side-by-side comparison with
> RangeQuery [6]. If developers feel that this class is useful, then by all
> means roll it into the code base. (90% of it is cut/pasted from
> DateFilter/RangeQuery anyway)
+1
DateFilter could be deprecated, and replaced with the more generally and
appropriately named RangeFilter. Should we also deprecate DateField, in
preference for DateTools?
Doug
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org