You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by Robert Muir <rc...@gmail.com> on 2010/10/27 12:29:11 UTC
inconsistency/performance trap of empty terms
Hello,
Mike and I were discussing some very unrelated stuff and the question
of how to handle the empty term came up...
I started thinking about this email:
http://www.lucidimagination.com/search/document/a8d3a8647e581a5b/patternreplacefilterfactory_creating_empty_string_as_a_term#f43d167b91c2ba07
So, looking through the analyzers, I think we should make a decision
about what to do with empty terms.
In my opinion there is a performance trap here, that might work like this:
1. a user, particularly say a solr user is using a combination of
tokenizers/filters and ends out with the "empty term" as basically a
mega-stopword, like what happened to that user.
2. due to this, their queries have terrible performance, especially if
they are 'auto-generating phrase queries' (the solr default)
3. but, its not possible that anyone can really even rely upon the
analyzers handling empty terms correctly, because we are so
inconsistent about it.
Just taking a quick glance through the analyzers, i noticed each one
seems to have willy-nilly code/TODO's regarding this empty term.
for example, the n-gramish tokenizers such as CJKTokenizer,
CommonGramsFilter, NGramTokenizer, etc explicitly avoid creating
these.
But there are inconsistencies:
TrimFilter explicitly creates/maintains empty terms.
NGramFilter doesnt seem to have this check, but the NGramTokenizer does.
PatternFilter documents it might create empty terms, but the
PatternTokenizer avoids them.
I am sure some of the stemmers probably create empty terms in some
situations (eg maybe it removes -alization suffix, but has no length
check, and if the term is "alization" it makes empty terms)
Anyway, I think its possible other users might be in this same
situation, with slow performance, and not even realizing it yet...
Obviously they can fix this if they go and add LengthFilter, but
should we be doing something different?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Andi Vajda <va...@osafoundation.org>.
On Oct 28, 2010, at 22:32, Robert Muir <rc...@gmail.com> wrote:
> On Thu, Oct 28, 2010 at 10:28 PM, Andi Vajda
> <va...@osafoundation.org> wrote:
>>
>> I've used this in a URL index. I needed to be able to distinguish
>> between
>> searching URLs that had, say, no path, from searching URLs without
>> matching
>> the path component. The absence of path was represented with an
>> empty token
>> in the path field.
>>
>
> but you didn't really need to use the empty term... you could have
> used something like U+001F INFORMATION SEPARATOR... and your whole
> index would have been an entire byte bigger?
Of course, anything not found in a url path could serve as 'empty'.
Andi..
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Thu, Oct 28, 2010 at 10:28 PM, Andi Vajda <va...@osafoundation.org> wrote:
>
> I've used this in a URL index. I needed to be able to distinguish between
> searching URLs that had, say, no path, from searching URLs without matching
> the path component. The absence of path was represented with an empty token
> in the path field.
>
but you didn't really need to use the empty term... you could have
used something like U+001F INFORMATION SEPARATOR... and your whole
index would have been an entire byte bigger?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Andi Vajda <va...@osafoundation.org>.
On Thu, 28 Oct 2010, Robert Muir wrote:
> On Thu, Oct 28, 2010 at 8:35 PM, <ka...@nokia.com> wrote:
>> In database queries, it is often useful to treat an empty value specially, and be able to search explicitly for records that have (for instance) no field X, or no value for field X. I can't regurgitate offhand all the precise situations that I've used this and claim that they would apply to a search engine, but it is conceivable that it could be helpful to somebody. Would your proposed change preclude current or future support for such null queries?
>
> in a database, not having a value for field X is 'null'.
>
> 1. null is different than empty term.
> 2. comparing this concept with an inverted index vs a database record
> really isn't a comparison.
>
> by not having 'empty' terms (terms of length=0), what searches would
> be affected.
>
> I still haven't heard a real use case (though surely perhaps there is
> someone abusing this somehow), but there is a serious performance trap
> that is definitely real.
I've used this in a URL index. I needed to be able to distinguish between
searching URLs that had, say, no path, from searching URLs without matching
the path component. The absence of path was represented with an empty token
in the path field.
Andi..
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Thu, Oct 28, 2010 at 8:35 PM, <ka...@nokia.com> wrote:
> In database queries, it is often useful to treat an empty value specially, and be able to search explicitly for records that have (for instance) no field X, or no value for field X. I can't regurgitate offhand all the precise situations that I've used this and claim that they would apply to a search engine, but it is conceivable that it could be helpful to somebody. Would your proposed change preclude current or future support for such null queries?
in a database, not having a value for field X is 'null'.
1. null is different than empty term.
2. comparing this concept with an inverted index vs a database record
really isn't a comparison.
by not having 'empty' terms (terms of length=0), what searches would
be affected.
I still haven't heard a real use case (though surely perhaps there is
someone abusing this somehow), but there is a serious performance trap
that is definitely real.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: inconsistency/performance trap of empty terms
Posted by ka...@nokia.com.
In database queries, it is often useful to treat an empty value specially, and be able to search explicitly for records that have (for instance) no field X, or no value for field X. I can't regurgitate offhand all the precise situations that I've used this and claim that they would apply to a search engine, but it is conceivable that it could be helpful to somebody. Would your proposed change preclude current or future support for such null queries?
Karl
-----Original Message-----
From: ext Robert Muir [mailto:rcmuir@gmail.com]
Sent: Thursday, October 28, 2010 8:21 PM
To: dev@lucene.apache.org
Subject: Re: inconsistency/performance trap of empty terms
On Thu, Oct 28, 2010 at 7:59 PM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : Anyway, I think its possible other users might be in this same
> : situation, with slow performance, and not even realizing it yet...
> : Obviously they can fix this if they go and add LengthFilter, but
> : should we be doing something different?
>
> On one level, ithink a big improvement might just be to start encouraging
> more use of LengthFilter with min=1 at the end of analyzers by including
> it at the end of more "example" field types -- we should probably end
> every analyzer with that and RemoveDuplicatesTokenFilterFactory as a
> general pattern.
why not just discard them completely in say, indexer/queryparser ?
>
> How individual Tokenizers and TokenFilters deal with empty tokens seems
> like something that should be cases by case -- the Ngram classes should
> allow/create them if the "min" value is 0, the pattern based classes
> should create them if the pattern matches and empty string, etc....
why should they create them? is there some use case for the empty term
that you have found (because i can't think of a use case, except
making your search engine slower!)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Sat, Oct 30, 2010 at 12:51 PM, DM Smith <dm...@gmail.com> wrote:
>
> On Oct 30, 2010, at 12:00 PM, Robert Muir wrote:
>
>> On Sat, Oct 30, 2010 at 11:54 AM, Yonik Seeley
>> <yo...@lucidimagination.com> wrote:
>>> If it's only for the QP, a simple method that one could override would suffice:
>>> QueryParser.getTokenStream(String field, String value)
>>>
>>> If it's not just for the QP, then we have Analyzer (as you've pointed out).
>>>
>>>
>>
>> right, but if we did this, it makes some things tricky (e.g. the user
>> has to manage reset(Reader)/reset() tokenStream reuse).
>> A tokenizer/tokenfilter they are using could be "heavy" in terms of
>> initialization cost.
>
> Maybe I'm missing something here. Can't there be an empty analyzer that takes a TokenStream as an argument to its constructor and wraps it with all the reuse goodness?
you have to also wrap the string in a reader, call
Tokenizer.reset(Reader), and call reset() on the entire chain [unless
in a method like this, the QP itself would be responsible, its not
clear]
so that means, the client app has to separately track the Tokenizer
(which needs its reader reset), and the Tokenstream chain (at least to
return its value, maybe reset it).
just saying, if you do this, you didn't get rid of Analyzer, you just
made everyone write their own in their QueryParser subclass.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by DM Smith <dm...@gmail.com>.
On Oct 30, 2010, at 12:00 PM, Robert Muir wrote:
> On Sat, Oct 30, 2010 at 11:54 AM, Yonik Seeley
> <yo...@lucidimagination.com> wrote:
>> If it's only for the QP, a simple method that one could override would suffice:
>> QueryParser.getTokenStream(String field, String value)
>>
>> If it's not just for the QP, then we have Analyzer (as you've pointed out).
>>
>>
>
> right, but if we did this, it makes some things tricky (e.g. the user
> has to manage reset(Reader)/reset() tokenStream reuse).
> A tokenizer/tokenfilter they are using could be "heavy" in terms of
> initialization cost.
Maybe I'm missing something here. Can't there be an empty analyzer that takes a TokenStream as an argument to its constructor and wraps it with all the reuse goodness?
>
> anyway, i'd like to get rid of Analyzer just as much as anyone, but
> making the API difficult/fragile/non-performant isn't a good way
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Sat, Oct 30, 2010 at 11:54 AM, Yonik Seeley
<yo...@lucidimagination.com> wrote:
> If it's only for the QP, a simple method that one could override would suffice:
> QueryParser.getTokenStream(String field, String value)
>
> If it's not just for the QP, then we have Analyzer (as you've pointed out).
>
>
right, but if we did this, it makes some things tricky (e.g. the user
has to manage reset(Reader)/reset() tokenStream reuse).
A tokenizer/tokenfilter they are using could be "heavy" in terms of
initialization cost.
anyway, i'd like to get rid of Analyzer just as much as anyone, but
making the API difficult/fragile/non-performant isn't a good way
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Sat, Oct 30, 2010 at 11:43 AM, Robert Muir <rc...@gmail.com> wrote:
> On Sat, Oct 30, 2010 at 11:40 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
>> On Sat, Oct 30, 2010 at 18:49, Uwe Schindler <uw...@thetaphi.de> wrote:
>>> In my opinion, we should not have analyzers at all (just my personal opinion). "new Field(name, TokenStream)" is much enough from consistency standpoint!
>>
>> Indeed, my friend!
>>
>
> i agree (for the indexer), but a lot of people like to use the
> QueryParser.... so we need a "tokenstream-factory" of some sort
> (Analyzer today).
If it's only for the QP, a simple method that one could override would suffice:
QueryParser.getTokenStream(String field, String value)
If it's not just for the QP, then we have Analyzer (as you've pointed out).
-Yonik
http://www.lucidimagination.com
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Sat, Oct 30, 2010 at 11:40 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
> On Sat, Oct 30, 2010 at 18:49, Uwe Schindler <uw...@thetaphi.de> wrote:
>> In my opinion, we should not have analyzers at all (just my personal opinion). "new Field(name, TokenStream)" is much enough from consistency standpoint!
>
> Indeed, my friend!
>
i agree (for the indexer), but a lot of people like to use the
QueryParser.... so we need a "tokenstream-factory" of some sort
(Analyzer today).
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Earwin Burrfoot <ea...@gmail.com>.
On Sat, Oct 30, 2010 at 18:49, Uwe Schindler <uw...@thetaphi.de> wrote:
> In my opinion, we should not have analyzers at all (just my personal opinion). "new Field(name, TokenStream)" is much enough from consistency standpoint!
Indeed, my friend!
--
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Phone: +7 (495) 683-567-4
ICQ: 104465785
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by DM Smith <dm...@gmail.com>.
On Oct 30, 2010, at 10:49 AM, Uwe Schindler wrote:
> we should not have analyzers at all (just my personal opinion)
+1 It should be sufficient to document order of their parts. They could be constructed from a spec using reflection.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: inconsistency/performance trap of empty terms
Posted by Uwe Schindler <uw...@thetaphi.de>.
+1, too. I hate those modes. In my opinion, we should not have analyzers at all (just my personal opinion). "new Field(name, TokenStream)" is much enough from consistency standpoint!
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Robert Muir [mailto:rcmuir@gmail.com]
> Sent: Saturday, October 30, 2010 3:15 PM
> To: dev@lucene.apache.org
> Subject: Re: inconsistency/performance trap of empty terms
>
> On Sat, Oct 30, 2010 at 9:00 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
> > Speaking of consistency, I think NOT_ANALYZED is superfluous. Drop
> > this mode, and it can be safely reproduced by a NotAnalyzingAnalyzer
> > (insert better name here).
> >
>
> +1. This is confusing and comes up often on the user list.
>
> The way I think it happens is like this:
> Joe Schmoe, like a good user, just fires up StandardAnalyzer at both index and
> query time.
> Joe realizes he has a field that really shouldnt be tokenized, and sets it to
> NOT_ANALYZED.
> Joe is confused that queries dont work the way he should when he does this,
> since its still analyzed by the queryparser with StandardAnalyzer.
>
> It would be far better to force him to use PerFieldAnalyzerWrapper +
> NotAnalyzingAnalyzer or whatever, since then it would work intuitively.
> besides, if he sets this NotAnalyzed, it actually goes thru 'analysis'
> anyway: SingleTokenAttributeSource buried in the indexer.
> And, in trunk, this means things like UTF-8 encoding are assumed, but really
> this should be completely outside of the indexer.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Sat, Oct 30, 2010 at 9:00 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
> Speaking of consistency, I think NOT_ANALYZED is superfluous. Drop
> this mode, and it can be safely reproduced by a NotAnalyzingAnalyzer
> (insert better name here).
>
+1. This is confusing and comes up often on the user list.
The way I think it happens is like this:
Joe Schmoe, like a good user, just fires up StandardAnalyzer at both
index and query time.
Joe realizes he has a field that really shouldnt be tokenized, and
sets it to NOT_ANALYZED.
Joe is confused that queries dont work the way he should when he does
this, since its still analyzed by the queryparser with
StandardAnalyzer.
It would be far better to force him to use PerFieldAnalyzerWrapper +
NotAnalyzingAnalyzer or whatever, since then it would work
intuitively.
besides, if he sets this NotAnalyzed, it actually goes thru 'analysis'
anyway: SingleTokenAttributeSource buried in the indexer.
And, in trunk, this means things like UTF-8 encoding are assumed, but
really this should be completely outside of the indexer.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Earwin Burrfoot <ea...@gmail.com>.
Speaking of consistency, I think NOT_ANALYZED is superfluous. Drop
this mode, and it can be safely reproduced by a NotAnalyzingAnalyzer
(insert better name here).
Agree with infinity scores and other artefacts. Corner cases lead to
corner cases, that's fine.
As was proposed earlier, we can shield layman users with LengthFilter
included in all out-of-the-box Analyzers.
On Sat, Oct 30, 2010 at 15:25, Robert Muir <rc...@gmail.com> wrote:
> On Sat, Oct 30, 2010 at 7:19 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
>> I'd say support them everywhere, and slip LengthFilter into all the
>> standard Analyzers, so people won't hit empty terms unless they opt-in
>> for it.
>> This is a most consistent approach.
>>
>
> And so there's no confusion, i want complete consistency... that
> includes KeywordAnalyzer in this case.
>
> what about IW's NOT_ANALYZED? lots of people use this and might not
> realize it will make empty terms, too.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Phone: +7 (495) 683-567-4
ICQ: 104465785
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Sat, Oct 30, 2010 at 7:19 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
> I'd say support them everywhere, and slip LengthFilter into all the
> standard Analyzers, so people won't hit empty terms unless they opt-in
> for it.
> This is a most consistent approach.
>
And so there's no confusion, i want complete consistency... that
includes KeywordAnalyzer in this case.
what about IW's NOT_ANALYZED? lots of people use this and might not
realize it will make empty terms, too.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Sat, Oct 30, 2010 at 7:19 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
> I'd say support them everywhere, and slip LengthFilter into all the
> standard Analyzers, so people won't hit empty terms unless they opt-in
> for it.
> This is a most consistent approach.
>
I'm definitely ok with an approach like that... just anything consistent really!
as far as queries go, i think you know, if a query says it scales
value over term length like fuzzyquery/spellchecking does, and you get
Infinity as the score, there shouldn't be any complaints... this is
completely correct behavior.
if you are gonna use the empty term, then you need to know what you
are getting into and be ready to deal with things like this.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Earwin Burrfoot <ea...@gmail.com>.
I'd say support them everywhere, and slip LengthFilter into all the
standard Analyzers, so people won't hit empty terms unless they opt-in
for it.
This is a most consistent approach.
On Sat, Oct 30, 2010 at 15:06, Robert Muir <rc...@gmail.com> wrote:
> On Sat, Oct 30, 2010 at 7:01 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
>> Mathematically an inverted index is keyed by strings. Any strings.
>> Empty term is just a case of a string of length 0.
>> So, for consistency, Lucene should support them. TermsEnum.seek("")
>> should position you into very beginning of terms list, etc.
>> If you drop the support, you have to check zero length damn
>> eeeeverywhere in the API where you accept terms. Or, thoroughly
>> document unpredictable erratic behaviour :)
>
> well, we are checking this already, in a lot of the analyzers.
>
> as i said originally, the biggest problems that we *must* solve are:
> 1. try to prevent the performance trap i mentioned, where people
> create the empty term as a mega-stopword without realizing it.
> 2. fix the analyzers to be consistent with regards to the empty
> term... for example, if we decide the empty term is supported, then
> they shouldnt be arbitrarily removing empty-term tokens.
>
> as far as TermsEnum, i myself have already had to special-case the
> empty term in TermsEnum implementations before... and I'm pretty
> fucking sure that we have long-standing bugs if you have an empty-term
> anywhere in your index (e.g. FuzzyQuery will divide by 0 to scale the
> boost, and you will get a strange exception from your collector
> because it will then have NaN/Inf/some sentinel value).
>
> just saying, its problematic today, doing nothing and leaving it the
> messy unambiguous situation it is now is no option.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
--
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Phone: +7 (495) 683-567-4
ICQ: 104465785
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Sat, Oct 30, 2010 at 7:01 AM, Earwin Burrfoot <ea...@gmail.com> wrote:
> Mathematically an inverted index is keyed by strings. Any strings.
> Empty term is just a case of a string of length 0.
> So, for consistency, Lucene should support them. TermsEnum.seek("")
> should position you into very beginning of terms list, etc.
> If you drop the support, you have to check zero length damn
> eeeeverywhere in the API where you accept terms. Or, thoroughly
> document unpredictable erratic behaviour :)
well, we are checking this already, in a lot of the analyzers.
as i said originally, the biggest problems that we *must* solve are:
1. try to prevent the performance trap i mentioned, where people
create the empty term as a mega-stopword without realizing it.
2. fix the analyzers to be consistent with regards to the empty
term... for example, if we decide the empty term is supported, then
they shouldnt be arbitrarily removing empty-term tokens.
as far as TermsEnum, i myself have already had to special-case the
empty term in TermsEnum implementations before... and I'm pretty
fucking sure that we have long-standing bugs if you have an empty-term
anywhere in your index (e.g. FuzzyQuery will divide by 0 to scale the
boost, and you will get a strange exception from your collector
because it will then have NaN/Inf/some sentinel value).
just saying, its problematic today, doing nothing and leaving it the
messy unambiguous situation it is now is no option.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Earwin Burrfoot <ea...@gmail.com>.
On Fri, Oct 29, 2010 at 21:50, Robert Muir <rc...@gmail.com> wrote:
> I was suggesting that mathematically, the empty term makes no sense in
> an inverted index, and we shouldn't allow it.
> Its one solution.
Mathematically an inverted index is keyed by strings. Any strings.
Empty term is just a case of a string of length 0.
So, for consistency, Lucene should support them. TermsEnum.seek("")
should position you into very beginning of terms list, etc.
If you drop the support, you have to check zero length damn
eeeeverywhere in the API where you accept terms. Or, thoroughly
document unpredictable erratic behaviour :)
A possible usecase for empty terms in analyzer stream is slipping in
various metadata. Paragraph/sentence delimiters, whatever. Nobody
precludes you from using "##PAR#BEGIN##" kind of things, but you may
want to leave term text alone and exploit other attributes.
--
Kirill Zakharenko/Кирилл Захаренко (earwin@gmail.com)
Phone: +7 (495) 683-567-4
ICQ: 104465785
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Fri, Oct 29, 2010 at 1:44 PM, Chris Hostetter
<ho...@fucit.org> wrote:
> In the IndexWriter: it seems like a bad idea.
it doesn't have to be there. but we have to do something.
Currently we have the worst of both worlds:
1. the empty term causes performance problems for people who don't
realize whats going on (especially with their analyzers etc)
2. the empty term isnt reliable and doesnt work anyway, lots of
analysis components throw it away.
We should decide if we either:
A. support the empty term.
B. do not support the empty term.
I was suggesting that mathematically, the empty term makes no sense in
an inverted index, and we shouldn't allow it.
Its one solution.
The other solution is to fully support it everywhere... but we have to
decide, it can't be the ambiguous situation it is today.
>
> Low level Lucene really shouldn't be making any assumptions about *how*
> the client code is using the library -- you and i may not have any good
> reasons for wanting an empty term, but we shouldn't put that as a
> hardcoded assumption in the low level code.
I think we can make a safe assumption here, the empty term makes *no
sense* for an inverted index.
I could argue that every document has this!
>
> It's essentially the converse issue of IndexWriter.maxFieldLength --
> which was deliberately changed to default to Integer.MAX_VALUE precisesly
> because of this "don't assume we know how people are using the library"
> issue -- but we could certianly make it configurable in the same way.
No its not, because we are talking about the empty *term* not an empty *field*.
An empty term makes no sense.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Ryan McKinley <ry...@gmail.com>.
Reading the discussion here... I think Roberts main point is that
lucene lets you make empty terms, but has inconsistent behavior with
them. We should either remove support, or make consistent behavior.
Sure, there are lots of options to avoid the problems. But should it
be necessary to work to avoid them?
On Fri, Oct 29, 2010 at 1:53 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> We don't even need the TF, LengthFilter does the job very easy, just set
> minTermLength to 1 and maxTermLength to Integer.MAX_VALUE.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
>> -----Original Message-----
>> From: Uwe Schindler [mailto:uwe@thetaphi.de]
>> Sent: Friday, October 29, 2010 7:49 PM
>> To: dev@lucene.apache.org
>> Subject: RE: inconsistency/performance trap of empty terms
>>
>> I am for the tokenfilter approach. Max Field Length is still to be
> deprecated in
>> favour of the TokenFilter.
>>
>> TF is very easy, just loop over incrementToken() until it returns false or
> a
>> termLength>0
>>
>> -----
>> Uwe Schindler
>> H.-H.-Meier-Allee 63, D-28213 Bremen
>> http://www.thetaphi.de
>> eMail: uwe@thetaphi.de
>>
>>
>> > -----Original Message-----
>> > From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
>> > Sent: Friday, October 29, 2010 7:45 PM
>> > To: Lucene Dev
>> > Subject: Re: inconsistency/performance trap of empty terms
>> >
>> >
>> > : why not just discard them completely in say, indexer/queryparser ?
>> >
>> > In QueryParser: maybe, that's a high level API with assumptions about
>> "human"
>> > interaction and text.
>> >
>> > In the IndexWriter: it seems like a bad idea.
>> >
>> > Low level Lucene really shouldn't be making any assumptions about
>> > *how*
>> the
>> > client code is using the library -- you and i may not have any good
>> reasons for
>> > wanting an empty term, but we shouldn't put that as a hardcoded
>> > assumption in the low level code.
>> >
>> > It's essentially the converse issue of IndexWriter.maxFieldLength --
>> > which
>> was
>> > deliberately changed to default to Integer.MAX_VALUE precisesly
>> > because of this "don't assume we know how people are using the library"
>> > issue -- but we could certianly make it configurable in the same way.
>> >
>> > (I see now that IndexWriter.maxFieldLength got deprecated in favor of
>> > IndexWriterConfig.maxFieldLength ... i thought i remembered that had
>> > been deprecated in favor of a TokenFilter that did the limiting, hence
>> > my
>> suggestion
>> > that we use the same pattern for "min term length" -- it could easily
>> > be
>> an
>> > IndexWriterConfig option as well, but using the TokenFilter approach
>> > seems more useful since it can be per field)
>> >
>> >
>> > -Hoss
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
>> > additional commands, e-mail: dev-help@lucene.apache.org
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
>> commands, e-mail: dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
> For additional commands, e-mail: dev-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: inconsistency/performance trap of empty terms
Posted by Uwe Schindler <uw...@thetaphi.de>.
We don't even need the TF, LengthFilter does the job very easy, just set
minTermLength to 1 and maxTermLength to Integer.MAX_VALUE.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Uwe Schindler [mailto:uwe@thetaphi.de]
> Sent: Friday, October 29, 2010 7:49 PM
> To: dev@lucene.apache.org
> Subject: RE: inconsistency/performance trap of empty terms
>
> I am for the tokenfilter approach. Max Field Length is still to be
deprecated in
> favour of the TokenFilter.
>
> TF is very easy, just loop over incrementToken() until it returns false or
a
> termLength>0
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
> > Sent: Friday, October 29, 2010 7:45 PM
> > To: Lucene Dev
> > Subject: Re: inconsistency/performance trap of empty terms
> >
> >
> > : why not just discard them completely in say, indexer/queryparser ?
> >
> > In QueryParser: maybe, that's a high level API with assumptions about
> "human"
> > interaction and text.
> >
> > In the IndexWriter: it seems like a bad idea.
> >
> > Low level Lucene really shouldn't be making any assumptions about
> > *how*
> the
> > client code is using the library -- you and i may not have any good
> reasons for
> > wanting an empty term, but we shouldn't put that as a hardcoded
> > assumption in the low level code.
> >
> > It's essentially the converse issue of IndexWriter.maxFieldLength --
> > which
> was
> > deliberately changed to default to Integer.MAX_VALUE precisesly
> > because of this "don't assume we know how people are using the library"
> > issue -- but we could certianly make it configurable in the same way.
> >
> > (I see now that IndexWriter.maxFieldLength got deprecated in favor of
> > IndexWriterConfig.maxFieldLength ... i thought i remembered that had
> > been deprecated in favor of a TokenFilter that did the limiting, hence
> > my
> suggestion
> > that we use the same pattern for "min term length" -- it could easily
> > be
> an
> > IndexWriterConfig option as well, but using the TokenFilter approach
> > seems more useful since it can be per field)
> >
> >
> > -Hoss
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For
> > additional commands, e-mail: dev-help@lucene.apache.org
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
RE: inconsistency/performance trap of empty terms
Posted by Uwe Schindler <uw...@thetaphi.de>.
I am for the tokenfilter approach. Max Field Length is still to be
deprecated in favour of the TokenFilter.
TF is very easy, just loop over incrementToken() until it returns false or a
termLength>0
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Chris Hostetter [mailto:hossman_lucene@fucit.org]
> Sent: Friday, October 29, 2010 7:45 PM
> To: Lucene Dev
> Subject: Re: inconsistency/performance trap of empty terms
>
>
> : why not just discard them completely in say, indexer/queryparser ?
>
> In QueryParser: maybe, that's a high level API with assumptions about
"human"
> interaction and text.
>
> In the IndexWriter: it seems like a bad idea.
>
> Low level Lucene really shouldn't be making any assumptions about *how*
the
> client code is using the library -- you and i may not have any good
reasons for
> wanting an empty term, but we shouldn't put that as a hardcoded assumption
> in the low level code.
>
> It's essentially the converse issue of IndexWriter.maxFieldLength -- which
was
> deliberately changed to default to Integer.MAX_VALUE precisesly because of
> this "don't assume we know how people are using the library"
> issue -- but we could certianly make it configurable in the same way.
>
> (I see now that IndexWriter.maxFieldLength got deprecated in favor of
> IndexWriterConfig.maxFieldLength ... i thought i remembered that had been
> deprecated in favor of a TokenFilter that did the limiting, hence my
suggestion
> that we use the same pattern for "min term length" -- it could easily be
an
> IndexWriterConfig option as well, but using the TokenFilter approach seems
> more useful since it can be per field)
>
>
> -Hoss
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional
> commands, e-mail: dev-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Chris Hostetter <ho...@fucit.org>.
: why not just discard them completely in say, indexer/queryparser ?
In QueryParser: maybe, that's a high level API with assumptions about
"human" interaction and text.
In the IndexWriter: it seems like a bad idea.
Low level Lucene really shouldn't be making any assumptions about *how*
the client code is using the library -- you and i may not have any good
reasons for wanting an empty term, but we shouldn't put that as a
hardcoded assumption in the low level code.
It's essentially the converse issue of IndexWriter.maxFieldLength --
which was deliberately changed to default to Integer.MAX_VALUE precisesly
because of this "don't assume we know how people are using the library"
issue -- but we could certianly make it configurable in the same way.
(I see now that IndexWriter.maxFieldLength got deprecated in favor of
IndexWriterConfig.maxFieldLength ... i thought i remembered that had been
deprecated in favor of a TokenFilter that did the limiting, hence my
suggestion that we use the same pattern for "min term length" -- it
could easily be an IndexWriterConfig option as well, but using the
TokenFilter approach seems more useful since it can be per field)
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Robert Muir <rc...@gmail.com>.
On Thu, Oct 28, 2010 at 7:59 PM, Chris Hostetter
<ho...@fucit.org> wrote:
>
> : Anyway, I think its possible other users might be in this same
> : situation, with slow performance, and not even realizing it yet...
> : Obviously they can fix this if they go and add LengthFilter, but
> : should we be doing something different?
>
> On one level, ithink a big improvement might just be to start encouraging
> more use of LengthFilter with min=1 at the end of analyzers by including
> it at the end of more "example" field types -- we should probably end
> every analyzer with that and RemoveDuplicatesTokenFilterFactory as a
> general pattern.
why not just discard them completely in say, indexer/queryparser ?
>
> How individual Tokenizers and TokenFilters deal with empty tokens seems
> like something that should be cases by case -- the Ngram classes should
> allow/create them if the "min" value is 0, the pattern based classes
> should create them if the pattern matches and empty string, etc....
why should they create them? is there some use case for the empty term
that you have found (because i can't think of a use case, except
making your search engine slower!)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: inconsistency/performance trap of empty terms
Posted by Chris Hostetter <ho...@fucit.org>.
: Anyway, I think its possible other users might be in this same
: situation, with slow performance, and not even realizing it yet...
: Obviously they can fix this if they go and add LengthFilter, but
: should we be doing something different?
On one level, ithink a big improvement might just be to start encouraging
more use of LengthFilter with min=1 at the end of analyzers by including
it at the end of more "example" field types -- we should probably end
every analyzer with that and RemoveDuplicatesTokenFilterFactory as a
general pattern.
How individual Tokenizers and TokenFilters deal with empty tokens seems
like something that should be cases by case -- the Ngram classes should
allow/create them if the "min" value is 0, the pattern based classes
should create them if the pattern matches and empty string, etc....
If individual classes can be made drasticly more efficient by ignoring
them (or specificly: by not needing to have branches based on explict
checks for an empty term) then we can definitley offer both
implementations and document carefully their valid usage.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org