You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jamie <ja...@mailarchiva.com> on 2014/03/28 12:00:20 UTC
Lucene 4.7 intermittently not applying query filter
Greetings
We have a problem whereby Lucene 4.7 occasionally does not apply a
filter query during searching. The problem is intermittent. One in
thirty or so searches will return what appears to be an unfiltered
result set. There are no exceptions or errors occurring.. just incorrect
results. We are using realtime search with multiple index readers. Our
software had been working fine with earlier versions of Lucene. I've
double checked the query submitted to lucene and it appears to be
correct. The query looks as follows:
2014-03-28 21:16:38 t.c.s.a.s.StandardSearch [DEBUG] start search
{searchquery='',query='*:*',filterQuery='QueryWrapperFilter(+archivedate:[201002280000
TO 201403282115] +cat:email +(to:"john.douglas@mycompany.com.au
john.douglas mycompany.com.au john douglas mycompany com au com.au"
to:"john.doe@mycompany.com.au john.doe mycompany.com.au john doe
mycompany com au com.au" from:"john.douglas@mycompany.com.au
john.douglas mycompany.com.au john douglas mycompany com au com.au"
from:"john.doe@mycompany.com.au john.doe mycompany.com.au john doe
mycompany com au com.au" cc:"john.douglas@mycompany.com.au john.douglas
mycompany.com.au john douglas mycompany com au com.au"
cc:"john.doe@mycompany.com.au john.doe mycompany.com.au john doe
mycompany com au com.au"))',sort='<long: "mydate">!'}
The string "john.doe@mycompany.com.au john.doe mycompany.com.au john doe
mycompany com au com.au" is the required expansion for the
UAX29URLEmailTokenizer. By using quotes, I am aiming for an exact match.
This works most of the time, but not all of the time (as it should).
I came across: https://issues.apache.org/jira/browse/LUCENE-5502 and
applied it, but it makes no difference. I tried to downgrade Lucene, but
it wont read the 4.6 indexes. Can anyone suggest a way forward?
Thanks for your recommendations
Jamie
-------------------------
public final class EmailAnalyzer extends StopwordAnalyzerBase {
public static final int DEFAULT_MAX_TOKEN_LENGTH =
StandardAnalyzer.DEFAULT_MAX_TOKEN_LENGTH;
private int maxTokenLength = DEFAULT_MAX_TOKEN_LENGTH;
public static final CharArraySet STOP_WORDS_SET =
StopAnalyzer.ENGLISH_STOP_WORDS_SET;
public EmailAnalyzer(Version matchVersion, CharArraySet stopWords) {
super(matchVersion, stopWords);
}
public EmailAnalyzer(Version matchVersion) {
this(matchVersion, STOP_WORDS_SET);
}
public EmailAnalyzer(Version matchVersion, Reader stopwords) throws
IOException {
this(matchVersion, loadStopwordSet(stopwords, matchVersion));
}
public void setMaxTokenLength(int length) {
maxTokenLength = length;
}
public int getMaxTokenLength() {
return maxTokenLength;
}
protected TokenStreamComponents createComponents(final String
fieldName, final Reader reader) {
final UAX29URLEmailTokenizer src = new
UAX29URLEmailTokenizer(matchVersion, reader);
src.setMaxTokenLength(maxTokenLength);
TokenStream tok = new EmailFilter(src);
tok = new LowerCaseFilter(matchVersion, tok);
return new TokenStreamComponents(src, tok) {
protected void setReader(final Reader reader) throws IOException {
src.setMaxTokenLength(EmailAnalyzer.this.maxTokenLength);
super.setReader(reader);
}
};
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Lucene 4.7 intermittently not applying query filter
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Jamie,
is your Query Filter also implemented by your team? If this is the case, maybe you are not correctly implementing the random access getDocIdSet(), bits(), or you don't correctly handle acceptDocs parameter in your own DocIdSet / Filter implementation, leading to random failures.
Uwe
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Jamie [mailto:jamie@mailarchiva.com]
> Sent: Friday, March 28, 2014 4:41 PM
> To: java-user@lucene.apache.org
> Subject: Re: Lucene 4.7 intermittently not applying query filter
>
> I beg your pardon. Its our EmailFilter class that emits the tokens. We do it this
> way, since users like to search using individual components of an email
> address. e.g. joe or mycompany.com.au. I think we may have a
> synchronization issue at play. I will perform some further testing and will get
> back to you. Thank you.
>
> On 2014/03/28, 5:28 PM, Steve Rowe wrote:
> > Jamie,
> >
> > UAX29URLEmailTokenizer does not emit email components as tokens;
> “john.doe@mycompany.com.au” will be tokenized as
> “john.doe@mycompany.com.au”, nothing more. That’s why I asked what
> EmailFilter does.
> >
> > If the filter really is ignored by Lucene, that would be a bug in Lucene. I
> think something else is likely going on, though, which is why I asked you for
> an example query matching too many docs and a doc it improperly matches.
> >
> > Steve
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 4.7 intermittently not applying query filter
Posted by Jamie <ja...@mailarchiva.com>.
I beg your pardon. Its our EmailFilter class that emits the tokens. We
do it this way, since users like to search using individual components
of an email address. e.g. joe or mycompany.com.au. I think we may have a
synchronization issue at play. I will perform some further testing and
will get back to you. Thank you.
On 2014/03/28, 5:28 PM, Steve Rowe wrote:
> Jamie,
>
> UAX29URLEmailTokenizer does not emit email components as tokens; “john.doe@mycompany.com.au” will be tokenized as “john.doe@mycompany.com.au”, nothing more. That’s why I asked what EmailFilter does.
>
> If the filter really is ignored by Lucene, that would be a bug in Lucene. I think something else is likely going on, though, which is why I asked you for an example query matching too many docs and a doc it improperly matches.
>
> Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 4.7 intermittently not applying query filter
Posted by Steve Rowe <sa...@gmail.com>.
Jamie,
UAX29URLEmailTokenizer does not emit email components as tokens; “john.doe@mycompany.com.au” will be tokenized as “john.doe@mycompany.com.au”, nothing more. That’s why I asked what EmailFilter does.
If the filter really is ignored by Lucene, that would be a bug in Lucene. I think something else is likely going on, though, which is why I asked you for an example query matching too many docs and a doc it improperly matches.
Steve
On Mar 28, 2014, at 10:54 AM, Jamie <ja...@mailarchiva.com> wrote:
> Steve
>
> Thank for the contact. I believe UAX29URLEmailTokenizer tokenizes email addresses as follows: john.doe@mycompany.com.au john.doe mycompany.com.au john doe mycompany com au com.au.We have an overridden query parser that swaps out anyaddress: with to, from, cc, bcc, etc. Inside the overridden query parser, we call getFieldQuery() to build the clauses...
>
> Query q = super.getFieldQuery(field, emailAddress, true);
> if (slop!=-1) {
> applySlop(q,slop);
> }
> clauses.add(new BooleanClause(q, BooleanClause.Occur.SHOULD));
>
> The query is outputted below. Sometimes when it is executed by Lucene, the filter is ignored.
>
> I am busy trying to isolate the issue, since the code is running in a wider system among other complexities.
>
> Jamie
>
> On 2014/03/28, 4:08 PM, Steve Rowe wrote:
>> Hi Jamie,
>>
>> What does EmailFilter do?
>>
>> Why is the expanded form "required for the UAX29URLEmailTokenizer"? Seems like an exact match would work on the email address alone, without the expanded components?
>>
>> Do you have an example of a query that reproducibly matches more documents than it should, and a document that matched but shouldn’t have?
>>
>> Steve
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 4.7 intermittently not applying query filter
Posted by Jamie <ja...@mailarchiva.com>.
Steve
Thank for the contact. I believe UAX29URLEmailTokenizer tokenizes email
addresses as follows: john.doe@mycompany.com.au john.doe
mycompany.com.au john doe mycompany com au com.au.We have an overridden
query parser that swaps out anyaddress: with to, from, cc, bcc, etc.
Inside the overridden query parser, we call getFieldQuery() to build the
clauses...
Query q = super.getFieldQuery(field, emailAddress, true);
if (slop!=-1) {
applySlop(q,slop);
}
clauses.add(new BooleanClause(q, BooleanClause.Occur.SHOULD));
The query is outputted below. Sometimes when it is executed by Lucene,
the filter is ignored.
I am busy trying to isolate the issue, since the code is running in a
wider system among other complexities.
Jamie
On 2014/03/28, 4:08 PM, Steve Rowe wrote:
> Hi Jamie,
>
> What does EmailFilter do?
>
> Why is the expanded form "required for the UAX29URLEmailTokenizer"? Seems like an exact match would work on the email address alone, without the expanded components?
>
> Do you have an example of a query that reproducibly matches more documents than it should, and a document that matched but shouldn’t have?
>
> Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Lucene 4.7 intermittently not applying query filter
Posted by Steve Rowe <sa...@gmail.com>.
Hi Jamie,
What does EmailFilter do?
Why is the expanded form "required for the UAX29URLEmailTokenizer"? Seems like an exact match would work on the email address alone, without the expanded components?
Do you have an example of a query that reproducibly matches more documents than it should, and a document that matched but shouldn’t have?
Steve
On Mar 28, 2014, at 7:00 AM, Jamie <ja...@mailarchiva.com> wrote:
> Greetings
>
> We have a problem whereby Lucene 4.7 occasionally does not apply a filter query during searching. The problem is intermittent. One in thirty or so searches will return what appears to be an unfiltered result set. There are no exceptions or errors occurring.. just incorrect results. We are using realtime search with multiple index readers. Our software had been working fine with earlier versions of Lucene. I've double checked the query submitted to lucene and it appears to be correct. The query looks as follows:
>
> 2014-03-28 21:16:38 t.c.s.a.s.StandardSearch [DEBUG] start search {searchquery='',query='*:*',filterQuery='QueryWrapperFilter(+archivedate:[201002280000 TO 201403282115] +cat:email +(to:"john.douglas@mycompany.com.au john.douglas mycompany.com.au john douglas mycompany com au com.au" to:"john.doe@mycompany.com.au john.doe mycompany.com.au john doe mycompany com au com.au" from:"john.douglas@mycompany.com.au john.douglas mycompany.com.au john douglas mycompany com au com.au" from:"john.doe@mycompany.com.au john.doe mycompany.com.au john doe mycompany com au com.au" cc:"john.douglas@mycompany.com.au john.douglas mycompany.com.au john douglas mycompany com au com.au" cc:"john.doe@mycompany.com.au john.doe mycompany.com.au john doe mycompany com au com.au"))',sort='<long: "mydate">!'}
>
> The string "john.doe@mycompany.com.au john.doe mycompany.com.au john doe mycompany com au com.au" is the required expansion for the UAX29URLEmailTokenizer. By using quotes, I am aiming for an exact match. This works most of the time, but not all of the time (as it should).
>
> I came across: https://issues.apache.org/jira/browse/LUCENE-5502 and applied it, but it makes no difference. I tried to downgrade Lucene, but it wont read the 4.6 indexes. Can anyone suggest a way forward?
>
> Thanks for your recommendations
>
> Jamie
>
> -------------------------
>
> public final class EmailAnalyzer extends StopwordAnalyzerBase {
>
> public static final int DEFAULT_MAX_TOKEN_LENGTH = StandardAnalyzer.DEFAULT_MAX_TOKEN_LENGTH;
> private int maxTokenLength = DEFAULT_MAX_TOKEN_LENGTH;
> public static final CharArraySet STOP_WORDS_SET = StopAnalyzer.ENGLISH_STOP_WORDS_SET;
>
> public EmailAnalyzer(Version matchVersion, CharArraySet stopWords) {
> super(matchVersion, stopWords);
> }
>
> public EmailAnalyzer(Version matchVersion) {
> this(matchVersion, STOP_WORDS_SET);
> }
>
> public EmailAnalyzer(Version matchVersion, Reader stopwords) throws IOException {
> this(matchVersion, loadStopwordSet(stopwords, matchVersion));
> }
>
> public void setMaxTokenLength(int length) {
> maxTokenLength = length;
> }
>
> public int getMaxTokenLength() {
> return maxTokenLength;
> }
>
> protected TokenStreamComponents createComponents(final String fieldName, final Reader reader) {
> final UAX29URLEmailTokenizer src = new UAX29URLEmailTokenizer(matchVersion, reader);
> src.setMaxTokenLength(maxTokenLength);
> TokenStream tok = new EmailFilter(src);
> tok = new LowerCaseFilter(matchVersion, tok);
> return new TokenStreamComponents(src, tok) {
> protected void setReader(final Reader reader) throws IOException {
> src.setMaxTokenLength(EmailAnalyzer.this.maxTokenLength);
> super.setReader(reader);
> }
> };
> }
> }
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org