You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Edwards, Joshua" <Jo...@capitalone.com> on 2014/08/18 21:17:00 UTC
Inconsistent Results Issue with Filter
So, as I continue to try to make things work in different ways with numbers, I have run into a very strange situation. When I perform a search (for example my_num:"thirty seven"), it returns the correct results. However, if I run this exact same search again, it returns no results, and continues giving no results. If I restart my instance, it gives results again (once). Any ideas on what might be happening? I've been using the Lucene Solr admin page, and I see that the tokens are there how I expect them to be - yet things aren't matching. Or, more specifically, seem to stop matching after the first search. I'm pretty sure that my Filter is somehow the culprit, I just don't know why a filter would cause the results to change from one query to the next, since the filter seems to be returning the same tokens each time (according to the analysis page), and also works in all of my unit tests.
Thanks,
Josh Edwards
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
RE: Inconsistent Results Issue with Filter
Posted by Chris Hostetter <ho...@fucit.org>.
: It is a custom filter. Essentially, what I'm trying to do is combine
: words that are associated with each other into a single token. Here is
: the incrementToken() method, where most of the work is being performed:
so clearly this is a custom TokenFilter ... but how exactly are you using
this filter? do you register it for a fieldType using the normal
schema.xml mechanisms -- or do you have it as part of some sort of custom
Query parser?
all of the details of your custom code mater....
: > numbers, I have run into a very strange situation. When I perform a >
: search (for example my_num:”thirty seven”), it returns the correct >
: results. However, if I run this exact same search again, it returns >
: no results, and continues giving no results. If I restart my instance,
: it gives results again (once).
...at first blush, that sounds like it could be a caching problem, which
could be explained if you have other custom code dealing with parsing the
query (to use your custom TokenFilter) or with modifying the request ...
you may have done this in a way that isn't being taken into account with
the caching layer.
tell us *everything * baout your custom code, and your configs, and your
schema, and what *exactly* do your request URLs look like, and what log
messages you see when you hit those urls, etc...
https://wiki.apache.org/solr/UsingMailingLists
details matter.
-Hoss
http://www.lucidworks.com/
RE: Inconsistent Results Issue with Filter
Posted by "Edwards, Joshua" <Jo...@capitalone.com>.
In case anyone was checking this out for me, I have this matter resolved. It appears that the culprit was that I wasn't resetting my Boolean flag in preparation of processing the next TokenStream.
Thanks!
Josh Edwards
-----Original Message-----
From: Edwards, Joshua [mailto:Joshua.Edwards@capitalone.com]
Sent: Tuesday, August 19, 2014 9:08 AM
To: dev@lucene.apache.org
Subject: RE: Inconsistent Results Issue with Filter
It is a custom filter. Essentially, what I'm trying to do is combine words that are associated with each other into a single token. Here is the incrementToken() method, where most of the work is being performed:
private String consumedToken;
private boolean alreadyReachedEOF = false;
public final boolean incrementToken() throws IOException {
//If there is a token sitting in the buffer, then return it without consuming the next token from the buffer.
if (consumedToken != null) {
input.clearAttributes();
termAtt.setEmpty().append(consumedToken);
consumedToken = null;
return true;
}
StringBuffer str = new StringBuffer();
while (true) {
State prePop = this.captureState();
boolean reachedEnd = alreadyReachedEOF || !input.incrementToken();
if (reachedEnd && str.length() == 0)
return false;
else if (reachedEnd) {
termAtt.setEmpty();
termAtt.append(str.toString().trim());
alreadyReachedEOF = true;
return true;
}
consumedToken = termAtt.toString(); //tokenMatchesCombineList does a String comparison, and if it is any of the items in the list it returns true, otherwise false.
if (tokenMatchesCombineList(consumedToken)) {
//Normalizeby removing dashes.
consumedToken = consumedToken.replaceAll("-", " ");
str.append(consumedToken + " ");
termAtt.setEmpty();
termAtt.append(str.toString().trim());
consumedToken = null;
} else if (str.length() == 0){
//Nothing in buffer, and a word that shouldn't be combined. Leave it alone.
consumedToken = null;
return true;
} else {
//A word that shouldn't be combined after some that should be. Put it back on the stack and return.
this.restoreState(prePop);
return true;
}
Thanks!
Josh Edwards
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Monday, August 18, 2014 5:47 PM
To: dev@lucene.apache.org
Subject: Re: Inconsistent Results Issue with Filter
Are you saying you have a custom filter in here? Or this is just OOB code? If the former, we'd need to see the salient parts of the code.
If the latter, some example docs would be helpful.
Best,
Erick
On Mon, Aug 18, 2014 at 12:17 PM, Edwards, Joshua <Jo...@capitalone.com> wrote:
> So, as I continue to try to make things work in different ways with
> numbers, I have run into a very strange situation. When I perform a
> search (for example my_num:”thirty seven”), it returns the correct
> results. However, if I run this exact same search again, it returns
> no results, and continues giving no results. If I restart my instance, it gives results again (once).
> Any ideas on what might be happening? I’ve been using the Lucene Solr
> admin page, and I see that the tokens are there how I expect them to
> be – yet things aren’t matching. Or, more specifically, seem to stop
> matching after the first search. I’m pretty sure that my Filter is
> somehow the culprit, I just don’t know why a filter would cause the
> results to change from one query to the next, since the filter seems
> to be returning the same tokens each time (according to the analysis
> page), and also works in all of my unit tests.
>
>
>
> Thanks,
>
> Josh Edwards
>
>
> ________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates. The information
> transmitted herewith is intended only for use by the individual or
> entity to which it is addressed. If the reader of this message is not
> the intended recipient, you are hereby notified that any review,
> retransmission, dissemination, distribution, copying or other use of,
> or taking of any action in reliance upon this information is strictly
> prohibited. If you have received this communication in error, please
> contact the sender and delete the material from your computer.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
RE: Inconsistent Results Issue with Filter
Posted by "Edwards, Joshua" <Jo...@capitalone.com>.
It is a custom filter. Essentially, what I'm trying to do is combine words that are associated with each other into a single token. Here is the incrementToken() method, where most of the work is being performed:
private String consumedToken;
private boolean alreadyReachedEOF = false;
public final boolean incrementToken() throws IOException {
//If there is a token sitting in the buffer, then return it without consuming the next token from the buffer.
if (consumedToken != null) {
input.clearAttributes();
termAtt.setEmpty().append(consumedToken);
consumedToken = null;
return true;
}
StringBuffer str = new StringBuffer();
while (true) {
State prePop = this.captureState();
boolean reachedEnd = alreadyReachedEOF || !input.incrementToken();
if (reachedEnd && str.length() == 0)
return false;
else if (reachedEnd) {
termAtt.setEmpty();
termAtt.append(str.toString().trim());
alreadyReachedEOF = true;
return true;
}
consumedToken = termAtt.toString();
//tokenMatchesCombineList does a String comparison, and if it is any of the items in the list it returns true, otherwise false.
if (tokenMatchesCombineList(consumedToken)) {
//Normalizeby removing dashes.
consumedToken = consumedToken.replaceAll("-", " ");
str.append(consumedToken + " ");
termAtt.setEmpty();
termAtt.append(str.toString().trim());
consumedToken = null;
} else if (str.length() == 0){
//Nothing in buffer, and a word that shouldn't be combined. Leave it alone.
consumedToken = null;
return true;
} else {
//A word that shouldn't be combined after some that should be. Put it back on the stack and return.
this.restoreState(prePop);
return true;
}
Thanks!
Josh Edwards
-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]
Sent: Monday, August 18, 2014 5:47 PM
To: dev@lucene.apache.org
Subject: Re: Inconsistent Results Issue with Filter
Are you saying you have a custom filter in here? Or this is just OOB code? If the former, we'd need to see the salient parts of the code.
If the latter, some example docs would be helpful.
Best,
Erick
On Mon, Aug 18, 2014 at 12:17 PM, Edwards, Joshua <Jo...@capitalone.com> wrote:
> So, as I continue to try to make things work in different ways with
> numbers, I have run into a very strange situation. When I perform a
> search (for example my_num:”thirty seven”), it returns the correct
> results. However, if I run this exact same search again, it returns
> no results, and continues giving no results. If I restart my instance, it gives results again (once).
> Any ideas on what might be happening? I’ve been using the Lucene Solr
> admin page, and I see that the tokens are there how I expect them to
> be – yet things aren’t matching. Or, more specifically, seem to stop
> matching after the first search. I’m pretty sure that my Filter is
> somehow the culprit, I just don’t know why a filter would cause the
> results to change from one query to the next, since the filter seems
> to be returning the same tokens each time (according to the analysis
> page), and also works in all of my unit tests.
>
>
>
> Thanks,
>
> Josh Edwards
>
>
> ________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates. The information
> transmitted herewith is intended only for use by the individual or
> entity to which it is addressed. If the reader of this message is not
> the intended recipient, you are hereby notified that any review,
> retransmission, dissemination, distribution, copying or other use of,
> or taking of any action in reliance upon this information is strictly
> prohibited. If you have received this communication in error, please
> contact the sender and delete the material from your computer.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org
________________________________________________________
The information contained in this e-mail is confidential and/or proprietary to Capital One and/or its affiliates. The information transmitted herewith is intended only for use by the individual or entity to which it is addressed. If the reader of this message is not the intended recipient, you are hereby notified that any review, retransmission, dissemination, distribution, copying or other use of, or taking of any action in reliance upon this information is strictly prohibited. If you have received this communication in error, please contact the sender and delete the material from your computer.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: Inconsistent Results Issue with Filter
Posted by Erick Erickson <er...@gmail.com>.
Are you saying you have a custom filter in here? Or this is just OOB
code? If the former,
we'd need to see the salient parts of the code.
If the latter, some example docs would be helpful.
Best,
Erick
On Mon, Aug 18, 2014 at 12:17 PM, Edwards, Joshua
<Jo...@capitalone.com> wrote:
> So, as I continue to try to make things work in different ways with numbers,
> I have run into a very strange situation. When I perform a search (for
> example my_num:”thirty seven”), it returns the correct results. However, if
> I run this exact same search again, it returns no results, and continues
> giving no results. If I restart my instance, it gives results again (once).
> Any ideas on what might be happening? I’ve been using the Lucene Solr admin
> page, and I see that the tokens are there how I expect them to be – yet
> things aren’t matching. Or, more specifically, seem to stop matching after
> the first search. I’m pretty sure that my Filter is somehow the culprit, I
> just don’t know why a filter would cause the results to change from one
> query to the next, since the filter seems to be returning the same tokens
> each time (according to the analysis page), and also works in all of my unit
> tests.
>
>
>
> Thanks,
>
> Josh Edwards
>
>
> ________________________________
>
> The information contained in this e-mail is confidential and/or proprietary
> to Capital One and/or its affiliates. The information transmitted herewith
> is intended only for use by the individual or entity to which it is
> addressed. If the reader of this message is not the intended recipient, you
> are hereby notified that any review, retransmission, dissemination,
> distribution, copying or other use of, or taking of any action in reliance
> upon this information is strictly prohibited. If you have received this
> communication in error, please contact the sender and delete the material
> from your computer.
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org