You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Stefan Groschupf <sg...@media-style.com> on 2003/12/08 20:20:50 UTC
boosting & StandardAnalyzer
Hi,
I notice something really strange.
I just tried the "document to query" thing with term frequencies and
term bosting based on the term frequence.
The code itself take may be 3 minutes, but i spend around 2 hours to
search a nullpointer exception i got in this line.
query = QueryParser.parse(searchStringBuffer.toString(),
IFieldNames.INSTANCE_VALUE, new StandardAnalyser());
I figure out this happen until my search string contains. "will"^13
I replace the StandardAnalyzer with a own Implementation of the Analyzer
that just do nothing and now it works.
Look like the StandardAnalyser has problem until stop word removal, when
term bosting is used.
Is that a bug or just a mistake by myself?
Stefan
--
open technology: www.media-style.com
open source: www.weta-group.net
open discussion: www.text-mining.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: boosting & StandardAnalyzer, stop words
Posted by Stefan Groschupf <sg...@media-style.com>.
>Perhaps we'd better continue this on lucene-dev.
>
>
Ok, i will subscribe this list and request again.
Thanks!
Stefan
--
open technology: http://www.media-style.com
open source: http://www.weta-group.net
open discussion: http://www.text-mining.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: boosting & StandardAnalyzer, stop words
Posted by Ype Kingma <yk...@xs4all.nl>.
On Tuesday 09 December 2003 17:58, Ype Kingma wrote:
> Stefan,
>
> I didn't provide the patch, I just remembered the code from
> a recent reading.
>
> I took another look whether there are more such cases
> in the Term() method, but I couldn't find anything clear
> in the .jj file. The generated .java file didn't help much either.
I meant to say that I couldn't anything that clearly points
to a possible NullPointerException.
The .jj file itself is quite understandable.
Perhaps we'd better continue this on lucene-dev.
Ype
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: boosting & StandardAnalyzer, stop words
Posted by Ype Kingma <yk...@xs4all.nl>.
Stefan,
I didn't provide the patch, I just remembered the code from
a recent reading.
I took another look whether there are more such cases
in the Term() method, but I couldn't find anything clear
in the .jj file. The generated .java file didn't help much either.
Could you provide a line number with the exception?
There are some attribute references that might cause
this.
Or perhaps someone else can give a better hint.
Kind regards,
Ype
On Tuesday 09 December 2003 12:38, Stefan Groschupf wrote:
> Ype,
>
> >It's a bug, and there is a fix for this in the latest CVS
> >near the end of the QueryParser.jj file:
> >
> > // avoid boosting null queries, such as those caused by stop words
> > if (q != null) {
> > q.setBoost(f);
> > }
>
> I had checked out the latest sources from public cvs. The posted code
> lines abouve are on the QueryParser.jj but this does not fix the problem.
> ;/ What could it be?
>
> *java.lang.NullPointerException
> at org.apache.lucene.queryParser.QueryParser.Term(Unknown Source)
> at org.apache.lucene.queryParser.QueryParser.Clause(Unknown Source)
> at org.apache.lucene.queryParser.QueryParser.Query(Unknown Source)
> at org.apache.lucene.queryParser.QueryParser.Clause(Unknown Source)
> at org.apache.lucene.queryParser.QueryParser.Query(Unknown Source)
> at org.apache.lucene.queryParser.QueryParser.parse(Unknown Source)
> at org.apache.lucene.queryParser.QueryParser.parse(Unknown Source)
>
> *
> Stefan
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: boosting & StandardAnalyzer, stop words
Posted by Stefan Groschupf <sg...@media-style.com>.
Ype,
>It's a bug, and there is a fix for this in the latest CVS
>near the end of the QueryParser.jj file:
>
> // avoid boosting null queries, such as those caused by stop words
> if (q != null) {
> q.setBoost(f);
> }
>
>
I had checked out the latest sources from public cvs. The posted code
lines abouve are on the QueryParser.jj but this does not fix the problem. ;/
What could it be?
*java.lang.NullPointerException
at org.apache.lucene.queryParser.QueryParser.Term(Unknown Source)
at org.apache.lucene.queryParser.QueryParser.Clause(Unknown Source)
at org.apache.lucene.queryParser.QueryParser.Query(Unknown Source)
at org.apache.lucene.queryParser.QueryParser.Clause(Unknown Source)
at org.apache.lucene.queryParser.QueryParser.Query(Unknown Source)
at org.apache.lucene.queryParser.QueryParser.parse(Unknown Source)
at org.apache.lucene.queryParser.QueryParser.parse(Unknown Source)
*
Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: boosting & StandardAnalyzer, stop words
Posted by Ype Kingma <yk...@xs4all.nl>.
Stefan,
It's a bug, and there is a fix for this in the latest CVS
near the end of the QueryParser.jj file:
// avoid boosting null queries, such as those caused by stop words
if (q != null) {
q.setBoost(f);
}
Kind regards,
Ype
On Monday 08 December 2003 20:20, Stefan Groschupf wrote:
> Hi,
>
> I notice something really strange.
>
> I just tried the "document to query" thing with term frequencies and
> term bosting based on the term frequence.
> The code itself take may be 3 minutes, but i spend around 2 hours to
> search a nullpointer exception i got in this line.
>
> query = QueryParser.parse(searchStringBuffer.toString(),
> IFieldNames.INSTANCE_VALUE, new StandardAnalyser());
>
> I figure out this happen until my search string contains. "will"^13
> I replace the StandardAnalyzer with a own Implementation of the Analyzer
> that just do nothing and now it works.
> Look like the StandardAnalyser has problem until stop word removal, when
> term bosting is used.
>
> Is that a bug or just a mistake by myself?
>
> Stefan
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org