You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Murdoch, Paul" <PA...@saic.com> on 2010/02/24 21:51:24 UTC
Phrase Search and NOT_ANALYZED
Hi,
I'm indexing a field using the StandardAnalyzer 2.9.
field = new Field(fieldName, fieldValue, Field.Store.YES,
Field.Index.NOT_ANALYZED);
Let's say fieldName is "name" and fieldValue is "something in the
index". When I perform the query...
name:"something in the index"
... I don't get a hit. I'm using the StandardAnalyzer for searching.
I've tested this with Luke and it doesn't work there either. What am I
missing?
Thanks,
Paul
RE: Phrase Search and NOT_ANALYZED
Posted by "Murdoch, Paul" <PA...@saic.com>.
I would still be interested in knowing why the combination of the StandardAnalyzer, a phrase built using double quotes with no stop words, and the QueryParser doesn't return hits while building the same query with the StadardAnalyzer and a PhraseQuery does?
Thanks,
Paul
-----Original Message-----
From: java-user-return-45156-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45156-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Murdoch, Paul
Sent: Wednesday, February 24, 2010 5:11 PM
To: java-user@lucene.apache.org
Subject: RE: Phrase Search and NOT_ANALYZED
PhraseQuery appears to be working. Thanks to all.
Paul
-----Original Message-----
From: java-user-return-45155-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45155-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Murdoch, Paul
Sent: Wednesday, February 24, 2010 5:01 PM
To: java-user@lucene.apache.org
Subject: RE: Phrase Search and NOT_ANALYZED
Thanks,
I've been looking at that one too. I'm trying to make it happen with the StandardAnalyzer. Unfortunately, I think I see some redesign for more robustness in the future.
Cheers,
Paul
-----Original Message-----
From: java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Robert Muir
Sent: Wednesday, February 24, 2010 4:55 PM
To: java-user@lucene.apache.org
Subject: Re: Phrase Search and NOT_ANALYZED
check out KeywordAnalyzer!
On Wed, Feb 24, 2010 at 4:51 PM, Murdoch, Paul <PA...@saic.com>wrote:
> It still happens if there are no stop words in the fieldValue. For
> instance if fieldValue was "paul murdoch", Luke would show the query as
> name:"paul murdoch" but no hits are returned. If I change to
> Field.Index.ANALYZED it works. The problem with ANALYZED is that there
> is a possibility of picking up false positive hits. With queries
> like...
>
> name:murdoch // assume "murdoch" is another separate field value
>
> Obviously here I just want "murdoch" but since "paul murdoch" was
> indexed ANALYZED that will be returned as well. Using quotes with
> Field.Index.NOT_ANALYZED isn't working for me. I'm thinking maybe it's
> the QueryParser so I'm going trying a PhraseQuery instead.
>
> Thanks,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> [mailto:java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> ] On Behalf Of Erick Erickson
> Sent: Wednesday, February 24, 2010 4:23 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase Search and NOT_ANALYZED
>
> What does Luke's explain show you? That'll show you a lot about how
> the query gets transformed......
>
> My first guess is that stop words are messing you up....
>
> Erick
>
> On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul
> <PA...@saic.com>wrote:
>
> > Hi,
> >
> >
> >
> > I'm indexing a field using the StandardAnalyzer 2.9.
> >
> >
> >
> > field = new Field(fieldName, fieldValue, Field.Store.YES,
> > Field.Index.NOT_ANALYZED);
> >
> >
> >
> > Let's say fieldName is "name" and fieldValue is "something in the
> > index". When I perform the query...
> >
> >
> >
> > name:"something in the index"
> >
> >
> >
> > ... I don't get a hit. I'm using the StandardAnalyzer for searching.
> > I've tested this with Luke and it doesn't work there either. What am
> I
> > missing?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Paul
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Robert Muir
rcmuir@gmail.com
RE: Phrase Search and NOT_ANALYZED
Posted by "Murdoch, Paul" <PA...@saic.com>.
PhraseQuery appears to be working. Thanks to all.
Paul
-----Original Message-----
From: java-user-return-45155-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45155-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Murdoch, Paul
Sent: Wednesday, February 24, 2010 5:01 PM
To: java-user@lucene.apache.org
Subject: RE: Phrase Search and NOT_ANALYZED
Thanks,
I've been looking at that one too. I'm trying to make it happen with the StandardAnalyzer. Unfortunately, I think I see some redesign for more robustness in the future.
Cheers,
Paul
-----Original Message-----
From: java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Robert Muir
Sent: Wednesday, February 24, 2010 4:55 PM
To: java-user@lucene.apache.org
Subject: Re: Phrase Search and NOT_ANALYZED
check out KeywordAnalyzer!
On Wed, Feb 24, 2010 at 4:51 PM, Murdoch, Paul <PA...@saic.com>wrote:
> It still happens if there are no stop words in the fieldValue. For
> instance if fieldValue was "paul murdoch", Luke would show the query as
> name:"paul murdoch" but no hits are returned. If I change to
> Field.Index.ANALYZED it works. The problem with ANALYZED is that there
> is a possibility of picking up false positive hits. With queries
> like...
>
> name:murdoch // assume "murdoch" is another separate field value
>
> Obviously here I just want "murdoch" but since "paul murdoch" was
> indexed ANALYZED that will be returned as well. Using quotes with
> Field.Index.NOT_ANALYZED isn't working for me. I'm thinking maybe it's
> the QueryParser so I'm going trying a PhraseQuery instead.
>
> Thanks,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> [mailto:java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> ] On Behalf Of Erick Erickson
> Sent: Wednesday, February 24, 2010 4:23 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase Search and NOT_ANALYZED
>
> What does Luke's explain show you? That'll show you a lot about how
> the query gets transformed......
>
> My first guess is that stop words are messing you up....
>
> Erick
>
> On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul
> <PA...@saic.com>wrote:
>
> > Hi,
> >
> >
> >
> > I'm indexing a field using the StandardAnalyzer 2.9.
> >
> >
> >
> > field = new Field(fieldName, fieldValue, Field.Store.YES,
> > Field.Index.NOT_ANALYZED);
> >
> >
> >
> > Let's say fieldName is "name" and fieldValue is "something in the
> > index". When I perform the query...
> >
> >
> >
> > name:"something in the index"
> >
> >
> >
> > ... I don't get a hit. I'm using the StandardAnalyzer for searching.
> > I've tested this with Luke and it doesn't work there either. What am
> I
> > missing?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Paul
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Robert Muir
rcmuir@gmail.com
RE: Phrase Search and NOT_ANALYZED
Posted by "Murdoch, Paul" <PA...@saic.com>.
Thanks,
I've been looking at that one too. I'm trying to make it happen with the StandardAnalyzer. Unfortunately, I think I see some redesign for more robustness in the future.
Cheers,
Paul
-----Original Message-----
From: java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Robert Muir
Sent: Wednesday, February 24, 2010 4:55 PM
To: java-user@lucene.apache.org
Subject: Re: Phrase Search and NOT_ANALYZED
check out KeywordAnalyzer!
On Wed, Feb 24, 2010 at 4:51 PM, Murdoch, Paul <PA...@saic.com>wrote:
> It still happens if there are no stop words in the fieldValue. For
> instance if fieldValue was "paul murdoch", Luke would show the query as
> name:"paul murdoch" but no hits are returned. If I change to
> Field.Index.ANALYZED it works. The problem with ANALYZED is that there
> is a possibility of picking up false positive hits. With queries
> like...
>
> name:murdoch // assume "murdoch" is another separate field value
>
> Obviously here I just want "murdoch" but since "paul murdoch" was
> indexed ANALYZED that will be returned as well. Using quotes with
> Field.Index.NOT_ANALYZED isn't working for me. I'm thinking maybe it's
> the QueryParser so I'm going trying a PhraseQuery instead.
>
> Thanks,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> [mailto:java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> ] On Behalf Of Erick Erickson
> Sent: Wednesday, February 24, 2010 4:23 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase Search and NOT_ANALYZED
>
> What does Luke's explain show you? That'll show you a lot about how
> the query gets transformed......
>
> My first guess is that stop words are messing you up....
>
> Erick
>
> On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul
> <PA...@saic.com>wrote:
>
> > Hi,
> >
> >
> >
> > I'm indexing a field using the StandardAnalyzer 2.9.
> >
> >
> >
> > field = new Field(fieldName, fieldValue, Field.Store.YES,
> > Field.Index.NOT_ANALYZED);
> >
> >
> >
> > Let's say fieldName is "name" and fieldValue is "something in the
> > index". When I perform the query...
> >
> >
> >
> > name:"something in the index"
> >
> >
> >
> > ... I don't get a hit. I'm using the StandardAnalyzer for searching.
> > I've tested this with Luke and it doesn't work there either. What am
> I
> > missing?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Paul
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Robert Muir
rcmuir@gmail.com
Re: Phrase Search and NOT_ANALYZED
Posted by Robert Muir <rc...@gmail.com>.
check out KeywordAnalyzer!
On Wed, Feb 24, 2010 at 4:51 PM, Murdoch, Paul <PA...@saic.com>wrote:
> It still happens if there are no stop words in the fieldValue. For
> instance if fieldValue was "paul murdoch", Luke would show the query as
> name:"paul murdoch" but no hits are returned. If I change to
> Field.Index.ANALYZED it works. The problem with ANALYZED is that there
> is a possibility of picking up false positive hits. With queries
> like...
>
> name:murdoch // assume "murdoch" is another separate field value
>
> Obviously here I just want "murdoch" but since "paul murdoch" was
> indexed ANALYZED that will be returned as well. Using quotes with
> Field.Index.NOT_ANALYZED isn't working for me. I'm thinking maybe it's
> the QueryParser so I'm going trying a PhraseQuery instead.
>
> Thanks,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> [mailto:java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> ] On Behalf Of Erick Erickson
> Sent: Wednesday, February 24, 2010 4:23 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase Search and NOT_ANALYZED
>
> What does Luke's explain show you? That'll show you a lot about how
> the query gets transformed......
>
> My first guess is that stop words are messing you up....
>
> Erick
>
> On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul
> <PA...@saic.com>wrote:
>
> > Hi,
> >
> >
> >
> > I'm indexing a field using the StandardAnalyzer 2.9.
> >
> >
> >
> > field = new Field(fieldName, fieldValue, Field.Store.YES,
> > Field.Index.NOT_ANALYZED);
> >
> >
> >
> > Let's say fieldName is "name" and fieldValue is "something in the
> > index". When I perform the query...
> >
> >
> >
> > name:"something in the index"
> >
> >
> >
> > ... I don't get a hit. I'm using the StandardAnalyzer for searching.
> > I've tested this with Luke and it doesn't work there either. What am
> I
> > missing?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Paul
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Robert Muir
rcmuir@gmail.com
RE: Phrase Search and NOT_ANALYZED
Posted by "Murdoch, Paul" <PA...@saic.com>.
It still happens if there are no stop words in the fieldValue. For
instance if fieldValue was "paul murdoch", Luke would show the query as
name:"paul murdoch" but no hits are returned. If I change to
Field.Index.ANALYZED it works. The problem with ANALYZED is that there
is a possibility of picking up false positive hits. With queries
like...
name:murdoch // assume "murdoch" is another separate field value
Obviously here I just want "murdoch" but since "paul murdoch" was
indexed ANALYZED that will be returned as well. Using quotes with
Field.Index.NOT_ANALYZED isn't working for me. I'm thinking maybe it's
the QueryParser so I'm going trying a PhraseQuery instead.
Thanks,
Paul
-----Original Message-----
From: java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
[mailto:java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
] On Behalf Of Erick Erickson
Sent: Wednesday, February 24, 2010 4:23 PM
To: java-user@lucene.apache.org
Subject: Re: Phrase Search and NOT_ANALYZED
What does Luke's explain show you? That'll show you a lot about how
the query gets transformed......
My first guess is that stop words are messing you up....
Erick
On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul
<PA...@saic.com>wrote:
> Hi,
>
>
>
> I'm indexing a field using the StandardAnalyzer 2.9.
>
>
>
> field = new Field(fieldName, fieldValue, Field.Store.YES,
> Field.Index.NOT_ANALYZED);
>
>
>
> Let's say fieldName is "name" and fieldValue is "something in the
> index". When I perform the query...
>
>
>
> name:"something in the index"
>
>
>
> ... I don't get a hit. I'm using the StandardAnalyzer for searching.
> I've tested this with Luke and it doesn't work there either. What am
I
> missing?
>
>
>
> Thanks,
>
>
>
> Paul
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Phrase Search and NOT_ANALYZED
Posted by Erick Erickson <er...@gmail.com>.
What does Luke's explain show you? That'll show you a lot about how
the query gets transformed......
My first guess is that stop words are messing you up....
Erick
On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul <PA...@saic.com>wrote:
> Hi,
>
>
>
> I'm indexing a field using the StandardAnalyzer 2.9.
>
>
>
> field = new Field(fieldName, fieldValue, Field.Store.YES,
> Field.Index.NOT_ANALYZED);
>
>
>
> Let's say fieldName is "name" and fieldValue is "something in the
> index". When I perform the query...
>
>
>
> name:"something in the index"
>
>
>
> ... I don't get a hit. I'm using the StandardAnalyzer for searching.
> I've tested this with Luke and it doesn't work there either. What am I
> missing?
>
>
>
> Thanks,
>
>
>
> Paul
>
>
>
>
RE: Phrase Search and NOT_ANALYZED
Posted by "Murdoch, Paul" <PA...@saic.com>.
It still fails even when there are no stop words. I'm going to try a
PhraseQuery instead of relying on the QueryParser.
Regards,
Paul
-----Original Message-----
From: java-user-return-45151-PAUL.B.MURDOCH=saic.com@lucene.apache.org
[mailto:java-user-return-45151-PAUL.B.MURDOCH=saic.com@lucene.apache.org
] On Behalf Of Digy
Sent: Wednesday, February 24, 2010 4:45 PM
To: java-user@lucene.apache.org
Subject: RE: Phrase Search and NOT_ANALYZED
Since it is not analyzed, your text is stored as a single term in the
index
[something in the index].
But the query
name:"something in the index"
is translated as :
find 4 consecutive terms which have values "something", "in","the" and
"index" respectively.
or if stop words are removed
find 2 consecutive terms which have values "something" and "index"
You should either index it ANALYZED or use TermQuery while searching.
DIGY
-----Original Message-----
From: Murdoch, Paul [mailto:PAUL.B.MURDOCH@saic.com]
Sent: Wednesday, February 24, 2010 10:51 PM
To: java-user@lucene.apache.org
Subject: Phrase Search and NOT_ANALYZED
Hi,
I'm indexing a field using the StandardAnalyzer 2.9.
field = new Field(fieldName, fieldValue, Field.Store.YES,
Field.Index.NOT_ANALYZED);
Let's say fieldName is "name" and fieldValue is "something in the
index". When I perform the query...
name:"something in the index"
... I don't get a hit. I'm using the StandardAnalyzer for searching.
I've tested this with Luke and it doesn't work there either. What am I
missing?
Thanks,
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Phrase Search and NOT_ANALYZED
Posted by Digy <di...@gmail.com>.
Since it is not analyzed, your text is stored as a single term in the index
[something in the index].
But the query
name:"something in the index"
is translated as :
find 4 consecutive terms which have values "something", "in","the" and
"index" respectively.
or if stop words are removed
find 2 consecutive terms which have values "something" and "index"
You should either index it ANALYZED or use TermQuery while searching.
DIGY
-----Original Message-----
From: Murdoch, Paul [mailto:PAUL.B.MURDOCH@saic.com]
Sent: Wednesday, February 24, 2010 10:51 PM
To: java-user@lucene.apache.org
Subject: Phrase Search and NOT_ANALYZED
Hi,
I'm indexing a field using the StandardAnalyzer 2.9.
field = new Field(fieldName, fieldValue, Field.Store.YES,
Field.Index.NOT_ANALYZED);
Let's say fieldName is "name" and fieldValue is "something in the
index". When I perform the query...
name:"something in the index"
... I don't get a hit. I'm using the StandardAnalyzer for searching.
I've tested this with Luke and it doesn't work there either. What am I
missing?
Thanks,
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org