You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by "Murdoch, Paul" <PA...@saic.com> on 2010/02/24 21:51:24 UTC

Phrase Search and NOT_ANALYZED

Hi,

 

I'm indexing a field using the StandardAnalyzer 2.9.

 

field = new Field(fieldName, fieldValue, Field.Store.YES,
Field.Index.NOT_ANALYZED);

 

Let's say fieldName is "name" and fieldValue is "something in the
index".  When I perform the query...

 

name:"something in the index"

 

... I don't get a hit.  I'm using the StandardAnalyzer for searching.
I've tested this with Luke and it doesn't work there either.  What am I
missing?

 

Thanks,

 

Paul

 


RE: Phrase Search and NOT_ANALYZED

Posted by "Murdoch, Paul" <PA...@saic.com>.
I would still be interested in knowing why the combination of the StandardAnalyzer, a phrase built using double quotes with no stop words, and the QueryParser doesn't return hits while building the same query with the StadardAnalyzer and a PhraseQuery does?

Thanks,

Paul

-----Original Message-----
From: java-user-return-45156-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45156-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Murdoch, Paul
Sent: Wednesday, February 24, 2010 5:11 PM
To: java-user@lucene.apache.org
Subject: RE: Phrase Search and NOT_ANALYZED

PhraseQuery appears to be working.  Thanks to all.

Paul 


-----Original Message-----
From: java-user-return-45155-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45155-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Murdoch, Paul
Sent: Wednesday, February 24, 2010 5:01 PM
To: java-user@lucene.apache.org
Subject: RE: Phrase Search and NOT_ANALYZED

Thanks,

I've been looking at that one too.  I'm trying to make it happen with the StandardAnalyzer.  Unfortunately, I think I see some redesign for more robustness in the future.

Cheers,

Paul 


-----Original Message-----
From: java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Robert Muir
Sent: Wednesday, February 24, 2010 4:55 PM
To: java-user@lucene.apache.org
Subject: Re: Phrase Search and NOT_ANALYZED

check out KeywordAnalyzer!

On Wed, Feb 24, 2010 at 4:51 PM, Murdoch, Paul <PA...@saic.com>wrote:

> It still happens if there are no stop words in the fieldValue. For
> instance if fieldValue was "paul murdoch", Luke would show the query as
> name:"paul murdoch" but no hits are returned.  If I change to
> Field.Index.ANALYZED it works.  The problem with ANALYZED is that there
> is a possibility of picking up false positive hits.  With queries
> like...
>
> name:murdoch    // assume "murdoch" is another separate field value
>
> Obviously here I just want "murdoch" but since "paul murdoch" was
> indexed ANALYZED that will be returned as well.  Using quotes with
> Field.Index.NOT_ANALYZED isn't working for me. I'm thinking maybe it's
> the QueryParser so I'm going trying a PhraseQuery instead.
>
> Thanks,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> [mailto:java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> ] On Behalf Of Erick Erickson
> Sent: Wednesday, February 24, 2010 4:23 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase Search and NOT_ANALYZED
>
> What does Luke's explain show you? That'll show you a lot about how
> the query gets transformed......
>
> My first guess is that stop words are messing you up....
>
> Erick
>
> On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul
> <PA...@saic.com>wrote:
>
> > Hi,
> >
> >
> >
> > I'm indexing a field using the StandardAnalyzer 2.9.
> >
> >
> >
> > field = new Field(fieldName, fieldValue, Field.Store.YES,
> > Field.Index.NOT_ANALYZED);
> >
> >
> >
> > Let's say fieldName is "name" and fieldValue is "something in the
> > index".  When I perform the query...
> >
> >
> >
> > name:"something in the index"
> >
> >
> >
> > ... I don't get a hit.  I'm using the StandardAnalyzer for searching.
> > I've tested this with Luke and it doesn't work there either.  What am
> I
> > missing?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Paul
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Robert Muir
rcmuir@gmail.com

RE: Phrase Search and NOT_ANALYZED

Posted by "Murdoch, Paul" <PA...@saic.com>.
PhraseQuery appears to be working.  Thanks to all.

Paul 


-----Original Message-----
From: java-user-return-45155-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45155-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Murdoch, Paul
Sent: Wednesday, February 24, 2010 5:01 PM
To: java-user@lucene.apache.org
Subject: RE: Phrase Search and NOT_ANALYZED

Thanks,

I've been looking at that one too.  I'm trying to make it happen with the StandardAnalyzer.  Unfortunately, I think I see some redesign for more robustness in the future.

Cheers,

Paul 


-----Original Message-----
From: java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Robert Muir
Sent: Wednesday, February 24, 2010 4:55 PM
To: java-user@lucene.apache.org
Subject: Re: Phrase Search and NOT_ANALYZED

check out KeywordAnalyzer!

On Wed, Feb 24, 2010 at 4:51 PM, Murdoch, Paul <PA...@saic.com>wrote:

> It still happens if there are no stop words in the fieldValue. For
> instance if fieldValue was "paul murdoch", Luke would show the query as
> name:"paul murdoch" but no hits are returned.  If I change to
> Field.Index.ANALYZED it works.  The problem with ANALYZED is that there
> is a possibility of picking up false positive hits.  With queries
> like...
>
> name:murdoch    // assume "murdoch" is another separate field value
>
> Obviously here I just want "murdoch" but since "paul murdoch" was
> indexed ANALYZED that will be returned as well.  Using quotes with
> Field.Index.NOT_ANALYZED isn't working for me. I'm thinking maybe it's
> the QueryParser so I'm going trying a PhraseQuery instead.
>
> Thanks,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> [mailto:java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> ] On Behalf Of Erick Erickson
> Sent: Wednesday, February 24, 2010 4:23 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase Search and NOT_ANALYZED
>
> What does Luke's explain show you? That'll show you a lot about how
> the query gets transformed......
>
> My first guess is that stop words are messing you up....
>
> Erick
>
> On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul
> <PA...@saic.com>wrote:
>
> > Hi,
> >
> >
> >
> > I'm indexing a field using the StandardAnalyzer 2.9.
> >
> >
> >
> > field = new Field(fieldName, fieldValue, Field.Store.YES,
> > Field.Index.NOT_ANALYZED);
> >
> >
> >
> > Let's say fieldName is "name" and fieldValue is "something in the
> > index".  When I perform the query...
> >
> >
> >
> > name:"something in the index"
> >
> >
> >
> > ... I don't get a hit.  I'm using the StandardAnalyzer for searching.
> > I've tested this with Luke and it doesn't work there either.  What am
> I
> > missing?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Paul
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Robert Muir
rcmuir@gmail.com

RE: Phrase Search and NOT_ANALYZED

Posted by "Murdoch, Paul" <PA...@saic.com>.
Thanks,

I've been looking at that one too.  I'm trying to make it happen with the StandardAnalyzer.  Unfortunately, I think I see some redesign for more robustness in the future.

Cheers,

Paul 


-----Original Message-----
From: java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org [mailto:java-user-return-45154-PAUL.B.MURDOCH=saic.com@lucene.apache.org] On Behalf Of Robert Muir
Sent: Wednesday, February 24, 2010 4:55 PM
To: java-user@lucene.apache.org
Subject: Re: Phrase Search and NOT_ANALYZED

check out KeywordAnalyzer!

On Wed, Feb 24, 2010 at 4:51 PM, Murdoch, Paul <PA...@saic.com>wrote:

> It still happens if there are no stop words in the fieldValue. For
> instance if fieldValue was "paul murdoch", Luke would show the query as
> name:"paul murdoch" but no hits are returned.  If I change to
> Field.Index.ANALYZED it works.  The problem with ANALYZED is that there
> is a possibility of picking up false positive hits.  With queries
> like...
>
> name:murdoch    // assume "murdoch" is another separate field value
>
> Obviously here I just want "murdoch" but since "paul murdoch" was
> indexed ANALYZED that will be returned as well.  Using quotes with
> Field.Index.NOT_ANALYZED isn't working for me. I'm thinking maybe it's
> the QueryParser so I'm going trying a PhraseQuery instead.
>
> Thanks,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> [mailto:java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> ] On Behalf Of Erick Erickson
> Sent: Wednesday, February 24, 2010 4:23 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase Search and NOT_ANALYZED
>
> What does Luke's explain show you? That'll show you a lot about how
> the query gets transformed......
>
> My first guess is that stop words are messing you up....
>
> Erick
>
> On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul
> <PA...@saic.com>wrote:
>
> > Hi,
> >
> >
> >
> > I'm indexing a field using the StandardAnalyzer 2.9.
> >
> >
> >
> > field = new Field(fieldName, fieldValue, Field.Store.YES,
> > Field.Index.NOT_ANALYZED);
> >
> >
> >
> > Let's say fieldName is "name" and fieldValue is "something in the
> > index".  When I perform the query...
> >
> >
> >
> > name:"something in the index"
> >
> >
> >
> > ... I don't get a hit.  I'm using the StandardAnalyzer for searching.
> > I've tested this with Luke and it doesn't work there either.  What am
> I
> > missing?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Paul
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Robert Muir
rcmuir@gmail.com

Re: Phrase Search and NOT_ANALYZED

Posted by Robert Muir <rc...@gmail.com>.
check out KeywordAnalyzer!

On Wed, Feb 24, 2010 at 4:51 PM, Murdoch, Paul <PA...@saic.com>wrote:

> It still happens if there are no stop words in the fieldValue. For
> instance if fieldValue was "paul murdoch", Luke would show the query as
> name:"paul murdoch" but no hits are returned.  If I change to
> Field.Index.ANALYZED it works.  The problem with ANALYZED is that there
> is a possibility of picking up false positive hits.  With queries
> like...
>
> name:murdoch    // assume "murdoch" is another separate field value
>
> Obviously here I just want "murdoch" but since "paul murdoch" was
> indexed ANALYZED that will be returned as well.  Using quotes with
> Field.Index.NOT_ANALYZED isn't working for me. I'm thinking maybe it's
> the QueryParser so I'm going trying a PhraseQuery instead.
>
> Thanks,
>
> Paul
>
>
> -----Original Message-----
> From: java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> [mailto:java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
> ] On Behalf Of Erick Erickson
> Sent: Wednesday, February 24, 2010 4:23 PM
> To: java-user@lucene.apache.org
> Subject: Re: Phrase Search and NOT_ANALYZED
>
> What does Luke's explain show you? That'll show you a lot about how
> the query gets transformed......
>
> My first guess is that stop words are messing you up....
>
> Erick
>
> On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul
> <PA...@saic.com>wrote:
>
> > Hi,
> >
> >
> >
> > I'm indexing a field using the StandardAnalyzer 2.9.
> >
> >
> >
> > field = new Field(fieldName, fieldValue, Field.Store.YES,
> > Field.Index.NOT_ANALYZED);
> >
> >
> >
> > Let's say fieldName is "name" and fieldValue is "something in the
> > index".  When I perform the query...
> >
> >
> >
> > name:"something in the index"
> >
> >
> >
> > ... I don't get a hit.  I'm using the StandardAnalyzer for searching.
> > I've tested this with Luke and it doesn't work there either.  What am
> I
> > missing?
> >
> >
> >
> > Thanks,
> >
> >
> >
> > Paul
> >
> >
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>


-- 
Robert Muir
rcmuir@gmail.com

RE: Phrase Search and NOT_ANALYZED

Posted by "Murdoch, Paul" <PA...@saic.com>.
It still happens if there are no stop words in the fieldValue. For
instance if fieldValue was "paul murdoch", Luke would show the query as
name:"paul murdoch" but no hits are returned.  If I change to
Field.Index.ANALYZED it works.  The problem with ANALYZED is that there
is a possibility of picking up false positive hits.  With queries
like...

name:murdoch	// assume "murdoch" is another separate field value

Obviously here I just want "murdoch" but since "paul murdoch" was
indexed ANALYZED that will be returned as well.  Using quotes with
Field.Index.NOT_ANALYZED isn't working for me. I'm thinking maybe it's
the QueryParser so I'm going trying a PhraseQuery instead.

Thanks,

Paul 


-----Original Message-----
From: java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
[mailto:java-user-return-45149-PAUL.B.MURDOCH=saic.com@lucene.apache.org
] On Behalf Of Erick Erickson
Sent: Wednesday, February 24, 2010 4:23 PM
To: java-user@lucene.apache.org
Subject: Re: Phrase Search and NOT_ANALYZED

What does Luke's explain show you? That'll show you a lot about how
the query gets transformed......

My first guess is that stop words are messing you up....

Erick

On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul
<PA...@saic.com>wrote:

> Hi,
>
>
>
> I'm indexing a field using the StandardAnalyzer 2.9.
>
>
>
> field = new Field(fieldName, fieldValue, Field.Store.YES,
> Field.Index.NOT_ANALYZED);
>
>
>
> Let's say fieldName is "name" and fieldValue is "something in the
> index".  When I perform the query...
>
>
>
> name:"something in the index"
>
>
>
> ... I don't get a hit.  I'm using the StandardAnalyzer for searching.
> I've tested this with Luke and it doesn't work there either.  What am
I
> missing?
>
>
>
> Thanks,
>
>
>
> Paul
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Re: Phrase Search and NOT_ANALYZED

Posted by Erick Erickson <er...@gmail.com>.
What does Luke's explain show you? That'll show you a lot about how
the query gets transformed......

My first guess is that stop words are messing you up....

Erick

On Wed, Feb 24, 2010 at 3:51 PM, Murdoch, Paul <PA...@saic.com>wrote:

> Hi,
>
>
>
> I'm indexing a field using the StandardAnalyzer 2.9.
>
>
>
> field = new Field(fieldName, fieldValue, Field.Store.YES,
> Field.Index.NOT_ANALYZED);
>
>
>
> Let's say fieldName is "name" and fieldValue is "something in the
> index".  When I perform the query...
>
>
>
> name:"something in the index"
>
>
>
> ... I don't get a hit.  I'm using the StandardAnalyzer for searching.
> I've tested this with Luke and it doesn't work there either.  What am I
> missing?
>
>
>
> Thanks,
>
>
>
> Paul
>
>
>
>

RE: Phrase Search and NOT_ANALYZED

Posted by "Murdoch, Paul" <PA...@saic.com>.
It still fails even when there are no stop words.  I'm going to try a
PhraseQuery instead of relying on the QueryParser.

Regards,

Paul 


-----Original Message-----
From: java-user-return-45151-PAUL.B.MURDOCH=saic.com@lucene.apache.org
[mailto:java-user-return-45151-PAUL.B.MURDOCH=saic.com@lucene.apache.org
] On Behalf Of Digy
Sent: Wednesday, February 24, 2010 4:45 PM
To: java-user@lucene.apache.org
Subject: RE: Phrase Search and NOT_ANALYZED

Since it is not analyzed, your text is stored as a single term in the
index
[something in the index].
But the query 
	name:"something in the index"
is translated as :
  find 4 consecutive terms which have values "something", "in","the" and
"index"  respectively.
or if stop words are removed 
  find 2 consecutive terms which have values "something" and "index"

You should either index it ANALYZED or use TermQuery while searching.

DIGY

-----Original Message-----
From: Murdoch, Paul [mailto:PAUL.B.MURDOCH@saic.com] 
Sent: Wednesday, February 24, 2010 10:51 PM
To: java-user@lucene.apache.org
Subject: Phrase Search and NOT_ANALYZED

Hi,

 

I'm indexing a field using the StandardAnalyzer 2.9.

 

field = new Field(fieldName, fieldValue, Field.Store.YES,
Field.Index.NOT_ANALYZED);

 

Let's say fieldName is "name" and fieldValue is "something in the
index".  When I perform the query...

 

name:"something in the index"

 

... I don't get a hit.  I'm using the StandardAnalyzer for searching.
I've tested this with Luke and it doesn't work there either.  What am I
missing?

 

Thanks,

 

Paul

 



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


RE: Phrase Search and NOT_ANALYZED

Posted by Digy <di...@gmail.com>.
Since it is not analyzed, your text is stored as a single term in the index
[something in the index].
But the query 
	name:"something in the index"
is translated as :
  find 4 consecutive terms which have values "something", "in","the" and
"index"  respectively.
or if stop words are removed 
  find 2 consecutive terms which have values "something" and "index"

You should either index it ANALYZED or use TermQuery while searching.

DIGY

-----Original Message-----
From: Murdoch, Paul [mailto:PAUL.B.MURDOCH@saic.com] 
Sent: Wednesday, February 24, 2010 10:51 PM
To: java-user@lucene.apache.org
Subject: Phrase Search and NOT_ANALYZED

Hi,

 

I'm indexing a field using the StandardAnalyzer 2.9.

 

field = new Field(fieldName, fieldValue, Field.Store.YES,
Field.Index.NOT_ANALYZED);

 

Let's say fieldName is "name" and fieldValue is "something in the
index".  When I perform the query...

 

name:"something in the index"

 

... I don't get a hit.  I'm using the StandardAnalyzer for searching.
I've tested this with Luke and it doesn't work there either.  What am I
missing?

 

Thanks,

 

Paul

 



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org