You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Paul Taylor <pa...@fastmail.fm> on 2007/05/12 19:02:54 UTC
Problem using wildcardsearch in phrase search
I seem to be having problems using a * in a phrase term query
This is my search String, its not finding any matches
54:"MusicIP PUID*"
If I match on a particular record it works ok
54:"MusicIP PUIDa39494bf-927e-1638-fb06-782ec55ac22d"
The problem appears to be the space character, because I have another
situation where there is no space and it works ok.
I use a keyword analyser to create the index so the value 'MusicIP
PUIDa39494bf-927e-1638-fb06-782ec55ac22d' is stored
as a single value, and then use the same analyser in my search code,
which is as follows:
public List <Integer> generalSearch(String luceneSearch)
{
System.out.println("Search Query Is"+luceneSearch);
List <Integer> matchingRows = new ArrayList<Integer>();
try
{
IndexSearcher is = new IndexSearcher(directory);
//Build a query based on the searchString and cached analyzer
QueryParser parser = new QueryParser(ROW_NUMBER,analyzer);
Query query = parser.parse(luceneSearch);
//run the search
Hits hits = is.search(query);
Iterator i = hits.iterator();
while(i.hasNext())
{
Document doc = ((Hit)i.next()).getDocument();
matchingRows.add(new
Integer(doc.getField(ROW_NUMBER).stringValue()));
}
}
catch (Exception e)
{
e.printStackTrace();
}
return matchingRows;
}
I cant see what the problem is, thanks paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Erick Erickson <er...@gmail.com>.
Somewhere in the list, I remember one of the guys who know what
they're talking about mentions something about KeywordAnalyzer
being "subject to the meta-semantics of the QueryParser".
So try looking at query.toString() in your example. What I think you'll
find is that KeywordAnalyzer doesn't quite do what you expect at
query time. This code
QueryParser qp = new QueryParser("field", new
KeywordAnalyzer());
Query q = qp.parse("Does this tokenize*");
System.out.println(q.toString());
Produces
field:Does field:this field:tokenize*
which isn't really what you're looking for, I don't think. In fact, I don't
see any difference between this and WhitespaceAnalyzer. What I
think you want is
Try rewriting the query and dumping the output to see how your
term expands...
If you haven't already, get a copy of Luke (google Lucene Luke). That'll
allow you to examine how different analyzers work at query time and
give you a much better idea of what is going on.....
But it's a beautiful Saturday, and I'm going to go outside and garden so....
Erick
On 5/12/07, Paul Taylor <pa...@fastmail.fm> wrote:
>
> I seem to be having problems using a * in a phrase term query
>
> This is my search String, its not finding any matches
> 54:"MusicIP PUID*"
>
> If I match on a particular record it works ok
> 54:"MusicIP PUIDa39494bf-927e-1638-fb06-782ec55ac22d"
>
> The problem appears to be the space character, because I have another
> situation where there is no space and it works ok.
>
> I use a keyword analyser to create the index so the value 'MusicIP
> PUIDa39494bf-927e-1638-fb06-782ec55ac22d' is stored
> as a single value, and then use the same analyser in my search code,
> which is as follows:
>
> public List <Integer> generalSearch(String luceneSearch)
> {
> System.out.println("Search Query Is"+luceneSearch);
> List <Integer> matchingRows = new ArrayList<Integer>();
> try
> {
> IndexSearcher is = new IndexSearcher(directory);
>
> //Build a query based on the searchString and cached analyzer
> QueryParser parser = new QueryParser(ROW_NUMBER,analyzer);
> Query query = parser.parse(luceneSearch);
> //run the search
> Hits hits = is.search(query);
> Iterator i = hits.iterator();
> while(i.hasNext())
> {
> Document doc = ((Hit)i.next()).getDocument();
> matchingRows.add(new
> Integer(doc.getField(ROW_NUMBER).stringValue()));
> }
> }
> catch (Exception e)
> {
> e.printStackTrace();
> }
> return matchingRows;
> }
>
> I cant see what the problem is, thanks paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: Problem using wildcardsearch in phrase search
Posted by Max Metral <ma...@artsalliancelabs.com>.
Instead of using QueryParser.Parse, what if you make a WildcardQuery
directly? I had similar troubles getting prefix queries (for ajax)
working properly, never did solve it.
-----Original Message-----
From: Paul Taylor [mailto:paul_t100@fastmail.fm]
Sent: Sunday, May 13, 2007 5:02 AM
To: java-user@lucene.apache.org
Subject: Re: Problem using wildcardsearch in phrase search
I think the KeywordAnlyser bit is maybe a red herring, the problem seems
to be that you cant use * within double quotes, I made some changes to
my data and index to remove the space character
If I fed 54:puid* to my code it generates a Prefix Query and works as
required
Search Query Is54:puid*
Parsed Search Query Is54:puid*of type:class
org.apache.lucene.search.PrefixQuery
but with the quotes (which I would need if my value contained spaces) I
only get a Term Query (which doesnt handle wildcards)
Search Query Is54:"puid*"
Parsed Search Query Is54:puid*of type:class
org.apache.lucene.search.TermQuery
so why is this ?
thanks Paul
code is:
public List <Integer> generalSearch(String luceneSearch)
{
System.out.println("Search Query Is"+luceneSearch);
List <Integer> matchingRows = new ArrayList<Integer>();
try
{
//make a new index searcher with the inmemory (RAM) index.
IndexSearcher is = new IndexSearcher(directory);
//Build a query based on the searchString and cached
analyzer
QueryParser parser = new QueryParser(ROW_NUMBER,analyzer);
Query query = parser.parse(luceneSearch);
System.out.println("Parsed Search Query
Is"+query.toString()+"of type:"+query.getClass());
//run the search
Hits hits = is.search(query);
Iterator i = hits.iterator();
while(i.hasNext())
{
Document doc = ((Hit)i.next()).getDocument();
matchingRows.add(new
Integer(doc.getField(ROW_NUMBER).stringValue()));
}
}
catch (Exception e)
{
e.printStackTrace();
}
System.out.println("Search Query Results
Set:"+matchingRows.size());
return matchingRows;
}
Mark Miller wrote:
>
>>
>>
>> Perhaps not like whitespaceanalyzer does in all cases, but this code
>>
>> QueryParser qp = new QueryParser("field", new
>> WhitespaceAnalyzer());
>>
>> Query q = qp.parse("Does this tokenize*");
>> System.out.println(q.toString());
>>
>> produces
>>
>> field:Does field:this field:tokenize*
>>
>>
>>
>> while this code
>> QueryParser qp = new QueryParser("field", new
>> KeywordAnalyzer());
>>
>> Query q = qp.parse("Does this tokenize*");
>> System.out.println(q.toString());
>>
>>
>> Produces:
>> field:Does field:this field:tokenize*
>>
>> The only difference is using KeywordAnalyzer rather than
>> WhitespaceAnalyzer.
>>
>> I sure don't see the difference, and it's puzzled me on and off.
>>
>> Erick
> QueryParser breaks up "Does this tokenize*" before it even gets to the
> Analyzer...each space separated term is fed to the analyzer one at a
> time...unless they are surrounded in quotes in which case the whole
> string in the quotes is fed into the analyzer.
>
> - Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Mark Miller <ma...@gmail.com>.
I do not know enough about PhraseQuery to say how hard it would be to
add support for wildcards, but I am sure there is some method of doing
it -- it has just not been done. From what I can tell it would be easier
to stop using PhraseQuery and use SpanQuery's if you wanted to do this.
Maybe someone more knowledgeable about PhraseQuery could elaborate on
some of the difficulties of adding wildcard support...then again I think
you may find discussion of this if you search the archives.
- Mark
Paul Taylor wrote:
> Mark Miller wrote:
>> You cannot use wildcards in quotes simply because the QueryParser
>> syntax does not look for such things...at the top level it is either
>> looking for a Wildcard token OR a Quoted token. There is good reason
>> for this: a phrase query does not support wildcards.
> OK thanks for all the info, as I said previously I worked round the
> problem so I dont need to do any coding, but the obvious followup
> question to above is:Why don't Phrase queries support wildcards, does
> it not just not make sense, or has it just not been done yet, as i
> said before there is nothing in the query syntax documentation to
> indicate it isnt supported.
>
> thanks paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Paul Taylor <pa...@fastmail.fm>.
Mark Miller wrote:
> You cannot use wildcards in quotes simply because the QueryParser
> syntax does not look for such things...at the top level it is either
> looking for a Wildcard token OR a Quoted token. There is good reason
> for this: a phrase query does not support wildcards.
OK thanks for all the info, as I said previously I worked round the
problem so I dont need to do any coding, but the obvious followup
question to above is:Why don't Phrase queries support wildcards, does it
not just not make sense, or has it just not been done yet, as i said
before there is nothing in the query syntax documentation to indicate it
isnt supported.
thanks paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Mark Miller <ma...@gmail.com>.
You cannot use wildcards in quotes simply because the QueryParser syntax
does not look for such things...at the top level it is either looking
for a Wildcard token OR a Quoted token. There is good reason for this: a
phrase query does not support wildcards. The hack that I suggested
(looking for a wildcard in the token before returning the TermQuery) was
only going to work because you where not generating a PhraseQuery with
your quotes...the KeywordAnalyzer only emits a single token keeping a
PhraseQuery from being generated....a TermQuery was being generated
instead...which you wanted to turn into a WildcardQuery if you found a *
or ? in the Term.
If you wanted Lucene to generally support wildcard queries within quotes
you have to do a bit of work. One way: instead of generating
PhraseQueries you override the QueryParser to generate SpanNearQueries,
then you need to implement a simple SpanWildcardQuery and change some
logic around in the QueryParser to look for wildcard characters in the
quoted token and generate the correct Query of SpanNearQuery's and
SpanWildcardQueries.
Again, I know I am not the clearest of explainers...I just have an itch
to try and help out anyway <g>.
- Mark
Paul Taylor wrote:
> Mark Miller wrote:
>> You can't use a wildcard within double quotes. The Lucene syntax
>> grammar does not look for such things.
> This is the bit I don't get (I have got round the problem), why can't
> you use wildcards within double quotes, this isnt mentioned anywhere
> in http://lucene.apache.org/java/docs/queryparsersyntax.html
>
> Paul
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Paul Taylor <pa...@fastmail.fm>.
Ive proposed a simple improvement in issue
https://issues.apache.org/jira/browse/LUCENE-884
thanks Paul
Chris Hostetter wrote:
> But as i said: if you have suggestions for clarifying the docs, please
> submit them as a patch. just saying the docs need to be improved without
> providing a specific suggestion for how they can be improved doesn't
> really help the situation.
>
>
>
> -Hoss
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Chris Hostetter <ho...@fucit.org>.
: queryparsersyntax page which is where I expect most novices (such as
: myself) start with lucene seems to indicate that wildcards can be used
: in, and this page is
: as far as one should need to go to understand basic query syntax, this
: page should be corrected.
if you have a suggestion for improving the page, please by all means
submit a patch with your suggestions. I've said it before and i'll say it
again: novice users are the best documentation writers because unlike
seasoned experinced usersor developers, they are ones just learning the
system who don't already have ingrained knowledge that they take for
granted.
: A Phrase is a group of words surrounded by double quotes such as "hello
: dolly".
:
: ....
:
: Wildcard Searches
: Lucene supports single and multiple character wildcard searches.
: To perform a multiple character wildcard search use the "*" symbol.
: Multiple character wildcard searches looks for 0 or more characters. For
: example, to search for test, tests or tester, you can use the search:
:
: test*
: You can also use the wildcard searches in the middle of a term.
:
: '
: there is nothing to indicate in the section on Wildcard Searches that it
: can be performed only on Single word terms not Phrase terms.
I would meanwhile argue that there is nothing in the description of a
Phrase to indicate that it can be anything other then what it says "a
group of words surrounded by double quotes" .. at no point does ist
suggest that other types of queries or syntax can be used inside the
quotes. likewise the discussion of Wildcards makes no mention of phrases
to suggest that wildcard characters can be used in a phrase.
But as i said: if you have suggestions for clarifying the docs, please
submit them as a patch. just saying the docs need to be improved without
providing a specific suggestion for how they can be improved doesn't
really help the situation.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Paul Taylor <pa...@fastmail.fm>.
Chris Hostetter wrote:
> : > You can't use a wildcard within double quotes. The Lucene syntax
> : > grammar does not look for such things.
>
> : This is the bit I don't get (I have got round the problem), why can't
> : you use wildcards within double quotes, this isnt mentioned anywhere in
> : http://lucene.apache.org/java/docs/queryparsersyntax.html
>
> Documenting a negated set against the universe of all posisble syntax
> isn't feasible -- ie: the docs list what things QueryParser can do, not
> the full list of all permutations of things it can't do.
>
>
Thats true, obviously but the point I was making was that the
queryparsersyntax page which is where I expect most novices (such as
myself) start with lucene seems to indicate that wildcards can be used
in, and this page is
as far as one should need to go to understand basic query syntax, this
page should be corrected.
Quoting:
'Terms: A query is broken up into terms and operators. There are two
types of terms: Single Terms and Phrases.
A Single Term is a single word such as "test" or "hello".
A Phrase is a group of words surrounded by double quotes such as "hello
dolly".
....
Wildcard Searches
Lucene supports single and multiple character wildcard searches.
To perform a multiple character wildcard search use the "*" symbol.
Multiple character wildcard searches looks for 0 or more characters. For
example, to search for test, tests or tester, you can use the search:
test*
You can also use the wildcard searches in the middle of a term.
'
there is nothing to indicate in the section on Wildcard Searches that it
can be performed only on Single word terms not Phrase terms.
> It can't deal with Span Queries,
No they are never mentioned(,I don't even know what they are.), so it is
not a problem.
> it can't handle fuzzy phrase queries,
>
no and it states this correctly ' Lucene supports fuzzy searches based
on the Levenshtein Distance, or Edit Distance algorithm. To do a fuzzy
search use the tilde, "~", symbol at the end of a Single word Term'
> If you read the section on Proximity Searches (aka: PhraseQuery) you'll
> eee it clearly says "Lucene supports finding words are a within a specific
> distance away." ... note that word "word". It does *not* say "...finding
> QUERIES are within...."
>
but why would I look here, I was just trying to understand query syntax,
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Chris Hostetter <ho...@fucit.org>.
: > You can't use a wildcard within double quotes. The Lucene syntax
: > grammar does not look for such things.
: This is the bit I don't get (I have got round the problem), why can't
: you use wildcards within double quotes, this isnt mentioned anywhere in
: http://lucene.apache.org/java/docs/queryparsersyntax.html
Documenting a negated set against the universe of all posisble syntax
isn't feasible -- ie: the docs list what things QueryParser can do, not
the full list of all permutations of things it can't do.
It can't deal with Span Queries, it can't handle fuzzy phrase queries, it
doesn't suppord math on field vaues, it doesn't support negative values
for boost, ... these are just some of the things it doesn't support.
If you read the section on Proximity Searches (aka: PhraseQuery) you'll
eee it clearly says "Lucene supports finding words are a within a specific
distance away." ... note that word "word". It does *not* say "...finding
QUERIES are within...." There is no implication that the QueryParser's
Proximity Searching support incudes any query types (like wildcards) other
then simple term matching.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Paul Taylor <pa...@fastmail.fm>.
Mark Miller wrote:
> You can't use a wildcard within double quotes. The Lucene syntax
> grammar does not look for such things.
This is the bit I don't get (I have got round the problem), why can't
you use wildcards within double quotes, this isnt mentioned anywhere in
http://lucene.apache.org/java/docs/queryparsersyntax.html
Paul
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Mark Miller <ma...@gmail.com>.
> I think the KeywordAnlyser bit is maybe a red herring, the problem
> seems to be that you cant use * within double quotes, I made some
> changes to my data and index to remove the space character
>
You can't use a wildcard within double quotes. The Lucene syntax grammar
does not look for such things. The KeywordAnalyzer is not a red herring
though...it will not work as expected. Like I said, the QueryParser
breaks up a query where it sees spaces BEFORE sending to an analyzer.
This is why Erik thinks that KeywordAnazlyer is the same as
WhiteSpaceAnalyzer...it is not...KeywordAnazlyer does not break up words
on spaces...it just looks like it when you use QueryParser because
QueryParser breaks up words by spaces before even using an Analyzer.
When you use KeywordAnalyzer for indexing it will not break on
spaces...when you use it with the QueryParser it WILL break on spaces.
> If I fed 54:puid* to my code it generates a Prefix Query and works as
> required
> Search Query Is54:puid*
> Parsed Search Query Is54:puid*of type:class
> org.apache.lucene.search.PrefixQuery
>
> but with the quotes (which I would need if my value contained spaces)
> I only get a Term Query (which doesnt handle wildcards)
> Search Query Is54:"puid*"
> Parsed Search Query Is54:puid*of type:class
> org.apache.lucene.search.TermQuery
I explained this, though apparently not well. QueryParser feeds chunks
of text to the analyzer. If there are spaces, each term is fed to the
analyzer one at a time and a TermQuery is generated for each chunk. If
you have something in quotes, the whole piece is fed to the
analyzer...if the analyzer generates more than one token, a phrasequery
is generated...if only ONE token is generated (as is the case when you
feed anything within quotes to a KeywordAnalyzer), then a TermQuery is
generated. The code does not look for any wildcards. Look at the
QueryParser code, getField in particular. One option you have is to
override getField, and before generating a TermQuery, scan for wildcards
and if you find one, return a wildcardquery instead of a term query.
I realize I have a weak grasp of my own language and can be a lot less
than clear...but the information you need has certainly been given in my
emails. Perhaps someone else could be a little more clearer than I am able.
- Mark
>
> so why is this ?
>
> thanks Paul
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Paul Taylor <pa...@fastmail.fm>.
I think the KeywordAnlyser bit is maybe a red herring, the problem seems
to be that you cant use * within double quotes, I made some changes to
my data and index to remove the space character
If I fed 54:puid* to my code it generates a Prefix Query and works as
required
Search Query Is54:puid*
Parsed Search Query Is54:puid*of type:class
org.apache.lucene.search.PrefixQuery
but with the quotes (which I would need if my value contained spaces) I
only get a Term Query (which doesnt handle wildcards)
Search Query Is54:"puid*"
Parsed Search Query Is54:puid*of type:class
org.apache.lucene.search.TermQuery
so why is this ?
thanks Paul
code is:
public List <Integer> generalSearch(String luceneSearch)
{
System.out.println("Search Query Is"+luceneSearch);
List <Integer> matchingRows = new ArrayList<Integer>();
try
{
//make a new index searcher with the inmemory (RAM) index.
IndexSearcher is = new IndexSearcher(directory);
//Build a query based on the searchString and cached analyzer
QueryParser parser = new QueryParser(ROW_NUMBER,analyzer);
Query query = parser.parse(luceneSearch);
System.out.println("Parsed Search Query
Is"+query.toString()+"of type:"+query.getClass());
//run the search
Hits hits = is.search(query);
Iterator i = hits.iterator();
while(i.hasNext())
{
Document doc = ((Hit)i.next()).getDocument();
matchingRows.add(new
Integer(doc.getField(ROW_NUMBER).stringValue()));
}
}
catch (Exception e)
{
e.printStackTrace();
}
System.out.println("Search Query Results Set:"+matchingRows.size());
return matchingRows;
}
Mark Miller wrote:
>
>>
>>
>> Perhaps not like whitespaceanalyzer does in all cases, but this code
>>
>> QueryParser qp = new QueryParser("field", new
>> WhitespaceAnalyzer());
>>
>> Query q = qp.parse("Does this tokenize*");
>> System.out.println(q.toString());
>>
>> produces
>>
>> field:Does field:this field:tokenize*
>>
>>
>>
>> while this code
>> QueryParser qp = new QueryParser("field", new
>> KeywordAnalyzer());
>>
>> Query q = qp.parse("Does this tokenize*");
>> System.out.println(q.toString());
>>
>>
>> Produces:
>> field:Does field:this field:tokenize*
>>
>> The only difference is using KeywordAnalyzer rather than
>> WhitespaceAnalyzer.
>>
>> I sure don't see the difference, and it's puzzled me on and off.
>>
>> Erick
> QueryParser breaks up "Does this tokenize*" before it even gets to the
> Analyzer...each space separated term is fed to the analyzer one at a
> time...unless they are surrounded in quotes in which case the whole
> string in the quotes is fed into the analyzer.
>
> - Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Mark Miller <ma...@gmail.com>.
>
>
> Perhaps not like whitespaceanalyzer does in all cases, but this code
>
> QueryParser qp = new QueryParser("field", new
> WhitespaceAnalyzer());
>
> Query q = qp.parse("Does this tokenize*");
> System.out.println(q.toString());
>
> produces
>
> field:Does field:this field:tokenize*
>
>
>
> while this code
> QueryParser qp = new QueryParser("field", new
> KeywordAnalyzer());
>
> Query q = qp.parse("Does this tokenize*");
> System.out.println(q.toString());
>
>
> Produces:
> field:Does field:this field:tokenize*
>
> The only difference is using KeywordAnalyzer rather than
> WhitespaceAnalyzer.
>
> I sure don't see the difference, and it's puzzled me on and off.
>
> Erick
QueryParser breaks up "Does this tokenize*" before it even gets to the
Analyzer...each space separated term is fed to the analyzer one at a
time...unless they are surrounded in quotes in which case the whole
string in the quotes is fed into the analyzer.
- Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Erick Erickson <er...@gmail.com>.
See below....
On 5/12/07, Mark Miller <ma...@gmail.com> wrote:
>
>
>
> Paul Taylor wrote:
> > I seem to be having problems using a * in a phrase term query
> >
> > This is my search String, its not finding any matches
> > 54:"MusicIP PUID*"
> >
> > If I match on a particular record it works ok
> > 54:"MusicIP PUIDa39494bf-927e-1638-fb06-782ec55ac22d"
> >
> "MusicIP PUID*" means to search for MusicIP within one of PUID*...I am
> pretty sure that KeywordAnalyzer does not split on whitespace like
> WhiteSpaceAnalyzer does...
Perhaps not like whitespaceanalyzer does in all cases, but this code
QueryParser qp = new QueryParser("field", new
WhitespaceAnalyzer());
Query q = qp.parse("Does this tokenize*");
System.out.println(q.toString());
produces
field:Does field:this field:tokenize*
while this code
QueryParser qp = new QueryParser("field", new
KeywordAnalyzer());
Query q = qp.parse("Does this tokenize*");
System.out.println(q.toString());
Produces:
field:Does field:this field:tokenize*
The only difference is using KeywordAnalyzer rather than WhitespaceAnalyzer.
I sure don't see the difference, and it's puzzled me on and off.
Erick
which means that MusicIP is never within one
> of PUID*..."MusicIP PUIDa39494bf-927e-1638-fb06-782ec55ac22d" is one
> whole token.
>
> - Mark
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Problem using wildcardsearch in phrase search
Posted by Mark Miller <ma...@gmail.com>.
Well I am confused so I suppose I'll let someone else give it a shot.
Just in case though...if you are using the query: fieldname:"MusicIP Puid*"
Then you should not...you need to leave out the quotes...quotes create a
phrasequery, and a phrasequery will not match what is in your index.
This may end up being the problem though...without the quotes, the space
will be treated as the default operator -- you may need to write your
own super simple parser that suits your needs instead...you don't want
the space to act as a default operator but as a normal token character.
Then again, I am not sure if you are using those quotes or not...
- Mark
Paul Taylor wrote:
> Mark Miller wrote:
>> Didn't you say you where using a phrasequery? If you are, things will
>> not work as expected. You need to leave the quotes out of your search
>> as a phrasequery will not match what you are putting in your index.
>> If you are not using a phrasequery then things should work as you
>> would expect. Can you provide any other info?
>>
>> - Mark
>
> No, I meant I was parsing a search String of the form
> fieldname:"MusicIP Puid*" (because I was using quotes and it is
> surrounded by double quotes I called it a phrase query), then I use
> the QueryParser().parse() to convert this search string into a Query
> (as shown in the code I posted earlier)
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Mark Miller <ma...@gmail.com>.
This just keeps running around in my head...
I was wrong on one point...if you use the KeywordAnalyzer and you put
your search in quotes then you will not generate a phrase query because
a PhraseQuery is only generated if the analyzer produces more than one
token. The problem is that, instead of a PhraseQuery, a TermQuery is
generated. You don't get to the wildcard handling part of the parser...a
TermQuery is returned. The QueryParser's javacc syntax will not
recognize wildcards in quotes and the handling for a quoted token does
not check for wildcards.
You might try an override in 'Query getFieldQuery(String field, String
queryText)' that scans the token for a wildcard and if it finds one,
generate a wildcard query instead of a termquery.
- Mark
Paul Taylor wrote:
> Mark Miller wrote:
>> Didn't you say you where using a phrasequery? If you are, things will
>> not work as expected. You need to leave the quotes out of your search
>> as a phrasequery will not match what you are putting in your index.
>> If you are not using a phrasequery then things should work as you
>> would expect. Can you provide any other info?
>>
>> - Mark
>
> No, I meant I was parsing a search String of the form
> fieldname:"MusicIP Puid*" (because I was using quotes and it is
> surrounded by double quotes I called it a phrase query), then I use
> the QueryParser().parse() to convert this search string into a Query
> (as shown in the code I posted earlier)
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Paul Taylor <pa...@fastmail.fm>.
Mark Miller wrote:
> Didn't you say you where using a phrasequery? If you are, things will
> not work as expected. You need to leave the quotes out of your search
> as a phrasequery will not match what you are putting in your index. If
> you are not using a phrasequery then things should work as you would
> expect. Can you provide any other info?
>
> - Mark
No, I meant I was parsing a search String of the form fieldname:"MusicIP
Puid*" (because I was using quotes and it is surrounded by double quotes
I called it a phrase query), then I use the QueryParser().parse() to
convert this search string into a Query (as shown in the code I posted
earlier)
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Mark Miller <ma...@gmail.com>.
Paul Taylor wrote:
> Mark Miller wrote:
>> "MusicIP PUID*" means to search for MusicIP within one of PUID*
> Sorry I dont understand, can you give me a further reference
>
>> ...I am pretty sure that KeywordAnalyzer does not split on whitespace
>> like WhiteSpaceAnalyzer does...which means that MusicIP is never
>> within one of PUID*..."MusicIP
>> PUIDa39494bf-927e-1638-fb06-782ec55ac22d" is one whole token.
>>
> I use keyword analyser for both creating the index and parsing the
> query, I was hoping that MusicIP Puid* would be treated as one thing
> when I search
>
>
Didn't you say you where using a phrasequery? If you are, things will
not work as expected. You need to leave the quotes out of your search as
a phrasequery will not match what you are putting in your index. If you
are not using a phrasequery then things should work as you would expect.
Can you provide any other info?
- Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Paul Taylor <pa...@fastmail.fm>.
Mark Miller wrote:
> "MusicIP PUID*" means to search for MusicIP within one of PUID*
Sorry I dont understand, can you give me a further reference
> ...I am pretty sure that KeywordAnalyzer does not split on whitespace
> like WhiteSpaceAnalyzer does...which means that MusicIP is never
> within one of PUID*..."MusicIP
> PUIDa39494bf-927e-1638-fb06-782ec55ac22d" is one whole token.
>
I use keyword analyser for both creating the index and parsing the
query, I was hoping that MusicIP Puid* would be treated as one thing
when I search
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Problem using wildcardsearch in phrase search
Posted by Mark Miller <ma...@gmail.com>.
Paul Taylor wrote:
> I seem to be having problems using a * in a phrase term query
>
> This is my search String, its not finding any matches
> 54:"MusicIP PUID*"
>
> If I match on a particular record it works ok
> 54:"MusicIP PUIDa39494bf-927e-1638-fb06-782ec55ac22d"
>
"MusicIP PUID*" means to search for MusicIP within one of PUID*...I am
pretty sure that KeywordAnalyzer does not split on whitespace like
WhiteSpaceAnalyzer does...which means that MusicIP is never within one
of PUID*..."MusicIP PUIDa39494bf-927e-1638-fb06-782ec55ac22d" is one
whole token.
- Mark
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org