You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by theDude_2 <ao...@webmd.net> on 2009/05/01 21:33:38 UTC
MultiFieldQueryParser - using a different analyzer per field...
Hello fellow Lucene developers!
I have a bit of a question - and I can't find the answer in my lucene
book....
Im trying to create a query that will query 2 fields using different
analyzers and combine the scores together to give me my "hits". The idea is
that for the one dataset I want a pure text match only, and for the other I
want to use the stemming concept by using a custom made analyzer.
Is there a way to do this?
--This is what I am thinking (conceptually)------
MultiFieldQueryParser mfqp1 = new MultiFieldQueryParser(field1, new
StandardAnalyzer(), boosts);
MultiFieldQueryParser mfqp2 = new MultiFieldQueryParser(field2, new
PositionalPorterStopAnalyzer(), boosts);
my MultiFieldQueryParser = mfqp1 + mfqp2
----------------
The issue that I see is that if I just use one analyzer, I lose out. I know
I need to query multiple fields, in multiple ways, but I just dont know how
to make this work....
Any ideas?
--
View this message in context: http://www.nabble.com/MultiFieldQueryParser---using-a-different-analyzer-per-field...-tp23338538p23338538.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: MultiFieldQueryParser - using a different analyzer per field...
Posted by Erick Erickson <er...@gmail.com>.
Hmmmm... If that's the case, would you suggest copying the
relevant fields, using the different analyzers at index time? Then perhaps
combining the two Queries returned from the MFQP into two clauses
in a BooleanQuery? Or is there a more elegant way?
Because you're right of course, trying to use two different
Analyzers on the same field(s) would produce...er..."interesting" results.
Best
Erick
On Fri, May 1, 2009 at 5:57 PM, Uwe Schindler <uw...@thetaphi.de> wrote:
> Hi Erick,
>
> this is not what he wants to do, I think.
>
> He wants both things at once: One query string that searches both fields
> with different analyzers. For searching more than one field at once, he can
> use MultiFieldQueryParser, but not if the analyzer for both fields is
> different (because the query string itself is also analyzed, but the two
> different analyzers will produce different tokens which cannot be directly
> searched in both fields). So the query string should produce two different
> anaylzed tokens and the search for each token group should be done in
> parallel.
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
> > -----Original Message-----
> > From: Erick Erickson [mailto:erickerickson@gmail.com]
> > Sent: Friday, May 01, 2009 11:42 PM
> > To: java-user@lucene.apache.org
> > Subject: Re: MultiFieldQueryParser - using a different analyzer per
> > field...
> >
> > This looks like a job for PerFieldAnalyzerWrapper, no
> > MultiFieldQueryparser required........
> >
> > Best
> > Erick
> >
> > On Fri, May 1, 2009 at 3:33 PM, theDude_2 <ao...@webmd.net> wrote:
> >
> > >
> > > Hello fellow Lucene developers!
> > >
> > > I have a bit of a question - and I can't find the answer in my lucene
> > > book....
> > >
> > > Im trying to create a query that will query 2 fields using different
> > > analyzers and combine the scores together to give me my "hits". The
> > idea
> > > is
> > > that for the one dataset I want a pure text match only, and for the
> > other I
> > > want to use the stemming concept by using a custom made analyzer.
> > >
> > > Is there a way to do this?
> > >
> > > --This is what I am thinking (conceptually)------
> > > MultiFieldQueryParser mfqp1 = new MultiFieldQueryParser(field1, new
> > > StandardAnalyzer(), boosts);
> > > MultiFieldQueryParser mfqp2 = new MultiFieldQueryParser(field2, new
> > > PositionalPorterStopAnalyzer(), boosts);
> > >
> > > my MultiFieldQueryParser = mfqp1 + mfqp2
> > > ----------------
> > >
> > > The issue that I see is that if I just use one analyzer, I lose out. I
> > > know
> > > I need to query multiple fields, in multiple ways, but I just dont know
> > how
> > > to make this work....
> > >
> > > Any ideas?
> > > --
> > > View this message in context:
> > > http://www.nabble.com/MultiFieldQueryParser---using-a-different-
> > analyzer-per-field...-tp23338538p23338538.html
> > > Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> > >
> > >
> > > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail: java-user-help@lucene.apache.org
> > >
> > >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
RE: MultiFieldQueryParser - using a different analyzer per field...
Posted by Uwe Schindler <uw...@thetaphi.de>.
Hi Erick,
this is not what he wants to do, I think.
He wants both things at once: One query string that searches both fields
with different analyzers. For searching more than one field at once, he can
use MultiFieldQueryParser, but not if the analyzer for both fields is
different (because the query string itself is also analyzed, but the two
different analyzers will produce different tokens which cannot be directly
searched in both fields). So the query string should produce two different
anaylzed tokens and the search for each token group should be done in
parallel.
-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Friday, May 01, 2009 11:42 PM
> To: java-user@lucene.apache.org
> Subject: Re: MultiFieldQueryParser - using a different analyzer per
> field...
>
> This looks like a job for PerFieldAnalyzerWrapper, no
> MultiFieldQueryparser required........
>
> Best
> Erick
>
> On Fri, May 1, 2009 at 3:33 PM, theDude_2 <ao...@webmd.net> wrote:
>
> >
> > Hello fellow Lucene developers!
> >
> > I have a bit of a question - and I can't find the answer in my lucene
> > book....
> >
> > Im trying to create a query that will query 2 fields using different
> > analyzers and combine the scores together to give me my "hits". The
> idea
> > is
> > that for the one dataset I want a pure text match only, and for the
> other I
> > want to use the stemming concept by using a custom made analyzer.
> >
> > Is there a way to do this?
> >
> > --This is what I am thinking (conceptually)------
> > MultiFieldQueryParser mfqp1 = new MultiFieldQueryParser(field1, new
> > StandardAnalyzer(), boosts);
> > MultiFieldQueryParser mfqp2 = new MultiFieldQueryParser(field2, new
> > PositionalPorterStopAnalyzer(), boosts);
> >
> > my MultiFieldQueryParser = mfqp1 + mfqp2
> > ----------------
> >
> > The issue that I see is that if I just use one analyzer, I lose out. I
> > know
> > I need to query multiple fields, in multiple ways, but I just dont know
> how
> > to make this work....
> >
> > Any ideas?
> > --
> > View this message in context:
> > http://www.nabble.com/MultiFieldQueryParser---using-a-different-
> analyzer-per-field...-tp23338538p23338538.html
> > Sent from the Lucene - Java Users mailing list archive at Nabble.com.
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: MultiFieldQueryParser - using a different analyzer per field...
Posted by Erick Erickson <er...@gmail.com>.
This looks like a job for PerFieldAnalyzerWrapper, no
MultiFieldQueryparser required........
Best
Erick
On Fri, May 1, 2009 at 3:33 PM, theDude_2 <ao...@webmd.net> wrote:
>
> Hello fellow Lucene developers!
>
> I have a bit of a question - and I can't find the answer in my lucene
> book....
>
> Im trying to create a query that will query 2 fields using different
> analyzers and combine the scores together to give me my "hits". The idea
> is
> that for the one dataset I want a pure text match only, and for the other I
> want to use the stemming concept by using a custom made analyzer.
>
> Is there a way to do this?
>
> --This is what I am thinking (conceptually)------
> MultiFieldQueryParser mfqp1 = new MultiFieldQueryParser(field1, new
> StandardAnalyzer(), boosts);
> MultiFieldQueryParser mfqp2 = new MultiFieldQueryParser(field2, new
> PositionalPorterStopAnalyzer(), boosts);
>
> my MultiFieldQueryParser = mfqp1 + mfqp2
> ----------------
>
> The issue that I see is that if I just use one analyzer, I lose out. I
> know
> I need to query multiple fields, in multiple ways, but I just dont know how
> to make this work....
>
> Any ideas?
> --
> View this message in context:
> http://www.nabble.com/MultiFieldQueryParser---using-a-different-analyzer-per-field...-tp23338538p23338538.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: MultiFieldQueryParser - using a different analyzer per field...
Posted by theDude_2 <ao...@webmd.net>.
Hey guys: original poster here, and I found a solution! I created a wrapper
that could accept multiple analyzers and then combined them into a search:
here is the code.
------wrapper class---------
public class PositionalPorterStopAnalyzer extends Analyzer {
private Set stopWords;
public PositionalPorterStopAnalyzer() {
this(StopAnalyzer.ENGLISH_STOP_WORDS);
}
public PositionalPorterStopAnalyzer(String[] stopList) {
stopWords = StopFilter.makeStopSet(stopList);
}
public TokenStream tokenStream(String fieldName, Reader reader) {
StopFilter stopFilter = new StopFilter(new LowerCaseTokenizer(reader),
stopWords);
stopFilter.setEnablePositionIncrements(true);
return new PorterStemFilter(stopFilter);
}
}
------creating the search---------
PerFieldAnalyzerWrapper pfawBoth = new PerFieldAnalyzerWrapper(
new StandardAnalyzer());
pfawBoth.addAnalyzer("Field_1", new StandardAnalyzer());
pfawBoth.addAnalyzer("Field_2", new PositionalPorterStopAnalyzer());
mfqp = new MultiFieldQueryParser(
fields,
pfawBoth,
boosts);
-----------
thanks for all the help
----------------
theDude_2 wrote:
>
> Hello fellow Lucene developers!
>
> I have a bit of a question - and I can't find the answer in my lucene
> book....
>
> Im trying to create a query that will query 2 fields using different
> analyzers and combine the scores together to give me my "hits". The idea
> is that for the one dataset I want a pure text match only, and for the
> other I want to use the stemming concept by using a custom made analyzer.
>
> Is there a way to do this?
>
> --This is what I am thinking (conceptually)------
> MultiFieldQueryParser mfqp1 = new MultiFieldQueryParser(field1, new
> StandardAnalyzer(), boosts);
> MultiFieldQueryParser mfqp2 = new MultiFieldQueryParser(field2, new
> PositionalPorterStopAnalyzer(), boosts);
>
> my MultiFieldQueryParser = mfqp1 + mfqp2
> ----------------
>
> The issue that I see is that if I just use one analyzer, I lose out. I
> know I need to query multiple fields, in multiple ways, but I just dont
> know how to make this work....
>
> Any ideas?
>
--
View this message in context: http://www.nabble.com/MultiFieldQueryParser---using-a-different-analyzer-per-field...-tp23338538p23367209.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org