You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by David Spencer <da...@tropo.com> on 2004/06/09 22:39:47 UTC
extensible query parser - Re: Proximity Searches behavior
Erik Hatcher wrote:
> On Jun 9, 2004, at 12:21 PM, David Spencer wrote:
>
>>> show us that most folks query with 1 - 3 words and do not use the
>>> any of the advanced features.
>>
>>
>>
>> But with automagic query expansion these things might be done behind
>> the scenes. Nutch, for one, expands simple queries to check against
>> multiple fields, with different boosts, and even gives a bonus for
>> terms that are near each other.
>
>
> Ah yes! Don't worry, I hadn't forgotten about Nutch. I'm tinkering
> with its query parsing and analysis as we speak in fact. Very clever
> indeed.
>
>>> The elegance of the query syntax is quite important, and QueryParser
>>> has gotten a bit hairy. I would enjoy discussions on creating new
>>> query parsers (one size doesn't fit all, I don't think) and what syntax
>>
>>
>>
>> I suggested in some email a while ago making the QueryParser
>> extensible at, runtime or startup time, so you can add other types if
>> queries that it doesn't support - so you have a way of registering
>> these other query types (SpanQuery, SubstringQuery etc) and then some
>> syntax like "span:foo" to invoke the query expander registered w/
>> "span" on "foo"...
>
>
> I would be curious to see how an implementation of this played out.
> For example, could I add my own syntax such that
>
> "some phrase" <-3-> "another phrase"
>
> could be parsed into a SpanNearQuery of two SpanNearQuery's?
>
> I like the idea of a flexible run-time grammar, but it sounds too good
> to be true in a general purpose kinda way.
My idea isn't perfect for humans, but at least lets you use queries not
hard coded.
You have something like
[1] how you register, could be in existing QueryParser
void register( String name, SubqueryParser qp)
[2] what you register
interface SubQueryParser
{
Query parse( String s); // parses string user enters, forms a Query...
}
[3] example of registration
register( "substring", new SubstringQP()); // instead of prefix matches
allows term anywhere
register( "span", new SurroundQP());
register( "syn", new SynonymExpanderQP()); // expands a word to include
synonyms
[4] syntax
normal query parser syntax but add something else like "NAME::TEXT"
(note 2 colons) so
this: "black syn::bird"
expands to calls in the new extensible query parser, something like
BooleanQuery bq = ...
bq.add( new TermQuery( "contents", "black"))
bq.add( SubstringParser.parse( "bird")) // really SynonymExpanderQP
return bq
behind the scenes SynonymExpanderQP expanded "bird" to the query
equivalent of, um, "bird avian^.5 wingedanimal^.5" or whatnot.
[5] the point
Be backward compatible and "natural" for existing query syntax, but
leave a hook so that if you innovate and define new query expansion code
there's some hope of someone using it as they can in theory drop it in
and use it w/o coding. Right now if you create some code in this area I
suspect there's little chance people will try it out as there's too much
friction to try it out.
>
> Erik
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: extensible query parser - Re: Proximity Searches behavior
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jun 10, 2004, at 10:26 AM, Terry Steichen wrote:
> Erik,
>
> When is "Lucene in Action" scheduled to be out?
To add to what Otis said - I've been working feverishly to come to
"terms" with SpanQuery*, the new sorting feature, Highlighter, Nutch
analysis and much more. Lucene in Action will be accurate with Lucene
1.4, so it has been a tough job to learn and keep up. We are nearing
the end though, and I can pretty confidently say that Otis and I will
be completely done with the manuscript sometime in July. We are behind
where we wanted to be, but thankfully that has worked out well with the
timing of a Lucene 1.4 final release.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: extensible query parser - Re: Proximity Searches behavior
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Terry, LIA is currently scheduled to be out in August, but we may get
it done a bit later than that.
Otis
--- Terry Steichen <te...@net-frame.com> wrote:
> Erik,
>
> When is "Lucene in Action" scheduled to be out?
>
> Regards,
>
> Terry
>
> ----- Original Message -----
> From: "Erik Hatcher" <er...@ehatchersolutions.com>
> To: "Lucene Users List" <lu...@jakarta.apache.org>
> Sent: Thursday, June 10, 2004 5:04 AM
> Subject: Re: extensible query parser - Re: Proximity Searches
> behavior
>
>
> > On Jun 9, 2004, at 4:39 PM, David Spencer wrote:
> > >> I like the idea of a flexible run-time grammar, but it sounds
> too
> > >> good to be true in a general purpose kinda way.
> > >
> > > My idea isn't perfect for humans, but at least lets you use
> queries
> > > not hard coded.
> >
> > But in my idealistic view, getting something (near) perfect for
> humans
> > is what a QueryParser is all about. And, of course, this is domain
> and
> > application specific in a lot of ways.
> >
> > > [5] the point
> > >
> > > Be backward compatible and "natural" for existing query syntax,
> but
> > > leave a hook so that if you innovate and define new query
> expansion
> > > code there's some hope of someone using it as they can in theory
> drop
> > > it in and use it w/o coding. Right now if you create some code in
> this
> > > area I suspect there's little chance people will try it out as
> there's
> > > too much friction to try it out.
> >
> > I'm still grasping for a happy medium between the current
> QueryParser
> > and this idea of an awkward syntax general purpose pluggable
> parser.
> >
> > Interestingly the current QueryParser is pluggable in some
> interesting
> > ways thanks to the getters for each query time being overridable.
> For
> > example, disallowing wildcard and fuzzy queries, enhancing range
> query
> > to handle different formats, and changing PhraseQuery into a
> > SpanNearQuery are all tricks I'm including in Lucene in Action.
> >
> > Erik
> >
> >
> >
> ---------------------------------------------------------------------
> > To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> > For additional commands, e-mail:
> lucene-user-help@jakarta.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: extensible query parser - Re: Proximity Searches behavior
Posted by Terry Steichen <te...@net-frame.com>.
Erik,
When is "Lucene in Action" scheduled to be out?
Regards,
Terry
----- Original Message -----
From: "Erik Hatcher" <er...@ehatchersolutions.com>
To: "Lucene Users List" <lu...@jakarta.apache.org>
Sent: Thursday, June 10, 2004 5:04 AM
Subject: Re: extensible query parser - Re: Proximity Searches behavior
> On Jun 9, 2004, at 4:39 PM, David Spencer wrote:
> >> I like the idea of a flexible run-time grammar, but it sounds too
> >> good to be true in a general purpose kinda way.
> >
> > My idea isn't perfect for humans, but at least lets you use queries
> > not hard coded.
>
> But in my idealistic view, getting something (near) perfect for humans
> is what a QueryParser is all about. And, of course, this is domain and
> application specific in a lot of ways.
>
> > [5] the point
> >
> > Be backward compatible and "natural" for existing query syntax, but
> > leave a hook so that if you innovate and define new query expansion
> > code there's some hope of someone using it as they can in theory drop
> > it in and use it w/o coding. Right now if you create some code in this
> > area I suspect there's little chance people will try it out as there's
> > too much friction to try it out.
>
> I'm still grasping for a happy medium between the current QueryParser
> and this idea of an awkward syntax general purpose pluggable parser.
>
> Interestingly the current QueryParser is pluggable in some interesting
> ways thanks to the getters for each query time being overridable. For
> example, disallowing wildcard and fuzzy queries, enhancing range query
> to handle different formats, and changing PhraseQuery into a
> SpanNearQuery are all tricks I'm including in Lucene in Action.
>
> Erik
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: lucene-user-help@jakarta.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org
Re: extensible query parser - Re: Proximity Searches behavior
Posted by Erik Hatcher <er...@ehatchersolutions.com>.
On Jun 9, 2004, at 4:39 PM, David Spencer wrote:
>> I like the idea of a flexible run-time grammar, but it sounds too
>> good to be true in a general purpose kinda way.
>
> My idea isn't perfect for humans, but at least lets you use queries
> not hard coded.
But in my idealistic view, getting something (near) perfect for humans
is what a QueryParser is all about. And, of course, this is domain and
application specific in a lot of ways.
> [5] the point
>
> Be backward compatible and "natural" for existing query syntax, but
> leave a hook so that if you innovate and define new query expansion
> code there's some hope of someone using it as they can in theory drop
> it in and use it w/o coding. Right now if you create some code in this
> area I suspect there's little chance people will try it out as there's
> too much friction to try it out.
I'm still grasping for a happy medium between the current QueryParser
and this idea of an awkward syntax general purpose pluggable parser.
Interestingly the current QueryParser is pluggable in some interesting
ways thanks to the getters for each query time being overridable. For
example, disallowing wildcard and fuzzy queries, enhancing range query
to handle different formats, and changing PhraseQuery into a
SpanNearQuery are all tricks I'm including in Lucene in Action.
Erik
---------------------------------------------------------------------
To unsubscribe, e-mail: lucene-user-unsubscribe@jakarta.apache.org
For additional commands, e-mail: lucene-user-help@jakarta.apache.org