You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Jeff French <je...@mdbconsulting.com> on 2008/08/12 01:18:39 UTC
Query to ignore certain phrases
We're trying to perform a query where if our intended search term/phrase is
part of a specific larger phrase, we want to ignore that particular match,
but not the entire document (unless of course there are no other hits with
our intended term/phrase). For example, a query like:
"white house" UNLESS "russian white house"
should not produce a match on the phrase:
"russian white house"
but should match:
"white house"
Where this differs from the NOT operator is that we don't want to rule out a
document just because it contains "russian white house", we just want to
ignore the hit, so that this phrase:
"... in the russian white house as opposed to the american white house
..."
would return the document.
Can this be accomplished using Lucene or Qsol QueryParser syntax, or do we
need to write something custom?
--
View this message in context: http://www.nabble.com/Query-to-ignore-certain-phrases-tp18935560p18935560.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Query to ignore certain phrases
Posted by Doron Cohen <cd...@gmail.com>.
I can't see how to accomplish this without writing some special code,
and not just because of query parsing.
Phrases are searched by iterating the participating term
positions and when a match is found say for "b c" there is no way
to know whether another query "a b c d" matches exactly the corresponding
position.
Writing your own query to do this, one straightforward way is for its scorer
to maintain two exact phrase sub-scorers, but, (1) use for the score just
the
phraseFreq() found by the scorer of the inner phrase ("b c"), and (2)
if both phraseFreq() are the same - that of the inner phrase and that
of the outer phrase ("a b c d") ignore the document all together.
If this is where you're heading, start in the package javadocs for the
search package,
and see the code of PhraseScorer.
Span queries are another direction I would check for this, perhaps a higher
level
code can suffice with spans, but I don't know for sure.
Doron
On Tue, Aug 12, 2008 at 2:18 AM, Jeff French <je...@mdbconsulting.com> wrote:
>
> We're trying to perform a query where if our intended search term/phrase is
> part of a specific larger phrase, we want to ignore that particular match,
> but not the entire document (unless of course there are no other hits with
> our intended term/phrase). For example, a query like:
>
> "white house" UNLESS "russian white house"
>
> should not produce a match on the phrase:
>
> "russian white house"
>
> but should match:
>
> "white house"
>
> Where this differs from the NOT operator is that we don't want to rule out
> a
> document just because it contains "russian white house", we just want to
> ignore the hit, so that this phrase:
>
> "... in the russian white house as opposed to the american white house
> ..."
>
> would return the document.
>
> Can this be accomplished using Lucene or Qsol QueryParser syntax, or do we
> need to write something custom?
> --
> View this message in context:
> http://www.nabble.com/Query-to-ignore-certain-phrases-tp18935560p18935560.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
Re: Query to ignore certain phrases
Posted by Doron Cohen <cd...@gmail.com>.
>
> I think it should look something like this
>
> "white house" NOT "russian white house"~1
"a b c"~1 just matches more 'easily' than "a b c".
It will match for instance "a b d c". The NOT however
excludes all documents which match this, unlike requested logic.
In fact,
Q1: "a b" NOT "a b c"~1
is worse than
Q2: "a b" NOT "a b c"
in that a document containing
"a b x c"
should have been matched, and will be matched by Q2 but not with Q1.
Both Q1 and Q2 will not match this:
"a b x y z a b c"
which should be matched...
> "white house" UNLESS "russian white house"
> >
> > should not produce a match on the phrase:
> >
> > "russian white house"
> >
> > but should match:
> >
> > "white house"
> >
> > Where this differs from the NOT operator is that we don't want to rule
> out
> > a
> > document just because it contains "russian white house", we just want to
> > ignore the hit, so that this phrase:
> >
> > "... in the russian white house as opposed to the american white house
> > ..."
> >
> > would return the document.
> >
> > Can this be accomplished using Lucene or Qsol QueryParser syntax, or do
> we
> > need to write something custom?
>
Re: Query to ignore certain phrases
Posted by Alexander Aristov <al...@gmail.com>.
I think it should look something like this
"white house" NOT "russian white house"~1
http://lucene.apache.org/java/docs/queryparsersyntax.html#Escaping%20Special%20Characters
Alex
On 12/08/2008, Jeff French <je...@mdbconsulting.com> wrote:
>
>
> We're trying to perform a query where if our intended search term/phrase is
> part of a specific larger phrase, we want to ignore that particular match,
> but not the entire document (unless of course there are no other hits with
> our intended term/phrase). For example, a query like:
>
> "white house" UNLESS "russian white house"
>
> should not produce a match on the phrase:
>
> "russian white house"
>
> but should match:
>
> "white house"
>
> Where this differs from the NOT operator is that we don't want to rule out
> a
> document just because it contains "russian white house", we just want to
> ignore the hit, so that this phrase:
>
> "... in the russian white house as opposed to the american white house
> ..."
>
> would return the document.
>
> Can this be accomplished using Lucene or Qsol QueryParser syntax, or do we
> need to write something custom?
> --
> View this message in context:
> http://www.nabble.com/Query-to-ignore-certain-phrases-tp18935560p18935560.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>
--
Best Regards
Alexander Aristov
RE: Query to ignore certain phrases
Posted by Steven A Rowe <sa...@syr.edu>.
Hi Jeff,
I don't know of a query parser that will allow you to acheive this.
However, if you can programmatically construct (at least a component of) your queries, then you may want to check out Lucene's SpanQuery functionality.
In particular, using your example, if you combine a SpanFirstQuery over a SpanNearQuery on "white house" (to cover the case of "white house" at the beginning of a field) with a SpanNearQuery over a SpanRegexQuery for (?!russian$) - i.e., anything but "russian" - and another SpanNearQuery on "white house", you may get what you want.
In a lisp-ish query-like syntax:
{or:{first:{near~0:white house}}{near~0:{regex:(?!russian)}{near~0:white house}}
<http://lucene.apache.org/java/2_3_2/api/org/apache/lucene/search/spans/SpanQuery.html>
Steve
On 08/11/2008 at 7:18 PM, Jeff French wrote:
>
> We're trying to perform a query where if our intended search term/phrase
> is part of a specific larger phrase, we want to ignore that particular
> match, but not the entire document (unless of course there are no other
> hits with our intended term/phrase). For example, a query like:
>
> "white house" UNLESS "russian white house"
>
> should not produce a match on the phrase:
>
> "russian white house"
>
> but should match:
>
> "white house"
>
> Where this differs from the NOT operator is that we don't want to rule
> out a document just because it contains "russian white house", we just
> want to ignore the hit, so that this phrase:
>
> "... in the russian white house as opposed to the american white house ..."
>
> would return the document.
>
> Can this be accomplished using Lucene or Qsol QueryParser syntax, or do
> we need to write something custom?
> --
> View this message in context: http://www.nabble.com/Query-to-ignore-certain-phrases-tp18935560p18935560.html
> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org