You are viewing a plain text version of this content. The canonical link for it is here.
Posted to java-user@lucene.apache.org by Steven A Rowe <sa...@syr.edu> on 2008/09/09 22:11:17 UTC
RE: Re: Replacing FAST functionality at sesam.no - ShingleFilter+exact matching
Hi mck,
On 09/09/2008 at 12:58 PM, Mck wrote:
> > *ShortVersion*
> > is there a way to make the ShingleFilter perform exact matching via
> > inserting ^ $ begin/end markers?
>
> Reading through the mailing list i see how exact matching can
> be done, a la STFW to myself...
>
> So the ShortVersion now stands:
>
> For my query "abcd efgh ijkl"
> Why does a (perfect looking) MultiPhraseQuery with
> termArrays[0] = { list_entry_shingles:abcd
> list_entry_shingles:abcd efgh
> list_entry_shingles:abcd efgh ijkl
> }
> termArrays[1] = { list_entry_shingles:efgh
> list_entry_shingles:efgh ijkl
> }
> termArrays[2] = { list_entry_shingles:ijkl }
>
> return only "abcd efgh ijkl" !?
>
> (when the field is indexed as TextField this is the only hit i get)
> (when the field is indexed as StrField i get zero hits!)
>
> When the index contains 9 entries:
> "abcd efgh ijkl", "abcd efgh", "efgh ijkl", "abcd", "efgh",
> "ijkl", "ijkl efgh", "efgh abcd", and "ijkl efgh abcd".
>
> Does this MultiPhraseQuery actually require a match against
> *every* item in each termArray on any document?
I've never used MultiPhraseQuery, but I *think* (based on the Javadocs) that it requires one match from each termArrays[] entry, contiguously, in the same sequence as the termArrays[] entries (unless you add slop, which I don't think you're doing).
A TextField index would have ("abcd", "efgh", "ijkl") for the "abcd efgh ijkl" document (assuming you used WhitespaceAnalyzer, which I believe you showed in one of your emails); unlike all of the other documents, one member from each of your query's termArrays[] entries is sequentially present, so I think that the behavior you're seeing is expected. If you add "abcd efgh ijkl mnop" as a document, it should also be matched.
Looks to me like MultiPhraseQuery is getting in the way. Shingles that begin at the same word are given the same position by ShingleFilter, and Solr's FieldQParserPlugin creates a MultiPhraseQuery when it encounters tokens in a query with the same position. I think what you want is to convert queries into shingle disjunctions (*any* matching shingle results in a hit), right?
Any Solr cognoscenti know how to arrange for Solr's query parser to avoid invoking MultiPhraseQuery?
Steve
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Re: Replacing FAST functionality at sesam.no -
ShingleFilter+exactmatching
Posted by Mck <mi...@semb.wever.org>.
> [snip] The option thus should be named something like
> "coterminalPositionIncrement". This seems like a reasonable addition,
> and a patch likely would be accepted, if it included unit tests.
Done.
https://issues.apache.org/jira/browse/LUCENE-1380
~mck
--
"The only thing I know, is that I know nothing." Socrates
| semb.wever.org | sesat.no | sesam.no |
Re: Replacing FAST functionality atsesam.no-ShingleFilter+exactmatching
Posted by Chris Hostetter <ho...@fucit.org>.
: Does it make sense for me to rewrite the ShingleFilter patch to ensure
: the first token returned always has positionIncrement=1 regardless if
: enablePositions is true or false?
I don't know anything about ShingleFilter -- i've never looked at it and
i'm not entirely certain i even understand what it does, let alone how
your patch is attempting to modify it -- but if you've got code that takes
in text and it produces multiple tokens as a result, then that first token
it produces should (probably) have a non-zero positionIncrement. if your
code takes in "a" Token and produces multiple tokens to replace it, then
the first token you produce should (probably) have the same
positionIncrement as the input Token.
(Disclaimer; it's been a lon time since i worked with the TokenStream
APIs, so i'm hoping my memory isn't faulty and someone else backs me up
with a "yeah, that's correct")
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
Re: Replacing FAST functionality
atsesam.no-ShingleFilter+exactmatching
Posted by Mck <mi...@semb.wever.org>.
> (i'll test too that BooleanQuery works as presumed in my case...)
Indeed it works beautifully with a BooleanQuery.
I've updated the patch to LUCENE-1380
~mck
--
"If you have any trouble sounding condescending, find a Unix user to
show you how it's done." Scott Adams
| semb.wever.org | sesat.no | sesam.no |
Re: Replacing FAST functionality
atsesam.no-ShingleFilter+exactmatching
Posted by Mck <mi...@semb.wever.org>.
Chris,
> the safe thing to do is make sure the first token has
> a positionIncrement of "1" and the 'synonyms" after that use an
> increment of "0"
Yes it makes sense to have at minimum the first token with
positionIncrement=1
I didn't see outside "the vacuum" at all, thank you for explaining.
Does it make sense for me to rewrite the ShingleFilter patch to ensure
the first token returned always has positionIncrement=1 regardless if
enablePositions is true or false?
(i'll test too that BooleanQuery works as presumed in my case...)
Would such a rewrite of the ShingleFilter patch be a substitute for the
custom Analyzer you talk about?
(i'm pushing to keep any patch restricted to the ShingleFilter since my
gut feeling is still that's where the change in behaviour is).
~mck
--
"Between two evils, I always pick the one I never tried before." Mae
West
| semb.wever.org | sesat.no | sesam.no |
RE: Re: Replacing FAST functionality atsesam.no-ShingleFilter+exactmatching
Posted by Chris Hostetter <ho...@fucit.org>.
: The query parser expects you to assign positionIncrement=0 for synonyms
: in this manner.
correct.
: The one kludge i see is that the QueryParser expects the total positions
: found to be greater than or equal to one. It might not be intentionally
: dealing with the total position count being zero. But the situation
: where you have many synonyms is the same as having one token and it
: having many synonyms, so positionCount=0 == positionCount=1.
there has definitely been some wonkiness in various places in the code
relating to the first token not having a positionIncremenet of "1" ... i
don't rememebr the details, and maybe it works fine even if every token in
a stream is "0" but the safe thing to do is make sure the first token has
a positionIncrement of "1" and the 'synonyms" after that use an increment
of "0"
This is important not only in case the Lucene internals freak out when
the "first" token has an increment of "0" but also because you have no way
of knowing if the first token you produce is really the first token being
given to the IndexWriter (or QueryParser or what have you)
To be a well behave TokenStream producer you can't assume you opperate in
a vacume:
1) multiple "Field" instances with the same field name could be added to a
document, with an Analyzer that uses your Filter but doesn't define any
particular positionIncrementGap ... if every token you produce has an
increment of "0" all the tokens from the second Field instance will have
the same resulting positions as all the tokens from the first Field
instance (ie: they will all be considered synonyms of each other)
2) I could write an Analyzer that uses your Filter but always adds a
starting "marker token" to the front of the TokenStream and a differnt
ending marker token to the end of hte stream (for doing creative things
with SpanNearQueries) ... if all the tokens you produce have a
positionIncrement of 0, the result would be that they would be considered
synonyms of the starting marker token.
: I would think that both should lead to a BooleanQuery being constructed
: by the QueryParser. (But the synonyms generated by the ShingleFilter are
: in fact phrases so maybe it is wiser to use the MultiPhraseQuery.)
If QueryParser gives an Analyzer a chunk of text, and it produces a stream
of tokens that all exist at the same position, it produces a a
BooleanQuery, if they are *all* at differnet positions it produces a
PhraseQuery, if *some* are at the same position, it produces a
MultiPhraseQuery ... this is fairly fundamental to how QueryParser works,
and can be relied upon.
-Hoss
---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org
RE: Re: Replacing FAST functionality
atsesam.no-ShingleFilter+exactmatching
Posted by Mck <mi...@semb.wever.org>.
Steve,
> Your solution, on the one hand, however, is a kludge: you are
> disabling position information (by assigning the same position to all
> tokens) in order to induce a particular behavior in the query parser,
> which may change in the future.
I disagree.
I'm not disabling position information to induce particular behaviour in
the query parser.
I'm intentionally setting position information to zero as I wish _all_
shingles and unigrams to be synonyms of each other.
The query parser expects you to assign positionIncrement=0 for synonyms
in this manner.
The one kludge i see is that the QueryParser expects the total positions
found to be greater than or equal to one. It might not be intentionally
dealing with the total position count being zero. But the situation
where you have many synonyms is the same as having one token and it
having many synonyms, so positionCount=0 == positionCount=1.
I would think that both should lead to a BooleanQuery being constructed
by the QueryParser. (But the synonyms generated by the ShingleFilter are
in fact phrases so maybe it is wiser to use the MultiPhraseQuery.)
So all in all the QueryParser is behaving exactly as i would expect it
to.
The only logic being induced is setting positionIncrement=0 to indicate
the token is a synonym of the previous token, and this logic is being
completely encapsulated to the ShingleFilter.
~mck
ps i cross-posted as i thought this was better for the dev list but am
not sure.
--
"Enlightenment is your ego's biggest disappointment." Yoginanda
| semb.wever.org | sesat.no | sesam.no |
Re: Replacing FAST functionality
atsesam.no-ShingleFilter+exactmatching
Posted by Mck <mi...@semb.wever.org>.
On Wed, 2008-09-10 at 20:16 +0200, Mck wrote:
> > Anyhow, I suggest you change the name of the option you're adding in
> > LUCENE-1380 to "disablePositions", and make it boolean -- this
> better
> > describes what you're trying to do. When true, all position
> > increments would be set to zero. It should default to false.
>
> disablePositions it is.
You mind if i reverse this to "enablePositions" (that by default is
true)?
I'm not very fond of double-negatives.
~mck
--
"If you are distressed by anything external, the pain is not due to the
thing itself, but to your estimate of it. This you have the power to
revoke." Marcus Aurelius
| semb.wever.org | sesat.no | sesam.no |
RE: Re: Replacing FAST functionality
atsesam.no-ShingleFilter+exactmatching
Posted by Mck <mi...@semb.wever.org>.
> Anyhow, I suggest you change the name of the option you're adding in
> LUCENE-1380 to "disablePositions", and make it boolean -- this better
> describes what you're trying to do. When true, all position
> increments would be set to zero. It should default to false.
disablePositions it is. thanks for helping steve.
~mck
--
"Do not seek to follow in the footsteps of those of old - seek what they
sought." Matsuo Basho
| semb.wever.org | sesat.no | sesam.no |
RE: Re: Replacing FAST functionality
atsesam.no-ShingleFilter+exactmatching
Posted by Mck <mi...@semb.wever.org>.
Steve,
> Your solution, on the one hand, however, is a kludge: you are
> disabling position information (by assigning the same position to all
> tokens) in order to induce a particular behavior in the query parser,
> which may change in the future.
I disagree.
I'm not disabling position information to induce particular behaviour in
the query parser.
I'm intentionally setting position information to zero as I wish _all_
shingles and unigrams to be synonyms of each other.
The query parser expects you to assign positionIncrement=0 for synonyms
in this manner.
The one kludge i see is that the QueryParser expects the total positions
found to be greater than or equal to one. It might not be intentionally
dealing with the total position count being zero. But the situation
where you have many synonyms is the same as having one token and it
having many synonyms, so positionCount=0 == positionCount=1.
I would think that both should lead to a BooleanQuery being constructed
by the QueryParser. (But the synonyms generated by the ShingleFilter are
in fact phrases so maybe it is wiser to use the MultiPhraseQuery.)
So all in all the QueryParser is behaving exactly as i would expect it
to.
The only logic being induced is setting positionIncrement=0 to indicate
the token is a synonym of the previous token, and this logic is being
completely encapsulated to the ShingleFilter.
~mck
ps i cross-posted as i thought this was better for the dev list but am
not sure.
--
"Enlightenment is your ego's biggest disappointment." Yoginanda
| semb.wever.org | sesat.no | sesam.no |
Re: Replacing FAST functionality at
sesam.no-ShingleFilter+exactmatching
Posted by Mck <mi...@semb.wever.org>.
> And if you have configured an analyzer that includes a query-time
> filter, it should be invoked, regardless of whether a phrase query is
> constructed.
sorry steve i failed to explain this so clearly.
Without phrasing the ShingleFilter is indeed invoked.
But it is used three separate times for each term
1) abcd
2) efgh
3) ijkl
So there is no shingles generated.
With phrasing the ShingleFilter it is used once
1) abcd efgh ijkl
And so all the shingles are generated.
I do not know how Solr and Lucene well enough to appreciate how the
query parsing is working together here.
But what i do see, just within
no.apache.jakarta.lucene.queryParser.QueryParser.getFieldQuery(..)
is that there are three possible return values:
BooleanQuery, MultiPhraseQuery, or PhraseQuery.
The remaining alternative is BooleanQuery and that happens when
positionCount (which is the sum of all the tokens' positionIncrements)
equals one. That's even tougher to achieve.
~mck
--
"Driving ambition is the last refuge of the failure." Oscar Wilde
| semb.wever.org | sesat.no | sesam.no |
Re: Replacing FAST functionality at sesam.no
-ShingleFilter+exactmatching
Posted by Mck <mi...@semb.wever.org>.
> > But this does not return the hits i want.
>
> Have you tried submitting the query without quotes? (That's where the
> PhraseQuery likely comes from.)
Yes. It does not work.
It returns just the unigrams, again the same behaviour as mentioned
earlier.
Debugging ShingleFilter in this case it shows that no shingles are ever
constructed. There are 3 separate tokens in the query and that's all.
The ShingleFilter appears to only work, at least for me, on phrases.
I would think this correct as each shingle is in fact a sub-phrase to
the larger original phrase. Is that presumption correct?
~mck
--
"Great spirits have always encountered violent opposition from mediocre
minds. The mediocre mind is incapable of understanding the man who
refuses to bow blindly to conventional prejudices and chooses instead to
express his opinions courageously and honestly." Albert Einstein
| semb.wever.org | sesat.no | sesam.no |
RE: Re: Replacing FAST functionality at sesam.no -
ShingleFilter+exactmatching
Posted by Mck <mi...@semb.wever.org>.
> probably better to change the one instance of .setPositionIncrement(0)
> to .setPositionIncrement(1) - that way, MultiPhraseQuery will not be
> invoked, and the standard disjunction thing should happen.
Tried this.
As you say i end up with instead a
PhraseQuery
terms = { list_entry_shingles:abcd
list_entry_shingles:abcd efgh
list_entry_shingles:abcd efgh ijkl
list_entry_shingles:efgh
list_entry_shingles:efgh ijkl
list_entry_shingles:ijkl }
But this does not return the hits i want.
(It returns one hit if TextField and zero hits if StrField, the same
behaviour i mentioned before).
~mck
--
"Traveller, there are no paths. Paths are made by walking." Australian
Aboriginal saying
| semb.wever.org | sesat.no | sesam.no |
Re: Replacing FAST functionality at sesam.no - ShingleFilter+exact
matching
Posted by Mck <mi...@semb.wever.org>.
> Looks to me like MultiPhraseQuery is getting in the way. Shingles
> that begin at the same word are given the same position by
> ShingleFilter, and Solr's FieldQParserPlugin creates a
> MultiPhraseQuery when it encounters tokens in a query with the same
> position. I think what you want is to convert queries into shingle
> disjunctions (*any* matching shingle results in a hit), right?
Yes you're right Steve. thank you.
One way, i see now, to get the behaviour i want is to set the unigrams'
positionIncrement to zero instead of one.
For example in ShingleFilter.fillOutputBuffer(..) replacing the two
ocurrances of
> .setPositionIncrement(1);
with
> .setPositionIncrement(0);
Then i end up with a MultiPhraseQuery with
termArrays[0] = { list_entry_shingles:abcd
list_entry_shingles:abcd efgh
list_entry_shingles:abcd efgh ijkl
list_entry_shingles:efgh
list_entry_shingles:efgh ijkl
list_entry_shingles:ijkl }
and it works perfectly :-)
I see no way of configuring this behaviour though.
If it is possible and someone can say how this would be a real godsend.
Otherwise would a patch to ShingleFilter that offers an option
"unigramPositionIncrement" (that defaults to 1) likely be accepted into
trunk?
~mck
--
"Between two evils, I always pick the one I never tried before." Mae
West
| semb.wever.org | sesat.no | sesam.no |
Re: Replacing FAST functionality at sesam.no - ShingleFilter+exact
matching
Posted by Mck <mi...@semb.wever.org>.
> Looks to me like MultiPhraseQuery is getting in the way. Shingles
> that begin at the same word are given the same position by
> ShingleFilter, and Solr's FieldQParserPlugin creates a
> MultiPhraseQuery when it encounters tokens in a query with the same
> position. I think what you want is to convert queries into shingle
> disjunctions (*any* matching shingle results in a hit), right?
Yes you're right Steve. thank you.
One way, i see now, to get the behaviour i want is to set the unigrams'
positionIncrement to zero instead of one.
For example in ShingleFilter.fillOutputBuffer(..) replacing the two
ocurrances of
> .setPositionIncrement(1);
with
> .setPositionIncrement(0);
Then i end up with a MultiPhraseQuery with
termArrays[0] = { list_entry_shingles:abcd
list_entry_shingles:abcd efgh
list_entry_shingles:abcd efgh ijkl
list_entry_shingles:efgh
list_entry_shingles:efgh ijkl
list_entry_shingles:ijkl }
and it works perfectly :-)
I see no way of configuring this behaviour though.
If it is possible and someone can say how this would be a real godsend.
Otherwise would a patch to ShingleFilter that offers an option
"unigramPositionIncrement" (that defaults to 1) likely be accepted into
trunk?
~mck
--
"Between two evils, I always pick the one I never tried before." Mae
West
| semb.wever.org | sesat.no | sesam.no |