You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Chris Harris <ry...@gmail.com> on 2009/01/13 01:29:43 UTC
Highlighting Trouble With Bigram Shingle Index
I'm running into some highlighting issues that appear to arise only
when I'm using a bigram shingle (ShingleFilterFactory) analyzer.
I started with a bigram-free situation along these lines:
<field name="body" type="noshingleText" indexed="false" stored="false" />
<!-- Stored text for use with highlighting: -->
<field name="kwic" type="noshingleText" indexed="false"
stored="true" compressed="true" multiValued="false" />
<copyField source="body" dest="kwic" maxLength="100000" />
<fieldType name="noshingleText" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
</analyzer>
</fieldType>
For performance reasons, though, I wanted to turn on bigram shingle
indexing on the body field. (For more information see
http://www.nabble.com/Using-Shingles-to-Increase-Phrase-Search-Performance-td19015758.html#a19015758)
In particular, I wanted to use this field type:
<fieldType name="shingleText" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ShingleFilterFactory" outputUnigrams="true" />
</analyzer>
<analyzer type="query">
<tokenizer class="solr.StandardTokenizerFactory"/>
<filter class="solr.StandardFilterFactory"/>
<filter class="solr.LowerCaseFilterFactory"/>
<filter class="solr.ShingleFilterFactory" outputUnigrams="false"
outputUnigramIfNoNgram="true" />
</analyzer>
</fieldType>
(Regarding outputUnigramsIfNoNgram parameter, see
http://issues.apache.org/jira/browse/SOLR-744.)
I wasn't sure if I should want to define my kwic field (the one I use
for highlighting) as type shingleText, to match the body field, or
type noshingleText. So I tried both. Neither work quite as desired.
[kwic as type shingleText]
If I have both body and kwic as type shingleText, then highlighting
more or less works, but there are some anomolies. The main thing is
that it really likes to pick fragments where the highlighted term
(e.g. "car") is the last term in the fragment:
... la la la la la <em>car</em> ...
... foo foo foo foo foo <em>car</em> ...
This should obviously happen some of the time, but this is happening
with like 95% of my fragments, which is statistically unexpected. And
unfortunate. And it doesn't happen if I turn of shingling.
Another issue is that, if there are two instances of a highlighted
term within a given fragment, it will often highlight not just those
instances, but all the terms in between, like this:
... boo boo bar <em>car la la la car</em> bar bar bar ...
This too doesn't seem to happen if I disable bigram indexing.
I haven't figured out why this is the case. One potential issue is that
the TokenGroup abstraction doesn't necessarily make sense if you have
a token stream of alternating unigrams and bigrams like this:
the, the cat, cat, cat went, went, went for, for, ...
Even if you could have a TokenGroup abstraction that makes sense, the current
implementation of TokenGroup.isDistinct looks like this:
return token.startOffset()>=endOffset
and it turns false most of the time in this case. (I can give some
explanation of why, but maybe I'll save that for later.)
I'm not sure if the highlighter can easily be made to accomodate
sequences of alternating unigrams and bigrams, or if highlighting
should really only be attempted on bigram-free token streams.
[kwic with type noshingleText]
If I set kwic to be of type noshingleText, then the above symptoms go
away. Some things are not quite right, though. The particular symptom
now is that if I do a quoted query like
"big dog"
then the correct results get returned, but no preview fragments are returned.
The underlying reason this happens is that an inappropriate Query
object is being passed
to the constructor for QueryScorer. The query that gets passed is
TermQuery:"big dog"
That is the Query that should be used for *searching* on my bigram body
field, but it's *not* the Query that should be used for *highlighting*; the
Query that should be used for highlighting is something like
PhraseQuery:"big dog"~0
What apparently is going on is that the highlighter is using the Query
object generated by the the *search* component to do highlighting.
One possibility is that the highlighter should
instead create a separate Query object for each hl.fl parameter; each
one would use the analyzer particular to the given *highlighting* field,
rather than the one for the default search field. There might be reasons why
that would be crazy, though.
Sorry this post is a little half-baked, but I'd really like to hear if
anyone has any ideas about how I might proceed with debugging.
Chris
Re: Highlighting Trouble With Bigram Shingle Index
Posted by Koji Sekiguchi <ko...@r.email.ne.jp>.
Chris,
Sorry about avoiding shingle part but:
> ... boo boo bar <em>car la la la car</em> bar bar bar ...
>
> This too doesn't seem to happen if I disable bigram indexing.
I've seen same thing with bigram tokens (not shingle) and reported it:
https://issues.apache.org/jira/browse/LUCENE-1489
Then I wrote a highlighter:
https://issues.apache.org/jira/browse/LUCENE-1522
I'm not shingle expert, but my highlighter has some pluggable interfaces,
a plugin may be written to solve your problem.
regards,
Koji
Chris Harris wrote:
> I'm running into some highlighting issues that appear to arise only
> when I'm using a bigram shingle (ShingleFilterFactory) analyzer.
>
> I started with a bigram-free situation along these lines:
>
> <field name="body" type="noshingleText" indexed="false" stored="false" />
> <!-- Stored text for use with highlighting: -->
> <field name="kwic" type="noshingleText" indexed="false"
> stored="true" compressed="true" multiValued="false" />
> <copyField source="body" dest="kwic" maxLength="100000" />
>
> <fieldType name="noshingleText" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.StandardFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.StandardFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> </analyzer>
> </fieldType>
>
> For performance reasons, though, I wanted to turn on bigram shingle
> indexing on the body field. (For more information see
> http://www.nabble.com/Using-Shingles-to-Increase-Phrase-Search-Performance-td19015758.html#a19015758)
> In particular, I wanted to use this field type:
>
> <fieldType name="shingleText" class="solr.TextField"
> positionIncrementGap="100">
> <analyzer type="index">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.StandardFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ShingleFilterFactory" outputUnigrams="true" />
> </analyzer>
> <analyzer type="query">
> <tokenizer class="solr.StandardTokenizerFactory"/>
> <filter class="solr.StandardFilterFactory"/>
> <filter class="solr.LowerCaseFilterFactory"/>
> <filter class="solr.ShingleFilterFactory" outputUnigrams="false"
> outputUnigramIfNoNgram="true" />
> </analyzer>
> </fieldType>
>
> (Regarding outputUnigramsIfNoNgram parameter, see
> http://issues.apache.org/jira/browse/SOLR-744.)
>
> I wasn't sure if I should want to define my kwic field (the one I use
> for highlighting) as type shingleText, to match the body field, or
> type noshingleText. So I tried both. Neither work quite as desired.
>
> [kwic as type shingleText]
>
> If I have both body and kwic as type shingleText, then highlighting
> more or less works, but there are some anomolies. The main thing is
> that it really likes to pick fragments where the highlighted term
> (e.g. "car") is the last term in the fragment:
>
> ... la la la la la <em>car</em> ...
> ... foo foo foo foo foo <em>car</em> ...
>
> This should obviously happen some of the time, but this is happening
> with like 95% of my fragments, which is statistically unexpected. And
> unfortunate. And it doesn't happen if I turn of shingling.
>
> Another issue is that, if there are two instances of a highlighted
> term within a given fragment, it will often highlight not just those
> instances, but all the terms in between, like this:
>
> ... boo boo bar <em>car la la la car</em> bar bar bar ...
>
> This too doesn't seem to happen if I disable bigram indexing.
>
> I haven't figured out why this is the case. One potential issue is that
> the TokenGroup abstraction doesn't necessarily make sense if you have
> a token stream of alternating unigrams and bigrams like this:
>
> the, the cat, cat, cat went, went, went for, for, ...
>
> Even if you could have a TokenGroup abstraction that makes sense, the current
> implementation of TokenGroup.isDistinct looks like this:
>
> return token.startOffset()>=endOffset
>
> and it turns false most of the time in this case. (I can give some
> explanation of why, but maybe I'll save that for later.)
>
> I'm not sure if the highlighter can easily be made to accomodate
> sequences of alternating unigrams and bigrams, or if highlighting
> should really only be attempted on bigram-free token streams.
>
> [kwic with type noshingleText]
>
> If I set kwic to be of type noshingleText, then the above symptoms go
> away. Some things are not quite right, though. The particular symptom
> now is that if I do a quoted query like
>
> "big dog"
>
> then the correct results get returned, but no preview fragments are returned.
>
> The underlying reason this happens is that an inappropriate Query
> object is being passed
> to the constructor for QueryScorer. The query that gets passed is
>
> TermQuery:"big dog"
>
> That is the Query that should be used for *searching* on my bigram body
> field, but it's *not* the Query that should be used for *highlighting*; the
> Query that should be used for highlighting is something like
>
> PhraseQuery:"big dog"~0
>
> What apparently is going on is that the highlighter is using the Query
> object generated by the the *search* component to do highlighting.
> One possibility is that the highlighter should
> instead create a separate Query object for each hl.fl parameter; each
> one would use the analyzer particular to the given *highlighting* field,
> rather than the one for the default search field. There might be reasons why
> that would be crazy, though.
>
> Sorry this post is a little half-baked, but I'd really like to hear if
> anyone has any ideas about how I might proceed with debugging.
>
> Chris
>
>