You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Trey Hyde <th...@centraldesktop.com> on 2010/10/12 20:41:58 UTC
useFastVectorHighlighter creates fragments with cut off terms, incomplete
The natives have been getting restless since I turned off the highlighter on our application quite a few months ago. In an attempt to rectify that I've migrated to Solr 3.1 (found it quite stable so far) in hopes to use the vector highlighter. So far, I'm not seeing a significant performance hit in the highlighter. I was spending many seconds highlighting results in 1.4 (1.3 way considerably faster) ... so big win there.
But ... the fragments often cut off words (preceding non-highlighted words).
Title field of "Apache MQ Message Queue Setup Notes (Dev)" is getting highlighted as "he MQ <em>Message Queue</em> Setup Notes (Dev)"
Some seem to highlight fine ... some not. No common thread that I can see.
From http://wiki.apache.org/solr/HighlightingParameters
hl.fragsize
"The size, in characters, of fragments to consider for highlighting. "0" indicates that the whole field value should be used (no fragmenting). This parameter accepts per-field overrides."
I thought that might help the situation but that results in
SEVERE: java.lang.IllegalArgumentException: fragCharSize(0) is too small. It must be 18 or higher.
at org.apache.lucene.search.vectorhighlight.SimpleFragListBuilder.createFieldFragList(SimpleFragListBuilder.java:36)
at org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getFieldFragList(FastVectorHighlighter.java:167)
at org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getBestFragments(FastVectorHighlighter.java:158)
at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByFastVectorHighlighter(DefaultSolrHighlighter.java:506)
at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:379)
...
I also set the value to a value larger than the possible size of the field but I still get a left truncated highlight in many cases.
hl.fragListBuilder and hl.fragmentsBuilder sound like they may be relevant but I haven't been able to locate any documentation on those.
This is my highlighter set up for the time being.
<str name="hl">true</str>
<str name="hl.useFastVectorHighlighter">true</str>
<str name="hl.snippets">3</str>
<str name="f.title.hl.fragsize">10000</str>
<str name="hl.fragsize">200</str>
<str name="hl.mergeContiguous">true</str>
<str name="hl.simple.pre"><b></str>
<str name="hl.simple.post"></b></str>
Any suggestions? Thanks.
I'm running revision 1021880 in the lusolr 3_1 branch.
Trey Hyde
thyde@centraldesktop.com
Central Desktop, Inc.
Organize, Share, Collaborate
Re: useFastVectorHighlighter creates fragments with cut off terms, incomplete
Posted by Trey Hyde <th...@centraldesktop.com>.
That must be it, thanks for the find. The patch must not have made it into 3.1 yet. I'll take a look at the diff and see what I can make work for our app in the meantime.
On Oct 12, 2010, at 12:01 PM, Markus Jelsma wrote:
> I believe it was this one
> https://issues.apache.org/jira/browse/LUCENE-1824
>
>> The natives have been getting restless since I turned off the highlighter
>> on our application quite a few months ago. In an attempt to rectify
>> that I've migrated to Solr 3.1 (found it quite stable so far) in hopes to
>> use the vector highlighter. So far, I'm not seeing a significant
>> performance hit in the highlighter. I was spending many seconds
>> highlighting results in 1.4 (1.3 way considerably faster) ... so big win
>> there.
>>
>>
>> But ... the fragments often cut off words (preceding non-highlighted
>> words).
>>
>> Title field of "Apache MQ Message Queue Setup Notes (Dev)" is getting
>> highlighted as "he MQ <em>Message Queue</em> Setup Notes (Dev)" Some seem
>> to highlight fine ... some not. No common thread that I can see.
>>
>> From http://wiki.apache.org/solr/HighlightingParameters
>> hl.fragsize
>>
>> "The size, in characters, of fragments to consider for highlighting. "0"
>> indicates that the whole field value should be used (no fragmenting). This
>> parameter accepts per-field overrides."
>>
>> I thought that might help the situation but that results in
>> SEVERE: java.lang.IllegalArgumentException: fragCharSize(0) is too small.
>> It must be 18 or higher. at
>> org.apache.lucene.search.vectorhighlight.SimpleFragListBuilder.createField
>> FragList(SimpleFragListBuilder.java:36) at
>> org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getFieldFra
>> gList(FastVectorHighlighter.java:167) at
>> org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getBestFrag
>> ments(FastVectorHighlighter.java:158) at
>> org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByFastVecto
>> rHighlighter(DefaultSolrHighlighter.java:506) at
>> org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSol
>> rHighlighter.java:379) ...
>>
>> I also set the value to a value larger than the possible size of the field
>> but I still get a left truncated highlight in many cases.
>>
>>
>> hl.fragListBuilder and hl.fragmentsBuilder sound like they may be relevant
>> but I haven't been able to locate any documentation on those.
>>
>> This is my highlighter set up for the time being.
>>
>> <str name="hl">true</str>
>> <str name="hl.useFastVectorHighlighter">true</str>
>> <str name="hl.snippets">3</str>
>> <str name="f.title.hl.fragsize">10000</str>
>> <str name="hl.fragsize">200</str>
>> <str name="hl.mergeContiguous">true</str>
>> <str name="hl.simple.pre"><b></str>
>> <str name="hl.simple.post"></b></str>
>>
>>
>> Any suggestions? Thanks.
>> I'm running revision 1021880 in the lusolr 3_1 branch.
>>
>>
>> Trey Hyde
>> thyde@centraldesktop.com
>> Central Desktop, Inc.
>> Organize, Share, Collaborate
Trey Hyde
thyde@centraldesktop.com
Central Desktop, Inc.
Organize, Share, Collaborate
Re: useFastVectorHighlighter creates fragments with cut off terms, incomplete
Posted by Markus Jelsma <ma...@openindex.io>.
I believe it was this one
https://issues.apache.org/jira/browse/LUCENE-1824
> The natives have been getting restless since I turned off the highlighter
> on our application quite a few months ago. In an attempt to rectify
> that I've migrated to Solr 3.1 (found it quite stable so far) in hopes to
> use the vector highlighter. So far, I'm not seeing a significant
> performance hit in the highlighter. I was spending many seconds
> highlighting results in 1.4 (1.3 way considerably faster) ... so big win
> there.
>
>
> But ... the fragments often cut off words (preceding non-highlighted
> words).
>
> Title field of "Apache MQ Message Queue Setup Notes (Dev)" is getting
> highlighted as "he MQ <em>Message Queue</em> Setup Notes (Dev)" Some seem
> to highlight fine ... some not. No common thread that I can see.
>
> From http://wiki.apache.org/solr/HighlightingParameters
> hl.fragsize
>
> "The size, in characters, of fragments to consider for highlighting. "0"
> indicates that the whole field value should be used (no fragmenting). This
> parameter accepts per-field overrides."
>
> I thought that might help the situation but that results in
> SEVERE: java.lang.IllegalArgumentException: fragCharSize(0) is too small.
> It must be 18 or higher. at
> org.apache.lucene.search.vectorhighlight.SimpleFragListBuilder.createField
> FragList(SimpleFragListBuilder.java:36) at
> org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getFieldFra
> gList(FastVectorHighlighter.java:167) at
> org.apache.lucene.search.vectorhighlight.FastVectorHighlighter.getBestFrag
> ments(FastVectorHighlighter.java:158) at
> org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByFastVecto
> rHighlighter(DefaultSolrHighlighter.java:506) at
> org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSol
> rHighlighter.java:379) ...
>
> I also set the value to a value larger than the possible size of the field
> but I still get a left truncated highlight in many cases.
>
>
> hl.fragListBuilder and hl.fragmentsBuilder sound like they may be relevant
> but I haven't been able to locate any documentation on those.
>
> This is my highlighter set up for the time being.
>
> <str name="hl">true</str>
> <str name="hl.useFastVectorHighlighter">true</str>
> <str name="hl.snippets">3</str>
> <str name="f.title.hl.fragsize">10000</str>
> <str name="hl.fragsize">200</str>
> <str name="hl.mergeContiguous">true</str>
> <str name="hl.simple.pre"><b></str>
> <str name="hl.simple.post"></b></str>
>
>
> Any suggestions? Thanks.
> I'm running revision 1021880 in the lusolr 3_1 branch.
>
>
> Trey Hyde
> thyde@centraldesktop.com
> Central Desktop, Inc.
> Organize, Share, Collaborate