You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Max <na...@gmail.com> on 2011/12/12 11:18:23 UTC
InvalidTokenOffsetsException in conjunction with highlighting and ICU
folding and edgeNgrams
Hi there,
when highlighting a field with this definition:
<fieldType name="name" class="solr.TextField"
positionIncrementGap="100">
<analyzer type="index">
<charFilter class="solr.HTMLStripCharFilterFactory"/>
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.ICUTransformFilterFactory" id="Any-Latin"/>
<filter class="solr.ICUFoldingFilterFactory"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1"
generateNumberParts="1"
catenateWords="1"
catenateNumbers="1"
catenateAll="0"
splitOnCaseChange="1"/>
<filter class="solr.EdgeNGramFilterFactory"
minGramSize="2" maxGramSize="15" side="front"/>
</analyzer>
<analyzer type="query">
<tokenizer class="solr.WhitespaceTokenizerFactory"/>
<filter class="solr.ICUTransformFilterFactory" id="Any-Latin"/>
<filter class="solr.ICUFoldingFilterFactory"/>
<filter class="solr.WordDelimiterFilterFactory"
generateWordParts="1"
generateNumberParts="1"
catenateWords="1"
catenateNumbers="1"
catenateAll="0"
splitOnCaseChange="1"/>
<filter class="solr.EdgeNGramFilterFactory"
minGramSize="2" maxGramSize="15" side="front"/>
</analyzer>
</fieldType>
containing this string:
"Mosfellsbær"
I get the following exception, if that field is in the highlight fields:
SEVERE: org.apache.solr.common.SolrException:
org.apache.lucene.search.highlight.InvalidTokenOffsetsException: Token
mosfellsbaer exceeds length of provided text sized 11
at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:497)
at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:401)
at org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:131)
at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:194)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1372)
at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:636)
Caused by: org.apache.lucene.search.highlight.InvalidTokenOffsetsException:
Token mosfellsbaer exceeds length of provided text sized 11
at org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:233)
at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:490)
I tried with solr 3.4 and 3.5, same error for both. Removing the char
filter didnt fix the problem either.
It seems like there is some weird stuff going on when folding the
string, it can be seen in the analysis view, too:
http://i.imgur.com/6B2Uh.png
The end offset remains 11 even after folding and transforming "æ" to
"ae", which seems wrong to me.
I also stumbled upon https://issues.apache.org/jira/browse/LUCENE-1500
which seems like a similiar issue.
Is there a workaround for that problem or is the field configuration wrong?
Re: InvalidTokenOffsetsException in conjunction with highlighting and
ICU folding and edgeNgrams
Posted by Robert Muir <rc...@gmail.com>.
On Mon, Dec 12, 2011 at 5:18 AM, Max <na...@gmail.com> wrote:
> The end offset remains 11 even after folding and transforming "æ" to
> "ae", which seems wrong to me.
End offsets refer to the *original text* so this is correct.
What is wrong, is EdgeNGramsFilter. See how it turns that 11 to a 12?
>
> I also stumbled upon https://issues.apache.org/jira/browse/LUCENE-1500
> which seems like a similiar issue.
>
> Is there a workaround for that problem or is the field configuration wrong?
For now, don't use EdgeNGrams.
--
lucidimagination.com
Re: InvalidTokenOffsetsException in conjunction with highlighting and
ICU folding and edgeNgrams
Posted by Max <na...@gmail.com>.
Robert, thank you for creating the issue in JIRA.
However, I need ngrams on that field – is there an alternative to the
EdgeNGramFilterFactory ?
Thanks!
On Mon, Dec 12, 2011 at 1:25 PM, Robert Muir <rc...@gmail.com> wrote:
> On Mon, Dec 12, 2011 at 5:18 AM, Max <na...@gmail.com> wrote:
>
>> It seems like there is some weird stuff going on when folding the
>> string, it can be seen in the analysis view, too:
>>
>> http://i.imgur.com/6B2Uh.png
>>
>
> I created a bug here, https://issues.apache.org/jira/browse/LUCENE-3642
>
> Thanks for the screenshot, makes it easy to do a test case here.
>
> --
> lucidimagination.com
Re: InvalidTokenOffsetsException in conjunction with highlighting and
ICU folding and edgeNgrams
Posted by Robert Muir <rc...@gmail.com>.
On Mon, Dec 12, 2011 at 5:18 AM, Max <na...@gmail.com> wrote:
> It seems like there is some weird stuff going on when folding the
> string, it can be seen in the analysis view, too:
>
> http://i.imgur.com/6B2Uh.png
>
I created a bug here, https://issues.apache.org/jira/browse/LUCENE-3642
Thanks for the screenshot, makes it easy to do a test case here.
--
lucidimagination.com