token exceeding provided text size error since Solr 3.2

[getagrip <ge...@web.de>]

A bug was introduced between Solr 3.1 and 3.2.

With Solr 3.2 we are now getting the follwing error when querying 
several pdf and word documents:

SEVERE: org.apache.solr.common.SolrException: 
org.apache.lucene.search.highlight.InvalidTokenOffsetsException: Token 
17 exceeds length of provided text sized 168
         at 
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:474)
         at 
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:378)
         at 
org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:116)
         at 
org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:194)
         at 
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1360)
         at 
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:356)
         at 
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:252)
         at 
org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
         at 
org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
         at 
org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
         at 
org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
         at 
org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
         at 
org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
         at 
org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
         at 
org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
         at 
org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
         at org.mortbay.jetty.Server.handle(Server.java:326)
         at 
org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
         at 
org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
         at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
         at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
         at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
         at 
org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
         at 
org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
Caused by: 
org.apache.lucene.search.highlight.InvalidTokenOffsetsException: Token 
17 exceeds length of provided text sized 168
         at 
org.apache.lucene.search.highlight.Highlighter.getBestTextFragments(Highlighter.java:233)
         at 
org.apache.solr.highlight.DefaultSolrHighlighter.doHighlightingByHighlighter(DefaultSolrHighlighter.java:467)
         ... 24 more