Solr mlt, ContentStream and mlt.fl

["karel04@gmail.com" <ka...@gmail.com>]

Hi,

I have recently upgraded to latest Solr4, rewrote the config files and
recreated the index.  However now I am having trouble with the MLT
handler and I am not sure whether this is by design or a mistake I am
doing.

If I am using MLT on indexed documents, everything works as expected,
I can set  mlt.fl to multiple fields etc.  But when I use stream.body
instead of an indexed document, I can only use exactly one field for
mlt.fl, otherwise an error is returned.

To sum up:
WORKS: .../solr/core1/select?stream.body=consumer%20electronics&qt=mlt&mlt.fl=title
WORKS: .../solr/core1/select?stream.body=consumer%20electronics&qt=mlt&mlt.fl=body
WORKS: .../solr/core1/select?q=id:12345&qt=mlt&mlt.fl=body
DOES NOT WORK: .../solr/core1/select?stream.body=consumer%20electronics&qt=mlt&mlt.fl=title,body

Searching on the error I found some references to CachingTokenFilter
on lucene mailing lists, but I do not really understand whether this
problem is caused by my misconfiguration or a bug in MLT.


Error output from Solr example with MLT enabled:
java.io.IOException: Stream closed
	at java.io.StringReader.ensureOpen(StringReader.java:56)
	at java.io.StringReader.read(StringReader.java:90)
	at org.apache.solr.schema.FieldType$DefaultAnalyzer$1.incrementToken(FieldType.java:417)
	at org.apache.lucene.search.similar.MoreLikeThis.addTermFrequencies(MoreLikeThis.java:891)
	at org.apache.lucene.search.similar.MoreLikeThis.retrieveTerms(MoreLikeThis.java:963)
	at org.apache.lucene.search.similar.MoreLikeThis.like(MoreLikeThis.java:634)
	at org.apache.solr.handler.MoreLikeThisHandler$MoreLikeThisHelper.getMoreLikeThis(MoreLikeThisHandler.java:370)
	at org.apache.solr.handler.MoreLikeThisHandler.handleRequestBody(MoreLikeThisHandler.java:149)
	at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
	at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316)
	at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:353)
	at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
	at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
	at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
	at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
	at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
	at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
	at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
	at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230)
	at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
	at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
	at org.mortbay.jetty.Server.handle(Server.java:326)
	at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
	at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
	at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
	at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
	at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
	at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
	at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

My actual error output:
Stream closed java.io.IOException: Stream closed at
java.io.StringReader.ensureOpen(StringReader.java:56) at
java.io.StringReader.read(StringReader.java:90) at
org.apache.lucene.analysis.CharReader.read(CharReader.java:54) at
org.apache.lucene.analysis.charfilter.MappingCharFilter.read(MappingCharFilter.java:126)
at org.apache.lucene.analysis.pattern.PatternReplaceCharFilter.read(PatternReplaceCharFilter.java:127)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:803)
at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:1010)
at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:168)
at org.apache.lucene.analysis.util.FilteringTokenFilter.incrementToken(FilteringTokenFilter.java:50)
at org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter.incrementToken(WordDelimiterFilter.java:224)
at org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:54)
at org.apache.lucene.analysis.compound.CompoundWordTokenFilterBase.incrementToken(CompoundWordTokenFilterBase.java:157)
at org.apache.lucene.analysis.snowball.SnowballFilter.incrementToken(SnowballFilter.java:77)
at org.apache.lucene.analysis.miscellaneous.RemoveDuplicatesTokenFilter.incrementToken(RemoveDuplicatesTokenFilter.java:54)
at org.apache.lucene.search.similar.MoreLikeThis.addTermFrequencies(MoreLikeThis.java:891)
at org.apache.lucene.search.similar.MoreLikeThis.retrieveTerms(MoreLikeThis.java:963)
at org.apache.lucene.search.similar.MoreLikeThis.like(MoreLikeThis.java:634)
at org.apache.solr.handler.MoreLikeThisHandler$MoreLikeThisHelper.getMoreLikeThis(MoreLikeThisHandler.java:370)
at org.apache.solr.handler.MoreLikeThisHandler.handleRequestBody(MoreLikeThisHandler.java:149)
at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1404) at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:353)
at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:248)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.RequestFilterValve.process(RequestFilterValve.java:269)
at org.apache.catalina.valves.RemoteAddrValve.invoke(RemoteAddrValve.java:81)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:857)
at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
at java.lang.Thread.run(Thread.java:636)