Solr MLT with stream.body returns different results on each shard

[Aaron Gibbons <ag...@synergydatasystems.com>]

I have a fresh install of Solr 5.2.1 with about 3 million docs freshly
indexed (I can also reproduce this issue on 4.10.0). When I use the Solr
MorelikeThisHandler with content stream I'm getting different results per
shard.

I also looked at using a standard MLT query, but I need to be able to
stream in a fairly large block of text for comparison that is not in the
index (different type of document). A standard MLT  query
http://testsolr2:8983/solr/mega/select?q=electronics&mlt.flt=text&mlt.mintf=0&fl=id,score
appears to return consistent results between shards.

Any reason why the content stream query would be different between shards?
Thank you for your help!
Aaron


*Content Stream Example:*
http://testsolr1:8983/solr/mega/mlt?stream.body=electronics&mlt.flt=text&mlt.mintf=0&fl=id,score
*Returns: *
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">3</int>
</lst>
<result name="response" numFound="1590" start="0">

http://testsolr2:8983/solr/mega/mlt?stream.body=electronics&mlt.flt=text&mlt.mintf=0&fl=id,score

*Returns: *
<response>
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">1</int>
</lst>
<result name="response" numFound="1619" start="0">

Re: Solr MLT with stream.body returns different results on each shard

[Chris Hostetter <ho...@fucit.org>]

: I have a fresh install of Solr 5.2.1 with about 3 million docs freshly
: indexed (I can also reproduce this issue on 4.10.0). When I use the Solr
: MorelikeThisHandler with content stream I'm getting different results per
: shard.

I haven't looked at the code recently but i'm 99% certain that the MLT 
handler in general doesn't work with distributed (ie: sharded) queries.  
(unlike the MLT component and the recently added MLT qparser)

I suspect that in the specific case of stream.body, what you are seeing is 
that the interesting terms are being computed relative the local tf/idf 
stats for that shard, and then only local results from that shard are 
being returned.

: I also looked at using a standard MLT query, but I need to be able to
: stream in a fairly large block of text for comparison that is not in the
: index (different type of document). A standard MLT  query

Until/unless the MLT parser supports arbitrary text (there's some mention 
of this in SOLR-7639 but i'm not sure what the status of that is) you 
might find that just POSTing all of your text as a regular query (q) using 
dismax or edismax is suitable for your needs -- that's essentially the 
equivilent of what MLTHandler does with a stream.body, except it tries to 
only focus on "interesting terms" based on tf/idf, but if your fields 
are all configured with stopword files anyway, then the results and 
performance may be similar.


-Hoss
http://www.lucidworks.com/