[jira] Updated: (SOLR-769) Support Document and Search Result clustering

["Brad Giaccio (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SOLR-769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Brad Giaccio updated SOLR-769:
------------------------------

    Attachment: clustering-componet-shard.patch

Okay I've rewritten the patch, as I suggested.  Now the clustering happens in finishStage for distributed queries and it happens in process for non-distributed  both by calling the new method clusterResults .   To make this happen I had to convert the interfaces and supporting code to use SolrDocumentList rather than DocList.

I've added a unit test which extends TestDistributedSearch,  I had to modify TestDistributedSearch and make a bunch of things protected.   This allowed me to write a very small test case (just had to override doTest)  and leave all the logic for creating shards, distributing docs, and comparing responses in TestDistributedSearch.  I felt this made for a very clean way to test a single distributed component.

> Support Document and Search Result clustering
> ---------------------------------------------
>
>                 Key: SOLR-769
>                 URL: https://issues.apache.org/jira/browse/SOLR-769
>             Project: Solr
>          Issue Type: New Feature
>            Reporter: Grant Ingersoll
>            Assignee: Grant Ingersoll
>            Priority: Minor
>             Fix For: 1.4
>
>         Attachments: clustering-componet-shard.patch, clustering-libs.tar, clustering-libs.tar, SOLR-769-analyzerClass.patch, SOLR-769-lib.zip, SOLR-769.patch, SOLR-769.patch, SOLR-769.patch, SOLR-769.patch, SOLR-769.patch, SOLR-769.patch, SOLR-769.patch, SOLR-769.patch, SOLR-769.patch, SOLR-769.patch, SOLR-769.patch, SOLR-769.tar, SOLR-769.zip
>
>
> Clustering is a useful tool for working with documents and search results, similar to the notion of dynamic faceting.  Carrot2 (http://project.carrot2.org/) is a nice, BSD-licensed, library for doing search results clustering.  Mahout (http://lucene.apache.org/mahout) is well suited for whole-corpus clustering.  
> The patch I lays out a contrib module that starts off w/ an integration of a SearchComponent for doing clustering and an implementation using Carrot.  In search results mode, it will use the DocList as the input for the cluster.   While Carrot2 comes w/ a Solr input component, it is not the same as the SearchComponent that I have in that the Carrot example actually submits a query to Solr, whereas my SearchComponent is just chained into the Component list and uses the ResponseBuilder to add in the cluster results.
> While not fully fleshed out yet, the collection based mode will take in a list of ids or just use the whole collection and will produce clusters.  Since this is a longer, typically offline task, there will need to be some type of storage mechanism (and replication??????) for the clusters.  I _may_ push this off to a separate JIRA issue, but I at least want to present the use case as part of the design of this component/contrib.  It may even make sense that we split this out, such that the building piece is something like an UpdateProcessor and then the SearchComponent just acts as a lookup mechanism.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.