You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Toke Eskildsen (JIRA)" <ji...@apache.org> on 2015/02/06 20:27:34 UTC
[jira] [Commented] (SOLR-1715) Externally Generated Filters
[ https://issues.apache.org/jira/browse/SOLR-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14309706#comment-14309706 ]
Toke Eskildsen commented on SOLR-1715:
--------------------------------------
We need something like this for defining subsets of our net archive index (billions of documents), so I am considering writing a patch. My idea for a solution is as follows:
In a request, a user can state a key and a URL. These two values are stored for the lifetime of the Solr JVM (or until updated/deleted). When a user requests a filter for the given key, it is evaluated to a filter docset, which is stored until the index is updated.
Evaluation of dhe content from the URL is processed in a streaming manner, without any size limit. Each line is treated as a Solr query and the result ing docIDs are enabled in the docset. Treating each line as a query means that "everything" can be expressed, from corpus spanning {{*:*}} through large {{TermsQueries}} to a billion {{ID:xxxxxx statements}}. The downside is size (and as part of that speed), so other types of streams (bitmap with the content for the docset, list of ID-values, etc) might be worth adding at a later point.
> Externally Generated Filters
> ----------------------------
>
> Key: SOLR-1715
> URL: https://issues.apache.org/jira/browse/SOLR-1715
> Project: Solr
> Issue Type: New Feature
> Components: search
> Reporter: Grant Ingersoll
> Priority: Minor
> Fix For: 4.9, Trunk
>
>
> I'd like to have a way of loading document filters from externally generated sources. For instance, I may have run a clustering task over my corpus and the results may be stored in HDFS (Hadoop). A given cluster will have a centroid and a bunch of points that are in the cluster. The points are labeled by doc id (the Solr doc id) from the index.
> Another example is I may have an external security system that can provide info on which documents are available to a user.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org