You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Andrzej Bialecki (JIRA)" <ji...@apache.org> on 2009/12/08 05:18:18 UTC
[jira] Created: (SOLR-1632) Distributed IDF
Distributed IDF
---------------
Key: SOLR-1632
URL: https://issues.apache.org/jira/browse/SOLR-1632
Project: Solr
Issue Type: New Feature
Components: search
Affects Versions: 1.5
Reporter: Andrzej Bialecki
Distributed IDF is a valuable enhancement for distributed search across non-uniform shards. This issue tracks the proposed implementation of an API to support this functionality in Solr.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1632) Distributed IDF
Posted by "Marc Sturlese (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793283#action_12793283 ]
Marc Sturlese commented on SOLR-1632:
-------------------------------------
Wich should be the value of the parameter shard.purpose to enable or disable the exact version of global IDF?
> Distributed IDF
> ---------------
>
> Key: SOLR-1632
> URL: https://issues.apache.org/jira/browse/SOLR-1632
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.5
> Reporter: Andrzej Bialecki
> Attachments: distrib.patch
>
>
> Distributed IDF is a valuable enhancement for distributed search across non-uniform shards. This issue tracks the proposed implementation of an API to support this functionality in Solr.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1632) Distributed IDF
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789379#action_12789379 ]
Otis Gospodnetic commented on SOLR-1632:
----------------------------------------
I didn't look a the patch, but from your comments it looks like you already have that "1 merged big idf map", which is really what I was aiming at, so that's good!
I was just thinking that this map (file) would be periodically updated and pushed to slaves, so that slaves can compute the global IDF *locally* instead of any kind of extra requests.
> Distributed IDF
> ---------------
>
> Key: SOLR-1632
> URL: https://issues.apache.org/jira/browse/SOLR-1632
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.5
> Reporter: Andrzej Bialecki
> Attachments: distrib.patch
>
>
> Distributed IDF is a valuable enhancement for distributed search across non-uniform shards. This issue tracks the proposed implementation of an API to support this functionality in Solr.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1632) Distributed IDF
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789120#action_12789120 ]
Otis Gospodnetic commented on SOLR-1632:
----------------------------------------
What about this approach: http://markmail.org/message/mjfmpzfspguepixx ?
> Distributed IDF
> ---------------
>
> Key: SOLR-1632
> URL: https://issues.apache.org/jira/browse/SOLR-1632
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.5
> Reporter: Andrzej Bialecki
> Attachments: distrib.patch
>
>
> Distributed IDF is a valuable enhancement for distributed search across non-uniform shards. This issue tracks the proposed implementation of an API to support this functionality in Solr.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1632) Distributed IDF
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated SOLR-1632:
------------------------------------
Attachment: distrib.patch
Initial implementation. This supports the current global IDF (i.e. none ;) ), and an exact version of global IDF that requires one additional request per query to obtain per-shard stats.
The design should be already flexible enough to implement LRU caching of docFreqs, and ultimately to implement other methods for global IDF calculation (e.g. based on estimation or re-ranking).
> Distributed IDF
> ---------------
>
> Key: SOLR-1632
> URL: https://issues.apache.org/jira/browse/SOLR-1632
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.5
> Reporter: Andrzej Bialecki
> Attachments: distrib.patch
>
>
> Distributed IDF is a valuable enhancement for distributed search across non-uniform shards. This issue tracks the proposed implementation of an API to support this functionality in Solr.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1632) Distributed IDF
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789174#action_12789174 ]
Andrzej Bialecki commented on SOLR-1632:
-----------------------------------------
I'm not sure what approach you are referring to. Following the terminology in that thread, this implementation follows the approach where there is a single merged big idf map at the master, and it's sent out to slaves on each query. However, when exactly this merging and sending happens is implementation-specific - in the ExactDFSource it happens on every query, but I hope the API can support other scenarios as well.
> Distributed IDF
> ---------------
>
> Key: SOLR-1632
> URL: https://issues.apache.org/jira/browse/SOLR-1632
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.5
> Reporter: Andrzej Bialecki
> Attachments: distrib.patch
>
>
> Distributed IDF is a valuable enhancement for distributed search across non-uniform shards. This issue tracks the proposed implementation of an API to support this functionality in Solr.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1632) Distributed IDF
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12793295#action_12793295 ]
Andrzej Bialecki commented on SOLR-1632:
-----------------------------------------
Shard.purpose is set by a concrete implementation of the DFSource, so I guess your question is "how to turn ExactDFSource on/off"? If that's the case, then put this in your solrconfig.xml:
{code:xml}
<globalIDF class="org.apache.solr.search.ExactDFCache"/>
{code}
> Distributed IDF
> ---------------
>
> Key: SOLR-1632
> URL: https://issues.apache.org/jira/browse/SOLR-1632
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.5
> Reporter: Andrzej Bialecki
> Attachments: distrib.patch
>
>
> Distributed IDF is a valuable enhancement for distributed search across non-uniform shards. This issue tracks the proposed implementation of an API to support this functionality in Solr.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1632) Distributed IDF
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrzej Bialecki updated SOLR-1632:
------------------------------------
Attachment: distrib-2.patch
Updated patch, contains also:
* LRU-based cache that optimizes requests using cached values of docFreq for known terms
* unit tests
> Distributed IDF
> ---------------
>
> Key: SOLR-1632
> URL: https://issues.apache.org/jira/browse/SOLR-1632
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.5
> Reporter: Andrzej Bialecki
> Attachments: distrib-2.patch, distrib.patch
>
>
> Distributed IDF is a valuable enhancement for distributed search across non-uniform shards. This issue tracks the proposed implementation of an API to support this functionality in Solr.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1632) Distributed IDF
Posted by "Andrzej Bialecki (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12789607#action_12789607 ]
Andrzej Bialecki commented on SOLR-1632:
-----------------------------------------
I believe the API that I propose would support such implementation as well. Please note that it's usually not feasible to compute and distribute the complete IDF table for all terms - you would have to replicate a union of all term dictionaries across the cluster. In practice, you limit the amount of information by various means, e.g. only distributing data related to the current request (this implementation) or reducing the frequency of updates (e.g. LRU caching), or approximating global DF with a constant for frequent terms (where the contribution of their IDF to the score would be negligible anyway).
> Distributed IDF
> ---------------
>
> Key: SOLR-1632
> URL: https://issues.apache.org/jira/browse/SOLR-1632
> Project: Solr
> Issue Type: New Feature
> Components: search
> Affects Versions: 1.5
> Reporter: Andrzej Bialecki
> Attachments: distrib.patch
>
>
> Distributed IDF is a valuable enhancement for distributed search across non-uniform shards. This issue tracks the proposed implementation of an API to support this functionality in Solr.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.