You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Jason Rennie (JIRA)" <ji...@apache.org> on 2008/09/30 14:26:44 UTC
[jira] Created: (SOLR-795) spellcheck: buildOnOptimize
spellcheck: buildOnOptimize
---------------------------
Key: SOLR-795
URL: https://issues.apache.org/jira/browse/SOLR-795
Project: Solr
Issue Type: New Feature
Components: spellchecker
Affects Versions: 1.3
Reporter: Jason Rennie
I see that there's an option to automatically rebuild the spelling index on a commit. That's a nice feature that we'll consider using, but we run commits every few thousand document updates, which would yield ~100 spelling index rebuilds a day. OTOH, we run an optimize about once/day which seems like a more appropriate schedule for rebuilding the spelling index.
Is there or could there be an option to rebuild the spelling index on optimize?
Grant:
Seems reasonable, could almost do it via the postOptimize call back already in the config, except the SpellCheckComponent's EvenListener is private static and has an empty postCommit implementation (which is what is called after optimization, since it is just like a commit in many ways)
Thus, a patch would be needed.
Shalin:
postCommit/postOptimize callbacks happen after commit/optimize but before a
new searcher is opened. Therefore, it is not possible to re-build spellcheck
index on those events without opening a IndexReader directly on the solr
index. That is why the event listener in SpellCheckComponent uses the
newSearcher listener to build on commits.
I don't think there is anything in the API currently to do what Jason wants.
Hoss:
FWIW: I believe it has to work that way because postCommit events might
modify the index. (but i'm just guessing)
couldn't the Listener's newSearcher() method just do something like
this...
if (rebuildOnlyAfterOptimize &&
! (newSearcher.getReader().isOptimized() &&
! oldSearcher.getReader().isOptimized()) {
return;
} else {
// current impl
}
...assuming a new "rebuildOnlyAfterOptimize" option was added?
Grant:
That seems reasonable.
Another thing to think about, is maybe it is useful to provide some event metadata to the events that contain information about what triggered them. Something like a SolrEvent class such that postCommit looks like
postCommit(SolrEvent evt)
and
public void newSearcher(SolrEvent evt, SolrIndexSearcher newSearcher, SolrIndexSearcher currentSearcher);
Of course, since SolrEventListener is an interface...
Shalin:
Yup, that will work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (SOLR-795) spellcheck: buildOnOptimize
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar reassigned SOLR-795:
------------------------------------------
Assignee: Shalin Shekhar Mangar
> spellcheck: buildOnOptimize
> ---------------------------
>
> Key: SOLR-795
> URL: https://issues.apache.org/jira/browse/SOLR-795
> Project: Solr
> Issue Type: New Feature
> Components: spellchecker
> Affects Versions: 1.3
> Reporter: Jason Rennie
> Assignee: Shalin Shekhar Mangar
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> I see that there's an option to automatically rebuild the spelling index on a commit. That's a nice feature that we'll consider using, but we run commits every few thousand document updates, which would yield ~100 spelling index rebuilds a day. OTOH, we run an optimize about once/day which seems like a more appropriate schedule for rebuilding the spelling index.
> Is there or could there be an option to rebuild the spelling index on optimize?
> Grant:
> Seems reasonable, could almost do it via the postOptimize call back already in the config, except the SpellCheckComponent's EvenListener is private static and has an empty postCommit implementation (which is what is called after optimization, since it is just like a commit in many ways)
> Thus, a patch would be needed.
> Shalin:
> postCommit/postOptimize callbacks happen after commit/optimize but before a
> new searcher is opened. Therefore, it is not possible to re-build spellcheck
> index on those events without opening a IndexReader directly on the solr
> index. That is why the event listener in SpellCheckComponent uses the
> newSearcher listener to build on commits.
> I don't think there is anything in the API currently to do what Jason wants.
> Hoss:
> FWIW: I believe it has to work that way because postCommit events might
> modify the index. (but i'm just guessing)
> couldn't the Listener's newSearcher() method just do something like
> this...
> if (rebuildOnlyAfterOptimize &&
> ! (newSearcher.getReader().isOptimized() &&
> ! oldSearcher.getReader().isOptimized()) {
> return;
> } else {
> // current impl
> }
> ...assuming a new "rebuildOnlyAfterOptimize" option was added?
> Grant:
> That seems reasonable.
> Another thing to think about, is maybe it is useful to provide some event metadata to the events that contain information about what triggered them. Something like a SolrEvent class such that postCommit looks like
> postCommit(SolrEvent evt)
> and
> public void newSearcher(SolrEvent evt, SolrIndexSearcher newSearcher, SolrIndexSearcher currentSearcher);
> Of course, since SolrEventListener is an interface...
> Shalin:
> Yup, that will work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-795) spellcheck: buildOnOptimize
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar updated SOLR-795:
---------------------------------------
Fix Version/s: 1.4
Marking for 1.4
> spellcheck: buildOnOptimize
> ---------------------------
>
> Key: SOLR-795
> URL: https://issues.apache.org/jira/browse/SOLR-795
> Project: Solr
> Issue Type: New Feature
> Components: spellchecker
> Affects Versions: 1.3
> Reporter: Jason Rennie
> Assignee: Shalin Shekhar Mangar
> Fix For: 1.4
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> I see that there's an option to automatically rebuild the spelling index on a commit. That's a nice feature that we'll consider using, but we run commits every few thousand document updates, which would yield ~100 spelling index rebuilds a day. OTOH, we run an optimize about once/day which seems like a more appropriate schedule for rebuilding the spelling index.
> Is there or could there be an option to rebuild the spelling index on optimize?
> Grant:
> Seems reasonable, could almost do it via the postOptimize call back already in the config, except the SpellCheckComponent's EvenListener is private static and has an empty postCommit implementation (which is what is called after optimization, since it is just like a commit in many ways)
> Thus, a patch would be needed.
> Shalin:
> postCommit/postOptimize callbacks happen after commit/optimize but before a
> new searcher is opened. Therefore, it is not possible to re-build spellcheck
> index on those events without opening a IndexReader directly on the solr
> index. That is why the event listener in SpellCheckComponent uses the
> newSearcher listener to build on commits.
> I don't think there is anything in the API currently to do what Jason wants.
> Hoss:
> FWIW: I believe it has to work that way because postCommit events might
> modify the index. (but i'm just guessing)
> couldn't the Listener's newSearcher() method just do something like
> this...
> if (rebuildOnlyAfterOptimize &&
> ! (newSearcher.getReader().isOptimized() &&
> ! oldSearcher.getReader().isOptimized()) {
> return;
> } else {
> // current impl
> }
> ...assuming a new "rebuildOnlyAfterOptimize" option was added?
> Grant:
> That seems reasonable.
> Another thing to think about, is maybe it is useful to provide some event metadata to the events that contain information about what triggered them. Something like a SolrEvent class such that postCommit looks like
> postCommit(SolrEvent evt)
> and
> public void newSearcher(SolrEvent evt, SolrIndexSearcher newSearcher, SolrIndexSearcher currentSearcher);
> Of course, since SolrEventListener is an interface...
> Shalin:
> Yup, that will work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Closed: (SOLR-795) spellcheck: buildOnOptimize
Posted by "Grant Ingersoll (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Grant Ingersoll closed SOLR-795.
--------------------------------
> spellcheck: buildOnOptimize
> ---------------------------
>
> Key: SOLR-795
> URL: https://issues.apache.org/jira/browse/SOLR-795
> Project: Solr
> Issue Type: New Feature
> Components: spellchecker
> Affects Versions: 1.3
> Reporter: Jason Rennie
> Assignee: Shalin Shekhar Mangar
> Fix For: 1.4
>
> Attachments: SOLR-795.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> I see that there's an option to automatically rebuild the spelling index on a commit. That's a nice feature that we'll consider using, but we run commits every few thousand document updates, which would yield ~100 spelling index rebuilds a day. OTOH, we run an optimize about once/day which seems like a more appropriate schedule for rebuilding the spelling index.
> Is there or could there be an option to rebuild the spelling index on optimize?
> Grant:
> Seems reasonable, could almost do it via the postOptimize call back already in the config, except the SpellCheckComponent's EvenListener is private static and has an empty postCommit implementation (which is what is called after optimization, since it is just like a commit in many ways)
> Thus, a patch would be needed.
> Shalin:
> postCommit/postOptimize callbacks happen after commit/optimize but before a
> new searcher is opened. Therefore, it is not possible to re-build spellcheck
> index on those events without opening a IndexReader directly on the solr
> index. That is why the event listener in SpellCheckComponent uses the
> newSearcher listener to build on commits.
> I don't think there is anything in the API currently to do what Jason wants.
> Hoss:
> FWIW: I believe it has to work that way because postCommit events might
> modify the index. (but i'm just guessing)
> couldn't the Listener's newSearcher() method just do something like
> this...
> if (rebuildOnlyAfterOptimize &&
> ! (newSearcher.getReader().isOptimized() &&
> ! oldSearcher.getReader().isOptimized()) {
> return;
> } else {
> // current impl
> }
> ...assuming a new "rebuildOnlyAfterOptimize" option was added?
> Grant:
> That seems reasonable.
> Another thing to think about, is maybe it is useful to provide some event metadata to the events that contain information about what triggered them. Something like a SolrEvent class such that postCommit looks like
> postCommit(SolrEvent evt)
> and
> public void newSearcher(SolrEvent evt, SolrIndexSearcher newSearcher, SolrIndexSearcher currentSearcher);
> Of course, since SolrEventListener is an interface...
> Shalin:
> Yup, that will work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SOLR-795) spellcheck: buildOnOptimize
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar resolved SOLR-795.
----------------------------------------
Resolution: Fixed
Committed revision 708653.
Thanks Jason!
> spellcheck: buildOnOptimize
> ---------------------------
>
> Key: SOLR-795
> URL: https://issues.apache.org/jira/browse/SOLR-795
> Project: Solr
> Issue Type: New Feature
> Components: spellchecker
> Affects Versions: 1.3
> Reporter: Jason Rennie
> Assignee: Shalin Shekhar Mangar
> Fix For: 1.4
>
> Attachments: SOLR-795.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> I see that there's an option to automatically rebuild the spelling index on a commit. That's a nice feature that we'll consider using, but we run commits every few thousand document updates, which would yield ~100 spelling index rebuilds a day. OTOH, we run an optimize about once/day which seems like a more appropriate schedule for rebuilding the spelling index.
> Is there or could there be an option to rebuild the spelling index on optimize?
> Grant:
> Seems reasonable, could almost do it via the postOptimize call back already in the config, except the SpellCheckComponent's EvenListener is private static and has an empty postCommit implementation (which is what is called after optimization, since it is just like a commit in many ways)
> Thus, a patch would be needed.
> Shalin:
> postCommit/postOptimize callbacks happen after commit/optimize but before a
> new searcher is opened. Therefore, it is not possible to re-build spellcheck
> index on those events without opening a IndexReader directly on the solr
> index. That is why the event listener in SpellCheckComponent uses the
> newSearcher listener to build on commits.
> I don't think there is anything in the API currently to do what Jason wants.
> Hoss:
> FWIW: I believe it has to work that way because postCommit events might
> modify the index. (but i'm just guessing)
> couldn't the Listener's newSearcher() method just do something like
> this...
> if (rebuildOnlyAfterOptimize &&
> ! (newSearcher.getReader().isOptimized() &&
> ! oldSearcher.getReader().isOptimized()) {
> return;
> } else {
> // current impl
> }
> ...assuming a new "rebuildOnlyAfterOptimize" option was added?
> Grant:
> That seems reasonable.
> Another thing to think about, is maybe it is useful to provide some event metadata to the events that contain information about what triggered them. Something like a SolrEvent class such that postCommit looks like
> postCommit(SolrEvent evt)
> and
> public void newSearcher(SolrEvent evt, SolrIndexSearcher newSearcher, SolrIndexSearcher currentSearcher);
> Of course, since SolrEventListener is an interface...
> Shalin:
> Yup, that will work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-795) spellcheck: buildOnOptimize
Posted by "Shalin Shekhar Mangar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Shalin Shekhar Mangar updated SOLR-795:
---------------------------------------
Attachment: SOLR-795.patch
Attached patch to add this feature.
*Changes*
# "buildOnOptimize" is added as a configuration
# SpellCheckerListener uses the values of buildOnCommit and buildOnOptimize to decide when to build the index
# To build index only on optimize, specify buildOnOptimize as true. No need to specify buildOnCommit.
I'll commit this in a day or two.
> spellcheck: buildOnOptimize
> ---------------------------
>
> Key: SOLR-795
> URL: https://issues.apache.org/jira/browse/SOLR-795
> Project: Solr
> Issue Type: New Feature
> Components: spellchecker
> Affects Versions: 1.3
> Reporter: Jason Rennie
> Assignee: Shalin Shekhar Mangar
> Fix For: 1.4
>
> Attachments: SOLR-795.patch
>
> Original Estimate: 2h
> Remaining Estimate: 2h
>
> I see that there's an option to automatically rebuild the spelling index on a commit. That's a nice feature that we'll consider using, but we run commits every few thousand document updates, which would yield ~100 spelling index rebuilds a day. OTOH, we run an optimize about once/day which seems like a more appropriate schedule for rebuilding the spelling index.
> Is there or could there be an option to rebuild the spelling index on optimize?
> Grant:
> Seems reasonable, could almost do it via the postOptimize call back already in the config, except the SpellCheckComponent's EvenListener is private static and has an empty postCommit implementation (which is what is called after optimization, since it is just like a commit in many ways)
> Thus, a patch would be needed.
> Shalin:
> postCommit/postOptimize callbacks happen after commit/optimize but before a
> new searcher is opened. Therefore, it is not possible to re-build spellcheck
> index on those events without opening a IndexReader directly on the solr
> index. That is why the event listener in SpellCheckComponent uses the
> newSearcher listener to build on commits.
> I don't think there is anything in the API currently to do what Jason wants.
> Hoss:
> FWIW: I believe it has to work that way because postCommit events might
> modify the index. (but i'm just guessing)
> couldn't the Listener's newSearcher() method just do something like
> this...
> if (rebuildOnlyAfterOptimize &&
> ! (newSearcher.getReader().isOptimized() &&
> ! oldSearcher.getReader().isOptimized()) {
> return;
> } else {
> // current impl
> }
> ...assuming a new "rebuildOnlyAfterOptimize" option was added?
> Grant:
> That seems reasonable.
> Another thing to think about, is maybe it is useful to provide some event metadata to the events that contain information about what triggered them. Something like a SolrEvent class such that postCommit looks like
> postCommit(SolrEvent evt)
> and
> public void newSearcher(SolrEvent evt, SolrIndexSearcher newSearcher, SolrIndexSearcher currentSearcher);
> Of course, since SolrEventListener is an interface...
> Shalin:
> Yup, that will work.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.