You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-dev@lucene.apache.org by "Noble Paul (JIRA)" <ji...@apache.org> on 2009/07/17 12:27:14 UTC
[jira] Created: (SOLR-1293) Support for large no:of cores and
faster loading/unloading of cores
Support for large no:of cores and faster loading/unloading of cores
-------------------------------------------------------------------
Key: SOLR-1293
URL: https://issues.apache.org/jira/browse/SOLR-1293
Project: Solr
Issue Type: New Feature
Reporter: Noble Paul
Fix For: 1.5
Solr , currently ,is not very suitable for a large no:of homogeneous cores where you require fast/frequent loading/unloading of cores . usually a core is required to be loaded just to fire a search query or to just index one document
The requirements of such a system are.
* Very efficient loading of cores . Solr cannot afford to read and parse and create Schema, SolrConfig Objects for each core each time the core has to be loaded ( SOLR-919 , SOLR-920)
* START STOP core . Currently it is only possible to unload a core (SOLR-880)
* Automatic loading of cores . If a core is present and it is not loaded and a request comes for that load it automatically before serving up a request
* As there are a large no:of cores , all the cores cannot be kept loaded always. There has to be an upper limit beyond which we need to unload a few cores (probably the least recently used ones)
* Automatic allotment of dataDir for cores. If the no:of cores is too high al the cores' dataDirs cannot live in the same dir. There is an upper limit on the no:of dirs you can create in a unix dir w/o affecting performance
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1293) Support for large no:of cores and
faster loading/unloading of cores
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737843#action_12737843 ]
Noble Paul commented on SOLR-1293:
----------------------------------
access control is not something we were trying to do in Solr. The Solr farm is setup behind our mail servers and it is never exposed outside.
> Support for large no:of cores and faster loading/unloading of cores
> -------------------------------------------------------------------
>
> Key: SOLR-1293
> URL: https://issues.apache.org/jira/browse/SOLR-1293
> Project: Solr
> Issue Type: New Feature
> Reporter: Noble Paul
> Fix For: 1.5
>
> Attachments: SOLR-1293.patch
>
>
> Solr , currently ,is not very suitable for a large no:of homogeneous cores where you require fast/frequent loading/unloading of cores . usually a core is required to be loaded just to fire a search query or to just index one document
> The requirements of such a system are.
> * Very efficient loading of cores . Solr cannot afford to read and parse and create Schema, SolrConfig Objects for each core each time the core has to be loaded ( SOLR-919 , SOLR-920)
> * START STOP core . Currently it is only possible to unload a core (SOLR-880)
> * Automatic loading of cores . If a core is present and it is not loaded and a request comes for that load it automatically before serving up a request
> * As there are a large no:of cores , all the cores cannot be kept loaded always. There has to be an upper limit beyond which we need to unload a few cores (probably the least recently used ones)
> * Automatic allotment of dataDir for cores. If the no:of cores is too high al the cores' dataDirs cannot live in the same dir. There is an upper limit on the no:of dirs you can create in a unix dir w/o affecting performance
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1293) Support for large no:of cores and
faster loading/unloading of cores
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737838#action_12737838 ]
Otis Gospodnetic commented on SOLR-1293:
----------------------------------------
Do you have any thoughts on handling the situation where each core belongs to a different party and each party has access *only* to its own core via Solr Admin (i.e. doesn't see all the other cores hosted by the instance)? Only the privileged administrator user can see and access all cores.
Have you done any work in on this or is this on your TODO?
> Support for large no:of cores and faster loading/unloading of cores
> -------------------------------------------------------------------
>
> Key: SOLR-1293
> URL: https://issues.apache.org/jira/browse/SOLR-1293
> Project: Solr
> Issue Type: New Feature
> Reporter: Noble Paul
> Fix For: 1.5
>
> Attachments: SOLR-1293.patch
>
>
> Solr , currently ,is not very suitable for a large no:of homogeneous cores where you require fast/frequent loading/unloading of cores . usually a core is required to be loaded just to fire a search query or to just index one document
> The requirements of such a system are.
> * Very efficient loading of cores . Solr cannot afford to read and parse and create Schema, SolrConfig Objects for each core each time the core has to be loaded ( SOLR-919 , SOLR-920)
> * START STOP core . Currently it is only possible to unload a core (SOLR-880)
> * Automatic loading of cores . If a core is present and it is not loaded and a request comes for that load it automatically before serving up a request
> * As there are a large no:of cores , all the cores cannot be kept loaded always. There has to be an upper limit beyond which we need to unload a few cores (probably the least recently used ones)
> * Automatic allotment of dataDir for cores. If the no:of cores is too high al the cores' dataDirs cannot live in the same dir. There is an upper limit on the no:of dirs you can create in a unix dir w/o affecting performance
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1293) Support for large no:of cores and
faster loading/unloading of cores
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737886#action_12737886 ]
Noble Paul commented on SOLR-1293:
----------------------------------
yes. if you use the core admin command status , use verbose=false so that you only get the minimum info
> Support for large no:of cores and faster loading/unloading of cores
> -------------------------------------------------------------------
>
> Key: SOLR-1293
> URL: https://issues.apache.org/jira/browse/SOLR-1293
> Project: Solr
> Issue Type: New Feature
> Reporter: Noble Paul
> Fix For: 1.5
>
> Attachments: SOLR-1293.patch
>
>
> Solr , currently ,is not very suitable for a large no:of homogeneous cores where you require fast/frequent loading/unloading of cores . usually a core is required to be loaded just to fire a search query or to just index one document
> The requirements of such a system are.
> * Very efficient loading of cores . Solr cannot afford to read and parse and create Schema, SolrConfig Objects for each core each time the core has to be loaded ( SOLR-919 , SOLR-920)
> * START STOP core . Currently it is only possible to unload a core (SOLR-880)
> * Automatic loading of cores . If a core is present and it is not loaded and a request comes for that load it automatically before serving up a request
> * As there are a large no:of cores , all the cores cannot be kept loaded always. There has to be an upper limit beyond which we need to unload a few cores (probably the least recently used ones)
> * Automatic allotment of dataDir for cores. If the no:of cores is too high al the cores' dataDirs cannot live in the same dir. There is an upper limit on the no:of dirs you can create in a unix dir w/o affecting performance
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SOLR-1293) Support for large no:of cores and
faster loading/unloading of cores
Posted by "Noble Paul (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Noble Paul updated SOLR-1293:
-----------------------------
Attachment: SOLR-1293.patch
The patch is untested. The internal patch we used was on an older svn version . That patch is just merged to trunk and I am submitting that. I plan to test this sometime soon. But for those need this really badly this can be a starting point.
refer the wiki page for usage details http://wiki.apache.org/solr/LotsOfCores
> Support for large no:of cores and faster loading/unloading of cores
> -------------------------------------------------------------------
>
> Key: SOLR-1293
> URL: https://issues.apache.org/jira/browse/SOLR-1293
> Project: Solr
> Issue Type: New Feature
> Reporter: Noble Paul
> Fix For: 1.5
>
> Attachments: SOLR-1293.patch
>
>
> Solr , currently ,is not very suitable for a large no:of homogeneous cores where you require fast/frequent loading/unloading of cores . usually a core is required to be loaded just to fire a search query or to just index one document
> The requirements of such a system are.
> * Very efficient loading of cores . Solr cannot afford to read and parse and create Schema, SolrConfig Objects for each core each time the core has to be loaded ( SOLR-919 , SOLR-920)
> * START STOP core . Currently it is only possible to unload a core (SOLR-880)
> * Automatic loading of cores . If a core is present and it is not loaded and a request comes for that load it automatically before serving up a request
> * As there are a large no:of cores , all the cores cannot be kept loaded always. There has to be an upper limit beyond which we need to unload a few cores (probably the least recently used ones)
> * Automatic allotment of dataDir for cores. If the no:of cores is too high al the cores' dataDirs cannot live in the same dir. There is an upper limit on the no:of dirs you can create in a unix dir w/o affecting performance
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-1293) Support for large no:of cores and
faster loading/unloading of cores
Posted by "Otis Gospodnetic (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/SOLR-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12737844#action_12737844 ]
Otis Gospodnetic commented on SOLR-1293:
----------------------------------------
OK, thanks.
When you go to your Solr Admin page today, it lists all cores, even if there are 10000 of them?
> Support for large no:of cores and faster loading/unloading of cores
> -------------------------------------------------------------------
>
> Key: SOLR-1293
> URL: https://issues.apache.org/jira/browse/SOLR-1293
> Project: Solr
> Issue Type: New Feature
> Reporter: Noble Paul
> Fix For: 1.5
>
> Attachments: SOLR-1293.patch
>
>
> Solr , currently ,is not very suitable for a large no:of homogeneous cores where you require fast/frequent loading/unloading of cores . usually a core is required to be loaded just to fire a search query or to just index one document
> The requirements of such a system are.
> * Very efficient loading of cores . Solr cannot afford to read and parse and create Schema, SolrConfig Objects for each core each time the core has to be loaded ( SOLR-919 , SOLR-920)
> * START STOP core . Currently it is only possible to unload a core (SOLR-880)
> * Automatic loading of cores . If a core is present and it is not loaded and a request comes for that load it automatically before serving up a request
> * As there are a large no:of cores , all the cores cannot be kept loaded always. There has to be an upper limit beyond which we need to unload a few cores (probably the least recently used ones)
> * Automatic allotment of dataDir for cores. If the no:of cores is too high al the cores' dataDirs cannot live in the same dir. There is an upper limit on the no:of dirs you can create in a unix dir w/o affecting performance
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.