[jira] [Commented] (SOLR-1028) Automatic core loading unloading for multicore

["Erick Erickson (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SOLR-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13490666#comment-13490666 ] 

Erick Erickson commented on SOLR-1028:
--------------------------------------

I think the main thrust of these changes is compatible with SolrCloud, but the more eyes the merrier. Here's my reasoning:

> the idea of rapidly opening/closing cores, and limiting the number of concurrently-open cores will have to be handled locally, which is what the bulk of these changes actually are. One bit of noise here will be that I refactored the ZK-case to the local-case, so it may look worse than it is.

> After I looked a bit more at the ZK instance, I noticed that the SolrCloud stuff _also_ has a new coreDescriptorProvider (I'll reconcile those two as part of SOLR-1306, BTW. There's no reason to have two). So whether the descriptor comes from ZK or comes from "some other place" _should_ (tm) be transparent on the level of these changes. 

> I think the biggest question for me about how ZK interacts with all this is mostly how opening/closing cores is _supposed_ to work during indexing. The whole notion of distributed indexing across a zillion rapidly opening/closing cores on a single machine really seems like something that shouldn't be happening during indexing at all. Or at least a way for users to shoot themselves in the foot. Imagine that you have 10K cores/machine, each with 3 replicas and you're randomly sending updates to those cores. Further imagine that your concurrently open core limit is 100. Throughput would be horrible. I suppose the right solution is that whoever is setting this up (and I assume they're pretty sophisticated) needs to index to a single core at a time until all the updates were sent, then go on to the _next_ core. Or pay the price speed-wise.

> The other bit I'm not clear about the ZK end is how we keep, say, 10K coreDescriptors in ZK with the 1M limit as has been mentioned. But again I don't think that is incompatible at all with these changes.

> I don't think all the JIRA's associated with SOLR-1293 need to be addressed. Some of them appear to be already done or have yet to be proved to be helpful. But since they're all local to the Solr instance anyway, I suspect they'll be the same whether in SolrCloud or not.

> If we go to a model where ZK runs transparently even in the "normal" case, then as long as the CoreDescriptorProvider is pluggable in that situation, I think we're good to go.

All that said, it would be a Good Thing if anyone can poke holes in my hand-waving before I back myself into a corner. Note that if anyone looks at this, they should look at SOLR-1306 in conjunction with this JIRA. Between the two of them the bulk of the changes I'm thinking about are handled.
                
> Automatic core loading unloading for multicore
> ----------------------------------------------
>
>                 Key: SOLR-1028
>                 URL: https://issues.apache.org/jira/browse/SOLR-1028
>             Project: Solr
>          Issue Type: New Feature
>          Components: multicore
>    Affects Versions: 4.0, 5.0
>            Reporter: Noble Paul
>            Assignee: Erick Erickson
>             Fix For: 4.1, 5.0
>
>         Attachments: SOLR-1028.patch, SOLR-1028.patch
>
>
> usecase: I have many small cores (say one per user) on a single Solr box . All the cores are not be always needed . But when I need it I should be able to directly issue a search request and the core must be STARTED automatically and the request must be served.
> This also requires that I must have an upper limit on the no:of cores that should be loaded at any given point in time. If the limit is crossed the CoreContainer must unload a core (preferably the least recently used core)  
> There must be a choice of specifying some cores as fixed. These cores must never be unloaded 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org