You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Srivatsan <ra...@gmail.com> on 2013/08/05 13:01:55 UTC
Collection - loadOnStartup
Hi,
I am using solr -4.3.0 for my search application. I will create collection
via CollectionAPI. I tried to pass "loadOnStartup" value also. But that
approach didnt work.
My question is, How to set loadOnStartup to false for cores getting created
by CollectionsAPI?
I am creating cores by the following snippet of code
/UpdateRequest req = new UpdateRequest("/admin/collections");//No I18N
req.setParam("action","create");
req.setParam("name","collection1");
req.setParam("numShards","1");
req.setParam("replicationFactor","3");
req.setParam("loadOnStartup","false");
server.request(req);/
Thanks
Srivatsan
--
View this message in context: http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531.html
Sent from the Solr - User mailing list archive at Nabble.com.
RE: Collection - loadOnStartup
Posted by Sudhir Kumar <Su...@eperium.com>.
unsubscribe
________________________________________
From: Srivatsan <ra...@gmail.com>
Sent: Wednesday, April 23, 2014 2:58 PM
To: solr-user@lucene.apache.org
Subject: Re: Collection - loadOnStartup
Dear Rich Mayfield,
Sorry for late reply.
We didn't continue further in that. We have moved to Elasticsearch.
Elasticsearch suits best for our case than Solr.
with regards
Ranjith Venkatesan
--
View this message in context: http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4132688.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection - loadOnStartup
Posted by Srivatsan <ra...@gmail.com>.
Dear Rich Mayfield,
Sorry for late reply.
We didn't continue further in that. We have moved to Elasticsearch.
Elasticsearch suits best for our case than Solr.
with regards
Ranjith Venkatesan
--
View this message in context: http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4132688.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection - loadOnStartup
Posted by Rich Mayfield <ma...@gmail.com>.
You're right, we're basically working around inherent problems. SolrCloud and
large numbers of cores is not a combination that yields reliable restarts.
Even under the best of conditions - a completely silent (no updates, no
selects) environment - if I restart two nodes, each containing ~800
replicas, I am not confident that all collections will spin up.
SOLR-5990 documents one such case where both replicas in many collections
end up in a recovery_failed state.
--
View this message in context: http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4131766.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection - loadOnStartup
Posted by Erick Erickson <er...@gmail.com>.
I would _really_ advise you avoid trying this without an absolutely
compelling, _demonstrated_ need.
Unless you have a _lot_ of cores, the load time
for even 100 or so isn't horrible, so if you're noticing a
significant lag time, it's probably worth looking at why that is before
jumping into using unsupported options. If you insist on doing this,
I'll be anxiously awaiting reports of whatever problems you have.
There are tests that hammer this functionality, see TestLazyCores
and the speed that cores hums right along.
So the first thing I'd look at is if your configuration is such that you
have large transaction logs that are getting replayed at startup, see:
http://wiki.apache.org/solr/SolrPerformanceProblems#Slow_startup
After that I'd try to pin down why you are seeing long load times
if that's not relevant.
And you haven't told us anything about your setup, this smells like
an XY problem.
Best
Erick
On Fri, Aug 30, 2013 at 10:13 AM, Srivatsan <ra...@gmail.com>wrote:
> I started looking in reducing the time taken to load cores during cluster
> restart. For initializing the core, building config file takes considerable
> amount of time. In our case, schema remains same for all collections(cores)
> . Hence it can be kept static, and avoid loading time.
>
> In the mean while i have gone through schemaless feature of solr-4.5. In
> such a case, do we need to define schema.xml file or it ll automatically
> assigns field types dynamically? If there is no need to define schema.xml
> file, then automatically core initialization takes lesser time rite?
>
>
> Thanks in advance
>
> Srivatsan
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4087510.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Collection - loadOnStartup
Posted by Srivatsan <ra...@gmail.com>.
I started looking in reducing the time taken to load cores during cluster
restart. For initializing the core, building config file takes considerable
amount of time. In our case, schema remains same for all collections(cores)
. Hence it can be kept static, and avoid loading time.
In the mean while i have gone through schemaless feature of solr-4.5. In
such a case, do we need to define schema.xml file or it ll automatically
assigns field types dynamically? If there is no need to define schema.xml
file, then automatically core initialization takes lesser time rite?
Thanks in advance
Srivatsan
--
View this message in context: http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4087510.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection - loadOnStartup
Posted by Erick Erickson <er...@gmail.com>.
Frankly, you're into somewhat uncharted waters, the whole lazy core
capability was designed for non-cloud mode.
Cores are initialized when the first request comes in that addresses
the core. Whether ZK and SolrCloud know a core is active before
the first time it's loaded I don't know.
I think you'll find that combining lazy core loading with SolrCloud will
have more issues, you've found two so far and no doubt there
are more lurking. If you'd like to help make SolrCloud work with
transient cores, patches welcome!
I won't have time/motivation to work on this for the foreseeable future,
I'm not even sure it _should_ be supported. So if all you need to
do is wait for a while, you have a work-around. Otherwise, I'm clueless.
Best
Erick
On Mon, Aug 12, 2013 at 8:54 AM, Srivatsan <ra...@gmail.com>wrote:
> Hi
> I manually edited core.properties file by setting /loadOnStartup=false/ on
> all cores. But here i am facing a problem. After starting the solr cloud
> servers, I couldnt index to any collection till particular interval. I am
> getting exception like "No Live Solr Servers".
>
> => If we submit a index request why does the cores didnt initialize at run
> time?
>
> => How the cores will be initialized if loadOnStartup=false?
>
>
> Thanks
>
> Srivatsan
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4083972.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Collection - loadOnStartup
Posted by Srivatsan <ra...@gmail.com>.
Hi
I manually edited core.properties file by setting /loadOnStartup=false/ on
all cores. But here i am facing a problem. After starting the solr cloud
servers, I couldnt index to any collection till particular interval. I am
getting exception like "No Live Solr Servers".
=> If we submit a index request why does the cores didnt initialize at run
time?
=> How the cores will be initialized if loadOnStartup=false?
Thanks
Srivatsan
--
View this message in context: http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4083972.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection - loadOnStartup
Posted by Rich Mayfield <ma...@gmail.com>.
I was actually going to try orchestrating SolrCloud restart myself using
loadOnStartup="false".
Did you pursue this any further?
With Solr 4.7.1...
I found that using core LOAD, RELOAD, and CREATE do not take a "down"
replica to "active". What I've found so far is that I can startup a
collection if I hit each replica with a "select". The first replica that
receives a "select" hangs; concurrently I hit the 2nd replica (of 2) with a
"select". The collection then comes up.
--
View this message in context: http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4131760.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection - loadOnStartup
Posted by Srivatsan <ra...@gmail.com>.
Thanks Erick
--
View this message in context: http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4082938.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection - loadOnStartup
Posted by Erick Erickson <er...@gmail.com>.
I don't think you can, really. Collections, at this point, is more
geared towards SolrCloud. The idea of lazy loading
matched with SolrCloud makes my head hurt.
I'm afraid for the nonce you'll have to individually edit the
solr.xml or core.properties files on the nodes once the
collections are created.
Best
Erick
On Tue, Aug 6, 2013 at 2:11 AM, Srivatsan <ra...@gmail.com>wrote:
> Then if so, how to set loadOnStartup for collectionsAPI in solr4.4 ???
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4082731.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Collection - loadOnStartup
Posted by Srivatsan <ra...@gmail.com>.
Then if so, how to set loadOnStartup for collectionsAPI in solr4.4 ???
--
View this message in context: http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4082731.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection - loadOnStartup
Posted by didier deshommes <df...@gmail.com>.
For Solr 4.3.0, I don't think you can pass loadOnStartup to the Collections
API, although the Cores API accepts it. That's been my experience anyway.
On Mon, Aug 5, 2013 at 6:27 AM, Srivatsan <ra...@gmail.com>wrote:
> No errors in zookeeper and solr. I m using CloudSolrServer for creating
> collections as said above.I just want to set loadOnStartup to false for
> cores in solr.xml. I dont want all cores to loadonstartup. Hence when
> creating collection, i m trying to set this parameter to false. But still i
> m getting same value for loadOnStartup in solr.xml
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4082546.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Collection - loadOnStartup
Posted by Srivatsan <ra...@gmail.com>.
No errors in zookeeper and solr. I m using CloudSolrServer for creating
collections as said above.I just want to set loadOnStartup to false for
cores in solr.xml. I dont want all cores to loadonstartup. Hence when
creating collection, i m trying to set this parameter to false. But still i
m getting same value for loadOnStartup in solr.xml
--
View this message in context: http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531p4082546.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Collection - loadOnStartup
Posted by Erick Erickson <er...@gmail.com>.
You haven't given us much information to go on. _how_ does it fail?
What do the logs show? Any error returned? What is the response from
the server? Is zookeeper showing any problems?
Best
Erick
On Mon, Aug 5, 2013 at 7:01 AM, Srivatsan <ra...@gmail.com>wrote:
> Hi,
>
> I am using solr -4.3.0 for my search application. I will create collection
> via CollectionAPI. I tried to pass "loadOnStartup" value also. But that
> approach didnt work.
>
> My question is, How to set loadOnStartup to false for cores getting created
> by CollectionsAPI?
>
> I am creating cores by the following snippet of code
>
> /UpdateRequest req = new UpdateRequest("/admin/collections");//No I18N
> req.setParam("action","create");
> req.setParam("name","collection1");
> req.setParam("numShards","1");
> req.setParam("replicationFactor","3");
> req.setParam("loadOnStartup","false");
> server.request(req);/
>
> Thanks
>
> Srivatsan
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Collection-loadOnStartup-tp4082531.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>