You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Shamik Bandopadhyay <sh...@gmail.com> on 2015/04/03 03:51:55 UTC
Unable to update config file using zkcli or RELOAD
Hi,
I'm facing a weird issue. I've a solr cloud cluster with 2 shards having
a replica each. I started the cluster
using -Dbootstrap_confdir=./solr/collection1/conf
-Dcollection.configName=myconf. After the cluster is up and running, I
added a new request handler (newhandler) and wanted to push it without
restarting the server. First, I tried the RELOAD option. I ran
http://54.151.xx.xxx:8983/solr/admin/cores?action=RELOAD&core=collection1
The command was successful, but when I logged in to the admin screen, the
solrconfig didn't show the request handler. Next I tried the zkcli script
on shard 1.
sh zkcli.sh -cmd upconfig -zkhost zoohost1:2181 -confname myconf -solrhome
/mnt/opt/solrhome/ -confdir /mnt/opt/solrhome/solr/collection1/conf/
The script ran successfully and I could see the updated solrconfig file in
Solr admin. But then, when I tried
http://54.151.xx.xxx:8983/solr/collection1/newhandler
I got a 404. Not sure what I'm doing wrong. Do I need to run the zkcli
script on each node? I'm using Solr 5.0.
Regards,
Shamik
Re: Unable to update config file using zkcli or RELOAD
Posted by Noble Paul <no...@gmail.com>.
The behavior has changed from Solr 5.0 onwards
Please refer to the "How does it work section" here
https://cwiki.apache.org/confluence/display/solr/Config+API
TL:DR
* Every node watches the conf set directory it is using
* Updating individual files WILL NOT trigger a config reload. BUt if you
modify the config set dir , it will trigger a reload. This is how config
API works
On Mon, Apr 6, 2015 at 2:34 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 4/5/2015 12:32 AM, Shai Erera wrote:
> > So, the questions that I have are:
> >
> > 1. It does look like Solr re-loads cores on configuration changes, is
> > that true?
> > 2. If (1) is YES, do I still need to manually invoke a collection
> RELOAD
> > explicitly after updating the configuration?
> > 3. Can someone explain the errors I see in the log, even though the
> test
> > passes and as I indicated, I'm able to index more documents and
> search them?
>
> My experience with SolrCloud is somewhat limited and even more limited
> in recent versions, but it was my understanding that config/schema
> changes do not result in new behavior without an explicit reload, unless
> you are using the managed schema or managed resources API to make the
> changes. I don't know much about how the managed APIs actually work.
>
> Trying to interpret the stacktraces makes my brain hurt. You probably
> know a lot more about the code for those areas than I do.
>
> Thanks,
> Shawn
>
>
--
-----------------------------------------------------
Noble Paul
Re: Unable to update config file using zkcli or RELOAD
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/5/2015 12:32 AM, Shai Erera wrote:
> So, the questions that I have are:
>
> 1. It does look like Solr re-loads cores on configuration changes, is
> that true?
> 2. If (1) is YES, do I still need to manually invoke a collection RELOAD
> explicitly after updating the configuration?
> 3. Can someone explain the errors I see in the log, even though the test
> passes and as I indicated, I'm able to index more documents and search them?
My experience with SolrCloud is somewhat limited and even more limited
in recent versions, but it was my understanding that config/schema
changes do not result in new behavior without an explicit reload, unless
you are using the managed schema or managed resources API to make the
changes. I don't know much about how the managed APIs actually work.
Trying to interpret the stacktraces makes my brain hurt. You probably
know a lot more about the code for those areas than I do.
Thanks,
Shawn
Re: Unable to update config file using zkcli or RELOAD
Posted by Shai Erera <se...@gmail.com>.
I was looking into a similar issue when I saw this thread. I wrote a simple
test which uploads a configuration to ZK, creates a collection and indexes
a document. Then I delete the configuration and I see these in the logs,
which is expected:
6299 [Thread-13] INFO org.apache.solr.cloud.ZkController – Check for
collection zkNode:mycollection
6299 [Thread-13] INFO org.apache.solr.cloud.ZkController – Collection
zkNode exists
6299 [Thread-13] INFO org.apache.solr.common.cloud.ZkStateReader – Load
collection config from:/collections/mycollection
6322 [Thread-13] ERROR org.apache.solr.common.cloud.ZkStateReader –
Specified config does not exist in ZooKeeper:ConfigurationManagementTest
6323 [Thread-13] WARN org.apache.solr.cloud.ZkController – listener
throws error
org.apache.solr.common.SolrException: Unable to reload core
[mycollection_shard1_replica1]
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:615)
Caused by: org.apache.solr.common.cloud.ZooKeeperException: Specified
config does not exist in ZooKeeper:ConfigurationManagementTest
at
org.apache.solr.common.cloud.ZkStateReader.readConfigName(ZkStateReader.java:171)
at
org.apache.solr.cloud.CloudConfigSetService.createCoreResourceLoader(CloudConfigSetService.java:37)
at
org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:58)
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:608)
But, the exception comes from CoreContainer.reload(), which seems to get
invoked when there was a configuration change in ZK?
Then my test indexes another document and verifies it exists by searching
for it, and it works, without me explicitly reloading the core. Is that
expected behavior?
Then I tested what happens if I re-create the configuration after deleting
it (to simulate a full configuration update), so I re-uploaded the
configuration after deleting it (and indexed the second document still).
Everything works as above, without me explicitly reloading the core.
Though, I do see this being logged after my test completes, and is in the
process of shutting down:
6184 [Thread-14] INFO org.apache.solr.core.CoreContainer – Reloading
SolrCore 'mycollection_shard1_replica1' using configuration from collection
mycollection
6188
[OverseerStateUpdate-93599389182590978-127.0.0.1:63070_solr-n_0000000000]
INFO org.apache.solr.cloud.overseer.ZkStateWriter – going to
update_collection /collections/mycollection/state.json version: 3
6227 [zkCallback-5-thread-1] INFO
org.apache.solr.common.cloud.ZkStateReader – A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged
path:/collections/mycollection/state.json, has occurred - updating...
6392 [main] INFO org.apache.solr.core.SolrCore –
SolrDeletionPolicy.onCommit: commits: num=2
commit{dir=NRTCachingDirectory(MMapDirectory@C:\Users\shaie\AppData\Local\Temp\1428213335695-0\solr\node1\mycollection_shard1_replica1\data\index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@46b0f5e9;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_1,generation=1}
commit{dir=NRTCachingDirectory(MMapDirectory@C:\Users\shaie\AppData\Local\Temp\1428213335695-0\solr\node1\mycollection_shard1_replica1\data\index
lockFactory=org.apache.lucene.store.NativeFSLockFactory@46b0f5e9;
maxCacheMB=48.0 maxMergeSizeMB=4.0),segFN=segments_2,generation=2}
6392 [main] INFO org.apache.solr.core.SolrCore – newest commit generation
= 2
6395 [Thread-14] WARN org.apache.solr.cloud.ZkController – listener
throws error
org.apache.solr.common.SolrException: Unable to reload core
[mycollection_shard1_replica1]
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:615)
Caused by: java.lang.IllegalStateException: IndexWriter has been closed
at
org.apache.solr.update.SolrCoreState.increfSolrCoreState(SolrCoreState.java:53)
at org.apache.solr.core.SolrCore.reload(SolrCore.java:442)
at org.apache.solr.core.CoreContainer.reload(CoreContainer.java:610)
In other times I run the test I get this:
7199 [Thread-14] INFO org.apache.solr.core.CoreContainer – replacing
core: mycollection_shard1_replica1
7200 [searcherExecutor-11-thread-1] WARN
org.apache.solr.core.JmxMonitoredMap – Failed to register info bean:
searcher
javax.management.InstanceAlreadyExistsException:
solr/mycollection_shard1_replica1:type=searcher,id=org.apache.solr.search.SolrIndexSearcher
at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:437)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerWithRepository(DefaultMBeanServerInterceptor.java:1898)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:966)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:900)
at
com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:324)
at
com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:522)
at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:153)
at org.apache.solr.core.JmxMonitoredMap.put(JmxMonitoredMap.java:52)
at
org.apache.solr.search.SolrIndexSearcher.register(SolrIndexSearcher.java:420)
at org.apache.solr.core.SolrCore.registerSearcher(SolrCore.java:1957)
I tried both SOFT_COMMIT and COMMIT in my update requests, the exceptions
alternate and seems to be a timing issue.
So, the questions that I have are:
1. It does look like Solr re-loads cores on configuration changes, is
that true?
2. If (1) is YES, do I still need to manually invoke a collection RELOAD
explicitly after updating the configuration?
3. Can someone explain the errors I see in the log, even though the test
passes and as I indicated, I'm able to index more documents and search them?
Shai
On Fri, Apr 3, 2015 at 7:22 PM, shamik <sh...@gmail.com> wrote:
> Thanks Shawn for the pointer, really appreciate it.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Unable-to-update-config-file-using-zkcli-or-RELOAD-tp4197376p4197494.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Unable to update config file using zkcli or RELOAD
Posted by shamik <sh...@gmail.com>.
Thanks Shawn for the pointer, really appreciate it.
--
View this message in context: http://lucene.472066.n3.nabble.com/Unable-to-update-config-file-using-zkcli-or-RELOAD-tp4197376p4197494.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Unable to update config file using zkcli or RELOAD
Posted by Shawn Heisey <ap...@elyograg.org>.
On 4/3/2015 12:28 AM, shamik wrote:
> Ok, I figured the steps in case someone needs a reference. It required both
> zkcli and RELOAD to update the changes.
>
> 1. Use zkcli to load the changes. I ran it from the node which used the
> bootstrapping.
>
> sh zkcli.sh -cmd upconfig -zkhost zoohost1:2181 -confname myconf -solrhome
> /mnt/opt/solrhome/ -confdir /mnt/opt/solrhome/solr/collection1/conf/
>
> 2. Use the same node to run the RELOAD
>
> http://54.151.xx.xxx:8983/solr/admin/cores?action=RELOAD&core=collection1
I was about to reply to your first message, then I read the second one.
You've found the essence of the correct procedure, which is to upload
the new config and then RELOAD.
With SolrCloud, you should reload using the Collections API, not the
CoreAdmin API. It will reload all the shard replicas (cores) in the
collection with one HTTP call. Since a fault tolerant collection will
involve at least two cores located on different servers, the Collections
API is a lot easier to manage.
https://cwiki.apache.org/confluence/display/solr/Collections+API#CollectionsAPI-api2
Thanks,
Shawn
Re: Unable to update config file using zkcli or RELOAD
Posted by shamik <sh...@gmail.com>.
Ok, I figured the steps in case someone needs a reference. It required both
zkcli and RELOAD to update the changes.
1. Use zkcli to load the changes. I ran it from the node which used the
bootstrapping.
sh zkcli.sh -cmd upconfig -zkhost zoohost1:2181 -confname myconf -solrhome
/mnt/opt/solrhome/ -confdir /mnt/opt/solrhome/solr/collection1/conf/
2. Use the same node to run the RELOAD
http://54.151.xx.xxx:8983/solr/admin/cores?action=RELOAD&core=collection1
--
View this message in context: http://lucene.472066.n3.nabble.com/Unable-to-update-config-file-using-zkcli-or-RELOAD-tp4197376p4197393.html
Sent from the Solr - User mailing list archive at Nabble.com.