You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by kaustubh147 <ka...@gmail.com> on 2013/11/01 04:18:13 UTC
unable to load core after cluster restart
Hi,
Glassfish 3.1.2.2
Solr 4.5
Zookeeper 3.4.5
We have set up a SolrCloud with 4 Solr nodes and 3 zookeeper instances.
I start the cluster for the first time with bootstrap_conf= true.... All the
nodes starts property.. I am creating cores (with the same name) on all 4
instances. I can add multiple cores on each of the instances... logically I
have 5 collections.
Now i am creating indexes.. and it automatically creates 4 copies of the
index, one for each instance in appropriate SolrHome directory... It will
work properly untill I restart the Solr cluster
as soon as I restart the cluster, it throws this error (refer below) and
none of the collection works properly...
ERROR - 2013-10-31 19:23:24.411; org.apache.solr.core.CoreContainer; Unable
to create core: xyz
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:834)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:625)
at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:256)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:557)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:241)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1477)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1589)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:821)
... 13 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain
timed out:
NativeFSLock@/mnt/emc/app_name/data-refresh/SolrCloud/SolrHome1/solr/xyz/data/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:673)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267)
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440)
... 15 more
ERROR - 2013-10-31 19:23:24.420; org.apache.solr.common.SolrException;
null:org.apache.solr.common.SolrException: Unable to create core: xyz
at
org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:936)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:568)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:249)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:241)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:834)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:625)
at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:256)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:557)
... 10 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1477)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1589)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:821)
... 13 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain
timed out:
NativeFSLock@/mnt/emc/app_name/data-refresh/SolrCloud/SolrHome1/solr/xyz/data/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:673)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267)
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440)
... 15 more
INFO - 2013-10-31 19:23:24.421; org.apache.solr.servlet.SolrDispatchFilter;
user.dir=/usr/wbol/glassfish3/glassfish/nodes/localhost-domain1/SolrCloud_01/config
INFO - 2013-10-31 19:23:24.421; org.apache.solr.servlet.SolrDispatchFilter;
SolrDispatchFilter.init() done
ERROR - 2013-10-31 19:23:24.556; org.apache.solr.update.SolrIndexWriter;
SolrIndexWriter was not closed prior to finalize(), indicates a bug --
POSSIBLE RESOURCE LEAK!!!
ERROR - 2013-10-31 19:23:24.558; org.apache.solr.update.SolrIndexWriter;
Error closing IndexWriter, trying rollback
java.lang.NullPointerException
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:962)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:923)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:885)
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:132)
at
org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:185)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
WARN - 2013-10-31 19:23:24.912; org.apache.solr.cloud.LeaderElector; Failed
setting watch
org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode =
NoNode for
/collections/xyz/leader_elect/shard1/election/234764442573733967-core_node2-n_0000000005
at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at org.apache.zookeeper.ZooKeeper.getData(ZooKeeper.java:1151)
at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:252)
at
org.apache.solr.common.cloud.SolrZkClient$7.execute(SolrZkClient.java:249)
at
org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:65)
at org.apache.solr.common.cloud.SolrZkClient.getData(SolrZkClient.java:249)
at
org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:117)
at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:55)
at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:129)
at
org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:519)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:495)
-------------------------------------------
It continuously tries to recover but never get success... it also deletes
collection xyz from the zookeeper
Some points to mention---
1. I have removed dataDir from solrconfig.xml as suggested by Shaun here...
http://lucene.472066.n3.nabble.com/Solr-4-3-0-Shard-instances-using-incorrect-data-directory-on-machine-boot-td4063799.html
2. I have provided absolute dataDir path in the core.properties file -
https://issues.apache.org/jira/browse/SOLR-4878
3. InstanceDir in each SolrHome have same name for every core/collection--
for example
SolrHome1/solr/xyz/conf
SolrHome1/solr/xyz/data
SolrHome1/solr/xyz/core.properties
SolrHome1/solr/pqr/conf
SolrHome1/solr/pqr/data
SolrHome1/solr/pqr/core.properties
SolrHome2/solr/xyz/conf
SolrHome2/solr/xyz/data
SolrHome2/solr/xyz/core.properties
SolrHome2/solr/pqr/conf
SolrHome2/solr/pqr/data
SolrHome2/solr/pqr/core.properties
...
3. The 4 SolrHome for each of the instances are on a single shared drive...
but are in different directories
4. All my collections and cores share the same solrconfig.xml
I am stuck with this problem since long.
Please help.
Thanks,
Kaustubh
--
View this message in context: http://lucene.472066.n3.nabble.com/unable-to-load-core-after-cluster-restart-tp4098731.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: unable to load core after cluster restart
Posted by kaustubh147 <ka...@gmail.com>.
Hi,
So finally we got our jdk/jre upgraded to 1.6.0-33. but it didnt solve the
problem. I am still seeing same write.lock error.
I was able to solve the problem by changing the lock type from native to
single.
but I am not sure about other ramifications of this approach.
Do you see any problems coming into my way with 1 shard/4 replication setup.
Thanks,
Kaustubh
--
View this message in context: http://lucene.472066.n3.nabble.com/unable-to-load-core-after-cluster-restart-tp4098731p4100538.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: unable to load core after cluster restart
Posted by kaustubh147 <ka...@gmail.com>.
Hi,
I tried "simple" lock type too. It is throwing similar error...
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain
timed out:
SimpleFSLock@/mnt/emc/app_name/data-prod-refresh/SolrCloud/SolrHome1/solr/collection1_shard1_replica2/data/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:673)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267)
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440)
I am trying to find if there is anything wrong with NFS/NFS implementation.
but similar setup works fine in QA1.
I even tried moving my Solrhome from mount drive to server's local drive...
but it yield the same error..
Thanks,
Kaustubh
--
View this message in context: http://lucene.472066.n3.nabble.com/unable-to-load-core-after-cluster-restart-tp4098731p4099687.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: unable to load core after cluster restart
Posted by Shawn Heisey <so...@elyograg.org>.
On 11/6/2013 1:18 PM, kaustubh147 wrote:
> I am attaching a small log file with debug option enabled.. log shows
> following process
>
> 1. first start of solr cluster
> 2. create collection "collection1"
> 3. shutdown cluster
> 4. start cluster again
>
> error is in only the 4th step...and it is coming after solr is trying to
> close data directory for some reason... class name is
> org.apache.solr.core.CachingDirectoryFactory..
>
> Thanks,
> Kaustubh
>
> solr_logs_for_post.txt
> <http://lucene.472066.n3.nabble.com/file/n4099641/solr_logs_for_post.txt>
Here's a relevant excerpt from that log:
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock
obtain timed out:
NativeFSLock@/mnt/emc/app_name/data-prod-refresh/SolrCloud/SolrHome3/solr/collection1/data/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:673)
at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at
org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267)
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440)
... 15 more
In a nutshell, it's trying to create the "write.lock" file in your index
directory and obtain a lock on it. Either the file already exists from
the previous Solr start, or the lock is failing.
You may need to change your lock type. Currently it's "native" but you
might need "simple" or "none" instead, for NFS to work properly.
You might also need to upgrade to NFSv4, which has much better locking
capability than earlier versions. Unfortunately it also tends to have
mounting problems, leading people to downgrade the version to 3 on their
servers.
There might be network problems, or bugs in your NFS server or NFS
client code, etc. The path suggests that the storage is an EMC device.
Check with them for possible bugs with their NFS implementation - you
might need newer software/firmware for the device. NIC firmware on your
server is another possible problem point.
Running with your index on NFS is problematic. I have never personally
tried it, but I see messages here and on IRC about problems with it. I
believe there are some people who do it successfully, perhaps someone
who's faced these problems and beat them could offer insight.
Thanks,
Shawn
Re: unable to load core after cluster restart
Posted by Shawn Heisey <so...@elyograg.org>.
On 11/6/2013 2:03 PM, Chris Geeringh wrote:
> --- In the case of Java 7, 1.7.0_40 and 1.7.0_45 have known problems with
> Lucene/Solr.
>
> Shawn, this is interesting. What are the problems, where are the documented?
https://issues.apache.org/jira/browse/LUCENE-5212
The issue comments say that one of the committers has worked around the
problem, but I'm not sure whether that fix has made it out into the wild
yet. Even if it has, JVM bugs are nasty and can be tripped by other
code besides Lucene/Solr, so it's better to completely avoid Java
versions that are known to have them.
Thanks,
Shawn
Re: unable to load core after cluster restart
Posted by Chris Geeringh <ge...@gmail.com>.
--- In the case of Java 7, 1.7.0_40 and 1.7.0_45 have known problems with
Lucene/Solr.
Shawn, this is interesting. What are the problems, where are the documented?
On 6 November 2013 20:18, kaustubh147 <ka...@gmail.com> wrote:
> Hi,
>
> Here is my
> solr.xml
>
>
>
> <solr>
>
> <solrcloud>
> <str name="host">${host:}</str>
> <int name="hostPort">28081</int>
> <str name="hostContext">/solr</str>
> <str name="zkHost">IP1:2181,IP2:2181,IP3:2181/mysolr</str>
> <int name="zkClientTimeout">15000</int>
> <bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
> </solrcloud>
>
> <shardHandlerFactory name="shardHandlerFactory"
> class="HttpShardHandlerFactory">
> <int name="socketTimeout">${socketTimeout:0}</int>
> <int name="connTimeout">${connTimeout:0}</int>
> </shardHandlerFactory>
>
> </solr>
>
>
> --------------------------------------------------------------------------------------------------------------
>
> and
> solrconfig.xml
>
>
>
> <?xml version="1.0" encoding="UTF-8" ?>
> <config>
> <luceneMatchVersion>4.5</luceneMatchVersion>
> <lib dir="../../../contrib/extraction/lib" regex=".*\.jar" />
> <lib dir="../../../dist/" regex="solr-cell-\d.*\.jar" />
> <lib dir="../../../contrib/clustering/lib/" regex=".*\.jar" />
> <lib dir="../../../dist/" regex="solr-clustering-\d.*\.jar" />
> <lib dir="../../../dist/" regex="solr-dataimporthandler-\d.*\.jar" />
> <lib dir="../../../contrib/dataimporthandler/lib/" regex=".*\.jar" />
> <lib dir="../../../contrib/langid/lib/" regex=".*\.jar" />
> <lib dir="../../../dist/" regex="solr-langid-\d.*\.jar" />
>
> <lib dir="../../../contrib/velocity/lib" regex=".*\.jar" />
> <lib dir="../../../dist/" regex="solr-velocity-\d.*\.jar" />
>
> <dataDir>${solr.data.dir:}</dataDir>
>
> <directoryFactory name="DirectoryFactory"
> class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
>
> <codecFactory class="solr.SchemaCodecFactory"/>
> <schemaFactory class="ClassicIndexSchemaFactory"/>
>
> <indexConfig>
> <writeLockTimeout>20000</writeLockTimeout>
> <lockType>${solr.lock.type:native}</lockType>
> <infoStream>true</infoStream>
> </indexConfig>
>
> <updateHandler class="solr.DirectUpdateHandler2">
>
> <updateLog>
> <str name="dir">${solr.data.dir:}</str>
> </updateLog>
>
> <autoCommit>
> <maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
> <openSearcher>false</openSearcher>
> </autoCommit>
>
> <autoSoftCommit>
> <maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
> </autoSoftCommit>
>
> </updateHandler>
>
>
> <query>
>
> <maxBooleanClauses>500000</maxBooleanClauses>
> <filterCache class="solr.FastLRUCache"
> size="512"
> initialSize="512"
> autowarmCount="0"/>
>
> <queryResultCache class="solr.LRUCache"
> size="512"
> initialSize="512"
> autowarmCount="0"/>
>
> <documentCache class="solr.LRUCache"
> size="512"
> initialSize="512"
> autowarmCount="0"/>
>
>
> <cache name="perSegFilter"
> class="solr.search.LRUCache"
> size="10"
> initialSize="0"
> autowarmCount="10"
> regenerator="solr.NoOpRegenerator" />
>
> <enableLazyFieldLoading>true</enableLazyFieldLoading>
>
> <queryResultWindowSize>20</queryResultWindowSize>
>
> <queryResultMaxDocsCached>200</queryResultMaxDocsCached>
>
> <listener event="newSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
>
> </arr>
> </listener>
> <listener event="firstSearcher" class="solr.QuerySenderListener">
> <arr name="queries">
> <lst>
> <str name="q">static firstSearcher warming in solrconfig.xml</str>
> </lst>
> </arr>
> </listener>
>
> <useColdSearcher>false</useColdSearcher>
>
> <maxWarmingSearchers>2</maxWarmingSearchers>
>
> </query>
>
> <requestDispatcher handleSelect="false" >
>
> <requestParsers enableRemoteStreaming="true"
> multipartUploadLimitInKB="2048000"
> formdataUploadLimitInKB="16384"
> addHttpRequestToContext="false"/>
>
> <httpCaching never304="true" />
>
> </requestDispatcher>
>
> <requestHandler name="/dataimport"
> class="org.apache.solr.handler.dataimport.DataImportHandler">
> <lst name="defaults">
> <str name="config">db-data-config.xml</str>
> </lst>
> </requestHandler>
>
> <requestHandler name="/select" class="solr.SearchHandler">
>
> <lst name="defaults">
> <str name="echoParams">explicit</str>
> <int name="rows">10</int>
> <str name="df">keyword</str>
> </lst>
>
> </requestHandler>
>
> <requestHandler name="/query" class="solr.SearchHandler">
> <lst name="defaults">
> <str name="echoParams">explicit</str>
> <str name="wt">json</str>
> <str name="indent">true</str>
> <str name="df">keyword</str>
> </lst>
> </requestHandler>
>
> <requestHandler name="/get" class="solr.RealTimeGetHandler">
> <lst name="defaults">
> <str name="omitHeader">true</str>
> <str name="wt">json</str>
> <str name="indent">true</str>
> </lst>
> </requestHandler>
>
> <requestHandler name="/update" class="solr.UpdateRequestHandler">
>
> </requestHandler>
>
> <requestHandler name="/update/json" class="solr.JsonUpdateRequestHandler">
> <lst name="defaults">
> <str name="stream.contentType">application/json</str>
> </lst>
> </requestHandler>
> <requestHandler name="/update/csv" class="solr.CSVRequestHandler">
> <lst name="defaults">
> <str name="stream.contentType">application/csv</str>
> </lst>
> </requestHandler>
>
> <requestHandler name="/update/extract"
> startup="lazy"
> class="solr.extraction.ExtractingRequestHandler" >
> <lst name="defaults">
> <str name="lowernames">true</str>
> <str name="uprefix">ignored_</str>
>
> <str name="captureAttr">true</str>
> <str name="fmap.a">links</str>
> <str name="fmap.div">ignored_</str>
> </lst>
> </requestHandler>
>
> <requestHandler name="/analysis/field"
> startup="lazy"
> class="solr.FieldAnalysisRequestHandler" />
>
> <requestHandler name="/analysis/document"
> class="solr.DocumentAnalysisRequestHandler"
> startup="lazy" />
>
> <requestHandler name="/admin/"
> class="solr.admin.AdminHandlers" />
>
> <requestHandler name="/admin/ping" class="solr.PingRequestHandler">
> <lst name="invariants">
> <str name="q">solrpingquery</str>
> </lst>
> <lst name="defaults">
> <str name="echoParams">all</str>
> </lst>
>
> </requestHandler>
>
> <requestHandler name="/debug/dump" class="solr.DumpRequestHandler" >
> <lst name="defaults">
> <str name="echoParams">explicit</str>
> <str name="echoHandler">true</str>
> </lst>
> </requestHandler>
>
> <requestHandler name="/replication" class="solr.ReplicationHandler" >
>
> </requestHandler>
>
> <searchComponent name="spellcheck" class="solr.SpellCheckComponent">
>
> <str name="queryAnalyzerFieldType">text_general</str>
>
> <lst name="spellchecker">
> <str name="name">default</str>
> <str name="field">keyword</str>
> <str name="classname">solr.DirectSolrSpellChecker</str>
>
> <str name="distanceMeasure">internal</str>
>
> <float name="accuracy">0.5</float>
>
> <int name="maxEdits">2</int>
>
> <int name="minPrefix">1</int>
>
> <int name="maxInspections">5</int>
>
> <int name="minQueryLength">4</int>
>
> <float name="maxQueryFrequency">0.01</float>
>
> </lst>
>
>
> <lst name="spellchecker">
> <str name="name">wordbreak</str>
> <str name="classname">solr.WordBreakSolrSpellChecker</str>
> <str name="field">name</str>
> <str name="combineWords">true</str>
> <str name="breakWords">true</str>
> <int name="maxChanges">10</int>
> </lst>
>
> </searchComponent>
>
> <requestHandler name="/spell" class="solr.SearchHandler" startup="lazy">
> <lst name="defaults">
> <str name="df">keyword</str>
>
> <str name="spellcheck.dictionary">default</str>
> <str name="spellcheck.dictionary">wordbreak</str>
> <str name="spellcheck">on</str>
> <str name="spellcheck.extendedResults">true</str>
> <str name="spellcheck.count">10</str>
> <str name="spellcheck.alternativeTermCount">5</str>
> <str name="spellcheck.maxResultsForSuggest">5</str>
> <str name="spellcheck.collate">true</str>
> <str name="spellcheck.collateExtendedResults">true</str>
> <str name="spellcheck.maxCollationTries">10</str>
> <str name="spellcheck.maxCollations">5</str>
> </lst>
> <arr name="last-components">
> <str>spellcheck</str>
> </arr>
> </requestHandler>
>
> <searchComponent name="tvComponent" class="solr.TermVectorComponent"/>
>
> <requestHandler name="/tvrh" class="solr.SearchHandler" startup="lazy">
> <lst name="defaults">
> <str name="df">keyword</str>
> <bool name="tv">true</bool>
> </lst>
> <arr name="last-components">
> <str>tvComponent</str>
> </arr>
> </requestHandler>
>
> <searchComponent name="clustering"
> enable="${solr.clustering.enabled:true}"
> class="solr.clustering.ClusteringComponent" >
>
> <lst name="engine">
> <str name="name">lingo</str>
>
> <str
>
> name="carrot.algorithm">org.carrot2.clustering.lingo.LingoClusteringAlgorithm</str>
>
> <str name="carrot.resourcesDir">clustering/carrot2</str>
> </lst>
>
>
> <lst name="engine">
> <str name="name">stc</str>
> <str
>
> name="carrot.algorithm">org.carrot2.clustering.stc.STCClusteringAlgorithm</str>
> </lst>
>
>
> <lst name="engine">
> <str name="name">kmeans</str>
> <str
>
> name="carrot.algorithm">org.carrot2.clustering.kmeans.BisectingKMeansClusteringAlgorithm</str>
> </lst>
> </searchComponent>
>
> <requestHandler name="/clustering"
> startup="lazy"
> enable="${solr.clustering.enabled:true}"
> class="solr.SearchHandler">
> <lst name="defaults">
> <bool name="clustering">true</bool>
> <bool name="clustering.results">true</bool>
>
>
> <str name="carrot.title">name</str>
>
> <str name="carrot.url">id</str>
>
> <str name="carrot.snippet">features</str>
>
> <bool name="carrot.produceSummary">true</bool>
>
>
>
> <bool name="carrot.outputSubClusters">false</bool>
>
>
> <str name="defType">edismax</str>
> <str name="qf">
> keyword^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
> </str>
> <str name="q.alt">*:*</str>
> <str name="rows">10</str>
> <str name="fl">*,score</str>
> </lst>
> <arr name="last-components">
> <str>clustering</str>
> </arr>
> </requestHandler>
>
> <searchComponent name="terms" class="solr.TermsComponent"/>
>
>
> <requestHandler name="/js"
> class="org.apache.solr.handler.js.JavaScriptRequestHandler"
> startup="lazy"/>
> <requestHandler name="/terms" class="solr.SearchHandler" startup="lazy">
> <lst name="defaults">
> <bool name="terms">true</bool>
> <bool name="distrib">false</bool>
> </lst>
> <arr name="components">
> <str>terms</str>
> </arr>
> </requestHandler>
>
> <searchComponent name="elevator" class="solr.QueryElevationComponent" >
>
> <str name="queryFieldType">string</str>
> <str name="config-file">elevate.xml</str>
> </searchComponent>
>
>
> <requestHandler name="/elevate" class="solr.SearchHandler" startup="lazy">
> <lst name="defaults">
> <str name="echoParams">explicit</str>
> <str name="df">keyword</str>
> </lst>
> <arr name="last-components">
> <str>elevator</str>
> </arr>
> </requestHandler>
>
> <searchComponent class="solr.HighlightComponent" name="highlight">
> <highlighting>
>
>
> <fragmenter name="gap"
> default="true"
> class="solr.highlight.GapFragmenter">
> <lst name="defaults">
> <int name="hl.fragsize">100</int>
> </lst>
> </fragmenter>
>
> <fragmenter name="regex"
> class="solr.highlight.RegexFragmenter">
> <lst name="defaults">
>
> <int name="hl.fragsize">70</int>
>
> <float name="hl.regex.slop">0.5</float>
>
> <str name="hl.regex.pattern">[-\w ,/\n\"']{20,200}</str>
> </lst>
> </fragmenter>
>
> <formatter name="html"
> default="true"
> class="solr.highlight.HtmlFormatter">
> <lst name="defaults">
> <str name="hl.simple.pre"></str>
> <str name="hl.simple.post"></str>
> </lst>
> </formatter>
>
> <encoder name="html"
> class="solr.highlight.HtmlEncoder" />
>
> <fragListBuilder name="simple"
> class="solr.highlight.SimpleFragListBuilder"/>
>
> <fragListBuilder name="single"
> class="solr.highlight.SingleFragListBuilder"/>
>
> <fragListBuilder name="weighted"
> default="true"
> class="solr.highlight.WeightedFragListBuilder"/>
>
> <fragmentsBuilder name="default"
> default="true"
> class="solr.highlight.ScoreOrderFragmentsBuilder">
>
> </fragmentsBuilder>
>
>
> <fragmentsBuilder name="colored"
> class="solr.highlight.ScoreOrderFragmentsBuilder">
> <lst name="defaults">
> <str name="hl.tag.pre"></str>
> <str name="hl.tag.post"></str>
> </lst>
> </fragmentsBuilder>
>
> <boundaryScanner name="default"
> default="true"
> class="solr.highlight.SimpleBoundaryScanner">
> <lst name="defaults">
> <str name="hl.bs.maxScan">10</str>
> <str name="hl.bs.chars">.,!? 	 </str>
> </lst>
> </boundaryScanner>
>
> <boundaryScanner name="breakIterator"
> class="solr.highlight.BreakIteratorBoundaryScanner">
> <lst name="defaults">
>
> <str name="hl.bs.type">WORD</str>
>
> <str name="hl.bs.language">en</str>
> <str name="hl.bs.country">US</str>
> </lst>
> </boundaryScanner>
> </highlighting>
> </searchComponent>
>
> <queryResponseWriter name="json" class="solr.JSONResponseWriter">
>
> <str name="content-type">text/plain; charset=UTF-8</str>
> </queryResponseWriter>
>
> <queryResponseWriter name="velocity" class="solr.VelocityResponseWriter"
> startup="lazy"/>
>
> <queryResponseWriter name="xslt" class="solr.XSLTResponseWriter">
> <int name="xsltCacheLifetimeSeconds">5</int>
> </queryResponseWriter>
>
> <admin>
> <defaultQuery>*:*</defaultQuery>
> </admin>
>
> </config>
>
>
>
> ---------------------------------------------------------------------------------------------------------------------------
>
>
> I am attaching a small log file with debug option enabled.. log shows
> following process
>
> 1. first start of solr cluster
> 2. create collection "collection1"
> 3. shutdown cluster
> 4. start cluster again
>
> error is in only the 4th step...and it is coming after solr is trying to
> close data directory for some reason... class name is
> org.apache.solr.core.CachingDirectoryFactory..
>
> Thanks,
> Kaustubh
>
> solr_logs_for_post.txt
> <http://lucene.472066.n3.nabble.com/file/n4099641/solr_logs_for_post.txt>
>
>
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/unable-to-load-core-after-cluster-restart-tp4098731p4099641.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: unable to load core after cluster restart
Posted by kaustubh147 <ka...@gmail.com>.
Hi,
Here is my
solr.xml
<solr>
<solrcloud>
<str name="host">${host:}</str>
<int name="hostPort">28081</int>
<str name="hostContext">/solr</str>
<str name="zkHost">IP1:2181,IP2:2181,IP3:2181/mysolr</str>
<int name="zkClientTimeout">15000</int>
<bool name="genericCoreNodeNames">${genericCoreNodeNames:true}</bool>
</solrcloud>
<shardHandlerFactory name="shardHandlerFactory"
class="HttpShardHandlerFactory">
<int name="socketTimeout">${socketTimeout:0}</int>
<int name="connTimeout">${connTimeout:0}</int>
</shardHandlerFactory>
</solr>
--------------------------------------------------------------------------------------------------------------
and
solrconfig.xml
<?xml version="1.0" encoding="UTF-8" ?>
<config>
<luceneMatchVersion>4.5</luceneMatchVersion>
<lib dir="../../../contrib/extraction/lib" regex=".*\.jar" />
<lib dir="../../../dist/" regex="solr-cell-\d.*\.jar" />
<lib dir="../../../contrib/clustering/lib/" regex=".*\.jar" />
<lib dir="../../../dist/" regex="solr-clustering-\d.*\.jar" />
<lib dir="../../../dist/" regex="solr-dataimporthandler-\d.*\.jar" />
<lib dir="../../../contrib/dataimporthandler/lib/" regex=".*\.jar" />
<lib dir="../../../contrib/langid/lib/" regex=".*\.jar" />
<lib dir="../../../dist/" regex="solr-langid-\d.*\.jar" />
<lib dir="../../../contrib/velocity/lib" regex=".*\.jar" />
<lib dir="../../../dist/" regex="solr-velocity-\d.*\.jar" />
<dataDir>${solr.data.dir:}</dataDir>
<directoryFactory name="DirectoryFactory"
class="${solr.directoryFactory:solr.NRTCachingDirectoryFactory}"/>
<codecFactory class="solr.SchemaCodecFactory"/>
<schemaFactory class="ClassicIndexSchemaFactory"/>
<indexConfig>
<writeLockTimeout>20000</writeLockTimeout>
<lockType>${solr.lock.type:native}</lockType>
<infoStream>true</infoStream>
</indexConfig>
<updateHandler class="solr.DirectUpdateHandler2">
<updateLog>
<str name="dir">${solr.data.dir:}</str>
</updateLog>
<autoCommit>
<maxTime>${solr.autoCommit.maxTime:15000}</maxTime>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>${solr.autoSoftCommit.maxTime:-1}</maxTime>
</autoSoftCommit>
</updateHandler>
<query>
<maxBooleanClauses>500000</maxBooleanClauses>
<filterCache class="solr.FastLRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<queryResultCache class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<documentCache class="solr.LRUCache"
size="512"
initialSize="512"
autowarmCount="0"/>
<cache name="perSegFilter"
class="solr.search.LRUCache"
size="10"
initialSize="0"
autowarmCount="10"
regenerator="solr.NoOpRegenerator" />
<enableLazyFieldLoading>true</enableLazyFieldLoading>
<queryResultWindowSize>20</queryResultWindowSize>
<queryResultMaxDocsCached>200</queryResultMaxDocsCached>
<listener event="newSearcher" class="solr.QuerySenderListener">
<arr name="queries">
</arr>
</listener>
<listener event="firstSearcher" class="solr.QuerySenderListener">
<arr name="queries">
<lst>
<str name="q">static firstSearcher warming in solrconfig.xml</str>
</lst>
</arr>
</listener>
<useColdSearcher>false</useColdSearcher>
<maxWarmingSearchers>2</maxWarmingSearchers>
</query>
<requestDispatcher handleSelect="false" >
<requestParsers enableRemoteStreaming="true"
multipartUploadLimitInKB="2048000"
formdataUploadLimitInKB="16384"
addHttpRequestToContext="false"/>
<httpCaching never304="true" />
</requestDispatcher>
<requestHandler name="/dataimport"
class="org.apache.solr.handler.dataimport.DataImportHandler">
<lst name="defaults">
<str name="config">db-data-config.xml</str>
</lst>
</requestHandler>
<requestHandler name="/select" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<int name="rows">10</int>
<str name="df">keyword</str>
</lst>
</requestHandler>
<requestHandler name="/query" class="solr.SearchHandler">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="wt">json</str>
<str name="indent">true</str>
<str name="df">keyword</str>
</lst>
</requestHandler>
<requestHandler name="/get" class="solr.RealTimeGetHandler">
<lst name="defaults">
<str name="omitHeader">true</str>
<str name="wt">json</str>
<str name="indent">true</str>
</lst>
</requestHandler>
<requestHandler name="/update" class="solr.UpdateRequestHandler">
</requestHandler>
<requestHandler name="/update/json" class="solr.JsonUpdateRequestHandler">
<lst name="defaults">
<str name="stream.contentType">application/json</str>
</lst>
</requestHandler>
<requestHandler name="/update/csv" class="solr.CSVRequestHandler">
<lst name="defaults">
<str name="stream.contentType">application/csv</str>
</lst>
</requestHandler>
<requestHandler name="/update/extract"
startup="lazy"
class="solr.extraction.ExtractingRequestHandler" >
<lst name="defaults">
<str name="lowernames">true</str>
<str name="uprefix">ignored_</str>
<str name="captureAttr">true</str>
<str name="fmap.a">links</str>
<str name="fmap.div">ignored_</str>
</lst>
</requestHandler>
<requestHandler name="/analysis/field"
startup="lazy"
class="solr.FieldAnalysisRequestHandler" />
<requestHandler name="/analysis/document"
class="solr.DocumentAnalysisRequestHandler"
startup="lazy" />
<requestHandler name="/admin/"
class="solr.admin.AdminHandlers" />
<requestHandler name="/admin/ping" class="solr.PingRequestHandler">
<lst name="invariants">
<str name="q">solrpingquery</str>
</lst>
<lst name="defaults">
<str name="echoParams">all</str>
</lst>
</requestHandler>
<requestHandler name="/debug/dump" class="solr.DumpRequestHandler" >
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="echoHandler">true</str>
</lst>
</requestHandler>
<requestHandler name="/replication" class="solr.ReplicationHandler" >
</requestHandler>
<searchComponent name="spellcheck" class="solr.SpellCheckComponent">
<str name="queryAnalyzerFieldType">text_general</str>
<lst name="spellchecker">
<str name="name">default</str>
<str name="field">keyword</str>
<str name="classname">solr.DirectSolrSpellChecker</str>
<str name="distanceMeasure">internal</str>
<float name="accuracy">0.5</float>
<int name="maxEdits">2</int>
<int name="minPrefix">1</int>
<int name="maxInspections">5</int>
<int name="minQueryLength">4</int>
<float name="maxQueryFrequency">0.01</float>
</lst>
<lst name="spellchecker">
<str name="name">wordbreak</str>
<str name="classname">solr.WordBreakSolrSpellChecker</str>
<str name="field">name</str>
<str name="combineWords">true</str>
<str name="breakWords">true</str>
<int name="maxChanges">10</int>
</lst>
</searchComponent>
<requestHandler name="/spell" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="df">keyword</str>
<str name="spellcheck.dictionary">default</str>
<str name="spellcheck.dictionary">wordbreak</str>
<str name="spellcheck">on</str>
<str name="spellcheck.extendedResults">true</str>
<str name="spellcheck.count">10</str>
<str name="spellcheck.alternativeTermCount">5</str>
<str name="spellcheck.maxResultsForSuggest">5</str>
<str name="spellcheck.collate">true</str>
<str name="spellcheck.collateExtendedResults">true</str>
<str name="spellcheck.maxCollationTries">10</str>
<str name="spellcheck.maxCollations">5</str>
</lst>
<arr name="last-components">
<str>spellcheck</str>
</arr>
</requestHandler>
<searchComponent name="tvComponent" class="solr.TermVectorComponent"/>
<requestHandler name="/tvrh" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="df">keyword</str>
<bool name="tv">true</bool>
</lst>
<arr name="last-components">
<str>tvComponent</str>
</arr>
</requestHandler>
<searchComponent name="clustering"
enable="${solr.clustering.enabled:true}"
class="solr.clustering.ClusteringComponent" >
<lst name="engine">
<str name="name">lingo</str>
<str
name="carrot.algorithm">org.carrot2.clustering.lingo.LingoClusteringAlgorithm</str>
<str name="carrot.resourcesDir">clustering/carrot2</str>
</lst>
<lst name="engine">
<str name="name">stc</str>
<str
name="carrot.algorithm">org.carrot2.clustering.stc.STCClusteringAlgorithm</str>
</lst>
<lst name="engine">
<str name="name">kmeans</str>
<str
name="carrot.algorithm">org.carrot2.clustering.kmeans.BisectingKMeansClusteringAlgorithm</str>
</lst>
</searchComponent>
<requestHandler name="/clustering"
startup="lazy"
enable="${solr.clustering.enabled:true}"
class="solr.SearchHandler">
<lst name="defaults">
<bool name="clustering">true</bool>
<bool name="clustering.results">true</bool>
<str name="carrot.title">name</str>
<str name="carrot.url">id</str>
<str name="carrot.snippet">features</str>
<bool name="carrot.produceSummary">true</bool>
<bool name="carrot.outputSubClusters">false</bool>
<str name="defType">edismax</str>
<str name="qf">
keyword^0.5 features^1.0 name^1.2 sku^1.5 id^10.0 manu^1.1 cat^1.4
</str>
<str name="q.alt">*:*</str>
<str name="rows">10</str>
<str name="fl">*,score</str>
</lst>
<arr name="last-components">
<str>clustering</str>
</arr>
</requestHandler>
<searchComponent name="terms" class="solr.TermsComponent"/>
<requestHandler name="/js"
class="org.apache.solr.handler.js.JavaScriptRequestHandler" startup="lazy"/>
<requestHandler name="/terms" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<bool name="terms">true</bool>
<bool name="distrib">false</bool>
</lst>
<arr name="components">
<str>terms</str>
</arr>
</requestHandler>
<searchComponent name="elevator" class="solr.QueryElevationComponent" >
<str name="queryFieldType">string</str>
<str name="config-file">elevate.xml</str>
</searchComponent>
<requestHandler name="/elevate" class="solr.SearchHandler" startup="lazy">
<lst name="defaults">
<str name="echoParams">explicit</str>
<str name="df">keyword</str>
</lst>
<arr name="last-components">
<str>elevator</str>
</arr>
</requestHandler>
<searchComponent class="solr.HighlightComponent" name="highlight">
<highlighting>
<fragmenter name="gap"
default="true"
class="solr.highlight.GapFragmenter">
<lst name="defaults">
<int name="hl.fragsize">100</int>
</lst>
</fragmenter>
<fragmenter name="regex"
class="solr.highlight.RegexFragmenter">
<lst name="defaults">
<int name="hl.fragsize">70</int>
<float name="hl.regex.slop">0.5</float>
<str name="hl.regex.pattern">[-\w ,/\n\"']{20,200}</str>
</lst>
</fragmenter>
<formatter name="html"
default="true"
class="solr.highlight.HtmlFormatter">
<lst name="defaults">
<str name="hl.simple.pre"></str>
<str name="hl.simple.post"></str>
</lst>
</formatter>
<encoder name="html"
class="solr.highlight.HtmlEncoder" />
<fragListBuilder name="simple"
class="solr.highlight.SimpleFragListBuilder"/>
<fragListBuilder name="single"
class="solr.highlight.SingleFragListBuilder"/>
<fragListBuilder name="weighted"
default="true"
class="solr.highlight.WeightedFragListBuilder"/>
<fragmentsBuilder name="default"
default="true"
class="solr.highlight.ScoreOrderFragmentsBuilder">
</fragmentsBuilder>
<fragmentsBuilder name="colored"
class="solr.highlight.ScoreOrderFragmentsBuilder">
<lst name="defaults">
<str name="hl.tag.pre"></str>
<str name="hl.tag.post"></str>
</lst>
</fragmentsBuilder>
<boundaryScanner name="default"
default="true"
class="solr.highlight.SimpleBoundaryScanner">
<lst name="defaults">
<str name="hl.bs.maxScan">10</str>
<str name="hl.bs.chars">.,!? 	 </str>
</lst>
</boundaryScanner>
<boundaryScanner name="breakIterator"
class="solr.highlight.BreakIteratorBoundaryScanner">
<lst name="defaults">
<str name="hl.bs.type">WORD</str>
<str name="hl.bs.language">en</str>
<str name="hl.bs.country">US</str>
</lst>
</boundaryScanner>
</highlighting>
</searchComponent>
<queryResponseWriter name="json" class="solr.JSONResponseWriter">
<str name="content-type">text/plain; charset=UTF-8</str>
</queryResponseWriter>
<queryResponseWriter name="velocity" class="solr.VelocityResponseWriter"
startup="lazy"/>
<queryResponseWriter name="xslt" class="solr.XSLTResponseWriter">
<int name="xsltCacheLifetimeSeconds">5</int>
</queryResponseWriter>
<admin>
<defaultQuery>*:*</defaultQuery>
</admin>
</config>
---------------------------------------------------------------------------------------------------------------------------
I am attaching a small log file with debug option enabled.. log shows
following process
1. first start of solr cluster
2. create collection "collection1"
3. shutdown cluster
4. start cluster again
error is in only the 4th step...and it is coming after solr is trying to
close data directory for some reason... class name is
org.apache.solr.core.CachingDirectoryFactory..
Thanks,
Kaustubh
solr_logs_for_post.txt
<http://lucene.472066.n3.nabble.com/file/n4099641/solr_logs_for_post.txt>
--
View this message in context: http://lucene.472066.n3.nabble.com/unable-to-load-core-after-cluster-restart-tp4098731p4099641.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: unable to load core after cluster restart
Posted by Shawn Heisey <so...@elyograg.org>.
On 11/6/2013 11:53 AM, kaustubh147 wrote:
> Hi All,
>
> I have further investigated the difference between both the environments.
>
> We have JDK 1.6.0_17 (VM 14.3-b01)on UAT and JDK 1.6.0_33 (VM 20.8-b03)on
> QA1. Can it be the reason behind this error?
>
> Is there a recommended jdk version for SolrCloud ?
For Solr 4.x, Java 6 is required, and Java 7 works just fine. Java 6 is
no longer receiving unpaid support from Oracle, and there will likely be
no more public versions available, which makes Java 7 attractive.
You don't need the JDK, the JRE is sufficient, but having the JDK is not
a problem. Oracle's product is recommended for Java 6. For Java 7,
OpenJDK is probably sufficient, because for that version OpenJDK is the
reference implementation.
Naturally, the latest version is always recommended, at least for Java
6. In the case of Java 7, 1.7.0_40 and 1.7.0_45 have known problems
with Lucene/Solr. For Java 7, 1.7.0_25 is currently the recommended
version, and hopefully Oracle will be fixing their bug in the next release.
I don't know what might be causing the problems you're seeing when
following my advice. You'd have to give a lot of config details and the
errors you're seeing.
Thanks,
Shawn
Re: unable to load core after cluster restart
Posted by kaustubh147 <ka...@gmail.com>.
Hi All,
I have further investigated the difference between both the environments.
We have JDK 1.6.0_17 (VM 14.3-b01)on UAT and JDK 1.6.0_33 (VM 20.8-b03)on
QA1. Can it be the reason behind this error?
Is there a recommended jdk version for SolrCloud ?
Thanks,
Kaustubh
--
View this message in context: http://lucene.472066.n3.nabble.com/unable-to-load-core-after-cluster-restart-tp4098731p4099621.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: unable to load core after cluster restart
Posted by kaustubh147 <ka...@gmail.com>.
Hi Shawn,
One thing I forget to mention here is the same setup (with no bootstrap) is
working fine in our QA1 environment. I did not have the bootstrap option
from start, I added it thinking it will solve the problem.
Nonetheless I followed Shawn's instructions, wherever it differed from my
old approach...
1. I moved my zkHost from JVM to solr.xml and added chroot in it
2. removed bootstrap option
3. created collections with URL template suggested (I have tried it earlier
too)
None of it worked for me... I am seeing same errors.. I am adding some more
logs before and after the error occurs
-------------------------------------------------
INFO - 2013-11-02 17:40:40.427;
org.apache.solr.update.DefaultSolrCoreState; closing IndexWriter with
IndexWriterCloser
INFO - 2013-11-02 17:40:40.428; org.apache.solr.core.SolrCore; [xyz]
Closing main searcher on request.
INFO - 2013-11-02 17:40:40.431;
org.apache.solr.core.CachingDirectoryFactory; Closing
NRTCachingDirectoryFactory - 1 directories currently being tracked
INFO - 2013-11-02 17:40:40.432;
org.apache.solr.core.CachingDirectoryFactory; looking to close
/mnt/emc/App_name/data-UAT-refresh/SolrCloud/SolrHome2/solr/xyz/data
[CachedDir<<refCount=0;path=/mnt/emc/App_name/data-UAT-refresh/SolrCloud/SolrHome2/solr/xyz/data;done=false>>]
INFO - 2013-11-02 17:40:40.432;
org.apache.solr.core.CachingDirectoryFactory; Closing directory:
/mnt/emc/App_name/data-UAT-refresh/SolrCloud/SolrHome2/solr/xyz/data
ERROR - 2013-11-02 17:40:40.433; org.apache.solr.core.CoreContainer; Unable
to create core: xyz
org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:834)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:625)
at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:256)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:555)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:247)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:239)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1477)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1589)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:821)
... 13 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain
timed out:
NativeFSLock@/mnt/emc/App_name/data-UAT-refresh/SolrCloud/SolrHome2/solr/xyz/data/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:695)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267)
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440)
... 15 more
ERROR - 2013-11-02 17:40:40.443; org.apache.solr.common.SolrException;
null:org.apache.solr.common.SolrException: Unable to create core: xyz
at
org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:934)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:566)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:247)
at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:239)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:834)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:625)
at org.apache.solr.core.ZkContainer.createFromZk(ZkContainer.java:256)
at org.apache.solr.core.CoreContainer.create(CoreContainer.java:555)
... 10 more
Caused by: org.apache.solr.common.SolrException: Error opening new searcher
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1477)
at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1589)
at org.apache.solr.core.SolrCore.<init>(SolrCore.java:821)
... 13 more
Caused by: org.apache.lucene.store.LockObtainFailedException: Lock obtain
timed out:
NativeFSLock@/mnt/emc/App_name/data-UAT-refresh/SolrCloud/SolrHome2/solr/xyz/data/index/write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:84)
at org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:695)
at org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:77)
at org.apache.solr.update.SolrIndexWriter.create(SolrIndexWriter.java:64)
at
org.apache.solr.update.DefaultSolrCoreState.createMainIndexWriter(DefaultSolrCoreState.java:267)
at
org.apache.solr.update.DefaultSolrCoreState.getIndexWriter(DefaultSolrCoreState.java:110)
at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1440)
... 15 more
INFO - 2013-11-02 17:40:40.445; org.apache.solr.servlet.SolrDispatchFilter;
user.dir=/usr/wbol/glassfish3/glassfish/nodes/UAT-refresh-App_name-app02/SolrCloud_02/config
INFO - 2013-11-02 17:40:40.446; org.apache.solr.servlet.SolrDispatchFilter;
SolrDispatchFilter.init() done
ERROR - 2013-11-02 17:40:40.609; org.apache.solr.update.SolrIndexWriter;
SolrIndexWriter was not closed prior to finalize(), indicates a bug --
POSSIBLE RESOURCE LEAK!!!
ERROR - 2013-11-02 17:40:40.627; org.apache.solr.update.SolrIndexWriter;
Error closing IndexWriter, trying rollback
java.lang.NullPointerException
at org.apache.lucene.index.IndexWriter.closeInternal(IndexWriter.java:984)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:945)
at org.apache.lucene.index.IndexWriter.close(IndexWriter.java:907)
at org.apache.solr.update.SolrIndexWriter.close(SolrIndexWriter.java:132)
at
org.apache.solr.update.SolrIndexWriter.finalize(SolrIndexWriter.java:185)
at java.lang.ref.Finalizer.invokeFinalizeMethod(Native Method)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:83)
at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
INFO - 2013-11-02 17:40:41.928;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
INFO - 2013-11-02 17:40:41.928;
org.apache.solr.common.cloud.ZkStateReader$2; A cluster state change:
WatchedEvent state:SyncConnected type:NodeDataChanged
path:/clusterstate.json, has occurred - updating... (live nodes size: 4)
INFO - 2013-11-02 17:40:42.266; org.apache.solr.servlet.SolrDispatchFilter;
[admin] webapp=null path=/admin/cores
params={indexInfo=false&_=1383439243017&wt=json} status=0 QTime=2
INFO - 2013-11-02 17:40:42.408; org.apache.solr.servlet.SolrDispatchFilter;
[admin] webapp=null path=/admin/info/system params={_=1383439243093&wt=json}
status=0 QTime=90
INFO - 2013-11-02 17:40:43.554; org.apache.solr.servlet.SolrDispatchFilter;
[admin] webapp=null path=/admin/info/logging
params={_=1383439244330&since=0&wt=json} status=0 QTime=14
------------------------------------------------------------------------
Even if I shut one of the node in the cluster, it does not recover and
throws the same error....
I did some more investigation and compared it with our QA1 environment...
I found out that....
1. when I start the cluster for the first time... and add cores into it...
regardless from Admin console or through URL.... it runs identical in both
QA1 and UAT
2. once I shutdown the cluster.... I see some files in zookeeper overseer
section in QA1 (which is running propertly) but does not show up in UAT
[zkhost:2181(CONNECTED) 3] ls /overseer/queue
[qn-0000004634, qn-0000004633, qn-0000004632, qn-0000004631, qn-0000004630,
qn-0000004616, qn-0000004615, qn-0000004619, qn-0000004618, qn-0000004617,
qn-0000004621, qn-0000004620, qn-0000004623, qn-0000004622, qn-0000004625,
qn-0000004624, qn-0000004627, qn-0000004626, qn-0000004629, qn-0000004628]
These files disappear as soon as the cluster starts... I think these files
help SolrCloud to recover.....
3. Moreover when I shutdown the cluster , the collection in zookeeper under
collections directory disappears in UAT environment but stays in QA1
environment
[zkhost:2181(CONNECTED) 3] ls /collections
4. lifecycle of index write.lock..... in UAT environment
1. when core first gets created - write.lock appears with 2 empty segment
files
2. write.lock stays during dataimport and subsequent searches
3. after cluster shut down - write.lock disappears...
4. during cluster restart write.lock appears again - and eventually becomes
the reason of the error.....
5. sometimes, one or two out of the 4 instances recoveres
I tried upgrading to 4.5.1, cause I saw some overseer related bug fixes in
4.5.1. but that also didnt turn the things right..... I see the same errors
in it too
Should I go back to 4.0/4.4.... did anybody else faced these problems???
Please help..
Thanks,
Kaustubh
--
View this message in context: http://lucene.472066.n3.nabble.com/unable-to-load-core-after-cluster-restart-tp4098731p4098991.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: unable to load core after cluster restart
Posted by Shawn Heisey <so...@elyograg.org>.
On 10/31/2013 9:18 PM, kaustubh147 wrote:
> Glassfish 3.1.2.2
> Solr 4.5
> Zookeeper 3.4.5
>
> We have set up a SolrCloud with 4 Solr nodes and 3 zookeeper instances.
>
> I start the cluster for the first time with bootstrap_conf= true.... All the
> nodes starts property.. I am creating cores (with the same name) on all 4
> instances. I can add multiple cores on each of the instances... logically I
> have 5 collections.
>
> Now i am creating indexes.. and it automatically creates 4 copies of the
> index, one for each instance in appropriate SolrHome directory... It will
> work properly untill I restart the Solr cluster
>
> as soon as I restart the cluster, it throws this error (refer below) and
> none of the collection works properly...
It's having problems with the index locking. Further down, you talk
about shared directories. I assume you're using a network filesystem,
like NFS. Lucene and Solr don't work well with network file systems,
and NFS in particular.
http://stackoverflow.com/questions/9599529/solr-over-nfs-problems
For SolrCloud, using the bootstrap options tends to create a lot of
confusion.
> 1. I have removed dataDir from solrconfig.xml as suggested by Shaun here...
>
> http://lucene.472066.n3.nabble.com/Solr-4-3-0-Shard-instances-using-incorrect-data-directory-on-machine-boot-td4063799.html
>
> 2. I have provided absolute dataDir path in the core.properties file -
> https://issues.apache.org/jira/browse/SOLR-4878
>
> 3. InstanceDir in each SolrHome have same name for every core/collection--
> for example
>
> SolrHome1/solr/xyz/conf
> SolrHome1/solr/xyz/data
> SolrHome1/solr/xyz/core.properties
> SolrHome1/solr/pqr/conf
> SolrHome1/solr/pqr/data
> SolrHome1/solr/pqr/core.properties
>
>
> SolrHome2/solr/xyz/conf
> SolrHome2/solr/xyz/data
> SolrHome2/solr/xyz/core.properties
> SolrHome2/solr/pqr/conf
> SolrHome2/solr/pqr/data
> SolrHome2/solr/pqr/core.properties
>
> ...
>
> 3. The 4 SolrHome for each of the instances are on a single shared drive...
> but are in different directories
>
> 4. All my collections and cores share the same solrconfig.xml
The fact that you're using SolrCloud changes things quite a bit. I
notice that you have a "conf" directory on all your cores. I'm very
curious as to why -- because with SolrCloud, the config that's used
isn't on the disk, it's in zookeeper ... and when you create a
collection from scratch, the cores will NOT contain a conf directory.
IMHO, you shouldn't be trying to control things like dataDir with
SolrCloud. In fact, you should let SolrCloud control practically all
aspects of your cores. Here's some more stuff that's my opinion about
how a SolrCloud should be begun and managed:
Set the solr home, either with a solr.solr.home java system property or
the JNDI property solr/home. In solr.xml, include settings for the
hostPort and hostContext, and include a zkHost parameter as well with
the following format, where "mysolr" is a zookeeper chroot and can be
anything you want it to be:
zoo1.example.com:2181,zoo2.example.com:2181,zoo3.example.com/mysolr
Upload one or more configs to zookeeper using the zkcli "upconfig" command.
The first time you start each Solr server, it should have no cores at
all, and you should never use any bootstrap options.
Once all servers are started, create a collection using the following
URL as a template:
http://server:port/admin/collections?action=CREATE&name=mytest&numShards=X&replicationFactor=N&collection.configName=mycfg
Use appropriate values for name, X and N, and N should be at least 2
unless you're building a dev cluster that doesn't need redundancy. For
the collection.configName, use a config name that you uploaded earlier
with zkcli.
Thanks,
Shawn