You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by dipti khullar <di...@gmail.com> on 2010/01/11 07:38:29 UTC
Could not start SOLR issue
Hi
We are running master/slave Solr 1.3 version on production since about 5
months.
Yesterday, we faced following issue on one of the slaves for the first time
because of which we had to restart the slave.
SEVERE: Could not start SOLR. Check solr/home property
java.lang.RuntimeException: java.io.FileNotFoundException: no segments* file
found in org.apache.lucene.store.FSDirectory@/opt/solr/solr_slave/solr/data/index:
files: null
I searched on forums but couldn't find any relevant info which could have
possibly caused the issue.
In snapinstaller logs, following failed logs were observed:
2010/01/11 04:20:06 started by solr
2010/01/11 04:20:06 command:
/opt/solr/solr_slave/solr/solr/bin/snapinstaller
2010/01/11 04:20:07 installing snapshot
/opt/solr/solr_slave/solr/data/snapshot.20100111041402
2010/01/11 04:20:07 notifing Solr to open a new Searcher
2010/01/11 04:20:07 failed to connect to Solr server
2010/01/11 04:20:07 snapshot installed but Solr server has not open a new
Searcher
2010/01/11 04:20:08 failed (elapsed time: 1 sec)
Configurations:
There are 2 search servers in a virtualized VMware environment. Each has 2
instances of Solr running on separates ports in tomcat.
Server 1: hosts 1 master(application 1), 1 slave (application 1)
Server 2: hosta 1 master (application 2), 1 slave (application 1)
Both servers have 4 CPUs and 4 GB RAM.
Master
- 4GB RAM
- 1GB JVM Heap memory is allocated to Solr
Slave1/Slave2:
- 4GB RAM
- 2GB JVM Heap memory is allocated to Solr
Can there be any possible reasons that solr/home property couldn't be found?
Thanks
Dipti
RE: update solr index
Posted by Marc Des Garets <ma...@192.com>.
I have 2 ways to update the index, either I use solrj using
SolrEmbeddedServer or I do it with an http query. If I do it with an
http query I indeed don't stop tomcat but I have to do some operations
(mainly taking instance out of the cluster) and I can't automate this
process when I can automate update which is my goal that's why I'm
trying to do update without having problems with garbage collection.
I have disabled cache as I don't need it. Is it running a number of old
queries to re-generate the cache anyway or it is a different cache you
are talking about? But I believe it still has to register a new
searcher. I don't know what the impact of this is though.
I guess I will go for running more tomcat running less indexes with
lower JVM heap.
Thank you for the link and thanks for the reply.
Marc
-----Original Message-----
From: Shalin Shekhar Mangar [mailto:shalinmangar@gmail.com]
Sent: 12 January 2010 07:49
To: solr-user@lucene.apache.org
Subject: Re: update solr index
On Mon, Jan 11, 2010 at 7:42 PM, Marc Des Garets
<ma...@192.com>wrote:
>
> I am running solr in tomcat and I have about 35 indexes (between 2 and
> 80 millions documents each). Currently if I try to update few
documents
> from an index (let's say the one which contains 80 millions documents)
> while tomcat is running and therefore receiving requests, I am getting
> few very long garbage collection (about 60sec). I am running tomcat
with
> -Xms10g -Xmx10g -Xmn2g -XX:PermSize=256m -XX:MaxPermSize=256m. I'm
using
> ConcMarkSweepGC.
>
> I have 2 questions:
> 1. Is solr doing something specific while an index is being updated
like
> updating something in memory which would cause the garbage collection?
>
Solr's caches are thrown away and a fixed number of old queries are
re-executed to re-generated the cache on the new index (known as
auto-warming). This happens on a commit.
>
> 2. Any idea how I could solve this problem? Currently I stop tomcat,
> update index, start tomcat. I would like to be able to update my index
> while tomcat is running. I was thinking about running more tomcat
> instance with less memory for each and each running few of my indexes.
> Do you think it would be the best way to go?
>
>
If you stop tomcat, how do you update the index? Are you running a
multi-core setup? Perhaps it is better to split up the indexes among
multiple boxes. Also, you should probably lower the JVM heap so that the
full GC pause doesn't make your index unavailable for such a long time.
Also see
http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles
/Scaling-Lucene-and-Solr
--
Regards,
Shalin Shekhar Mangar.
----------------------------------------------------------
This transmission is strictly confidential, possibly legally privileged, and intended solely for the
addressee. Any views or opinions expressed within it are those of the author and do not necessarily
represent those of 192.com, i-CD Publishing (UK) Ltd or any of it's subsidiary companies. If you
are not the intended recipient then you must not disclose, copy or take any action in reliance of this
transmission. If you have received this transmission in error, please notify the sender as soon as
possible. No employee or agent is authorised to conclude any binding agreement on behalf of
i-CD Publishing (UK) Ltd with another party by email without express written confirmation by an
authorised employee of the Company. http://www.192.com (Tel: 08000 192 192). i-CD Publishing (UK) Ltd
is incorporated in England and Wales, company number 3148549, VAT No. GB 673128728.
Re: update solr index
Posted by Shalin Shekhar Mangar <sh...@gmail.com>.
On Mon, Jan 11, 2010 at 7:42 PM, Marc Des Garets <ma...@192.com>wrote:
>
> I am running solr in tomcat and I have about 35 indexes (between 2 and
> 80 millions documents each). Currently if I try to update few documents
> from an index (let's say the one which contains 80 millions documents)
> while tomcat is running and therefore receiving requests, I am getting
> few very long garbage collection (about 60sec). I am running tomcat with
> -Xms10g -Xmx10g -Xmn2g -XX:PermSize=256m -XX:MaxPermSize=256m. I'm using
> ConcMarkSweepGC.
>
> I have 2 questions:
> 1. Is solr doing something specific while an index is being updated like
> updating something in memory which would cause the garbage collection?
>
Solr's caches are thrown away and a fixed number of old queries are
re-executed to re-generated the cache on the new index (known as
auto-warming). This happens on a commit.
>
> 2. Any idea how I could solve this problem? Currently I stop tomcat,
> update index, start tomcat. I would like to be able to update my index
> while tomcat is running. I was thinking about running more tomcat
> instance with less memory for each and each running few of my indexes.
> Do you think it would be the best way to go?
>
>
If you stop tomcat, how do you update the index? Are you running a
multi-core setup? Perhaps it is better to split up the indexes among
multiple boxes. Also, you should probably lower the JVM heap so that the
full GC pause doesn't make your index unavailable for such a long time.
Also see
http://www.lucidimagination.com/Community/Hear-from-the-Experts/Articles/Scaling-Lucene-and-Solr
--
Regards,
Shalin Shekhar Mangar.
update solr index
Posted by Marc Des Garets <ma...@192.com>.
Hi,
I am running solr in tomcat and I have about 35 indexes (between 2 and
80 millions documents each). Currently if I try to update few documents
from an index (let's say the one which contains 80 millions documents)
while tomcat is running and therefore receiving requests, I am getting
few very long garbage collection (about 60sec). I am running tomcat with
-Xms10g -Xmx10g -Xmn2g -XX:PermSize=256m -XX:MaxPermSize=256m. I'm using
ConcMarkSweepGC.
I have 2 questions:
1. Is solr doing something specific while an index is being updated like
updating something in memory which would cause the garbage collection?
2. Any idea how I could solve this problem? Currently I stop tomcat,
update index, start tomcat. I would like to be able to update my index
while tomcat is running. I was thinking about running more tomcat
instance with less memory for each and each running few of my indexes.
Do you think it would be the best way to go?
Thanks,
Marc
----------------------------------------------------------
This transmission is strictly confidential, possibly legally privileged, and intended solely for the
addressee. Any views or opinions expressed within it are those of the author and do not necessarily
represent those of 192.com, i-CD Publishing (UK) Ltd or any of it's subsidiary companies. If you
are not the intended recipient then you must not disclose, copy or take any action in reliance of this
transmission. If you have received this transmission in error, please notify the sender as soon as
possible. No employee or agent is authorised to conclude any binding agreement on behalf of
i-CD Publishing (UK) Ltd with another party by email without express written confirmation by an
authorised employee of the Company. http://www.192.com (Tel: 08000 192 192). i-CD Publishing (UK) Ltd
is incorporated in England and Wales, company number 3148549, VAT No. GB 673128728.
Re: Could not start SOLR issue
Posted by dipti khullar <di...@gmail.com>.
We were able to resolve the problem by restarting the slave. Also these
failed snapshot install incidents occur after the exception was observed,
which seems logically correct also.
"Could not start SOLR. Check solr/home property"
We just want to avoid such instances for future. Is it possible that an any
instance of time solr/home property can get corrupted?
One more thing we observed was that tomcat-users.xml was overwritten. Should
we debug towards that also?
Thanks
Dipti
On Mon, Jan 11, 2010 at 6:55 PM, Grant Ingersoll <gs...@apache.org>wrote:
>
> On Jan 11, 2010, at 1:38 AM, dipti khullar wrote:
>
> > Hi
> >
> > We are running master/slave Solr 1.3 version on production since about 5
> > months.
> >
> > Yesterday, we faced following issue on one of the slaves for the first
> time
> > because of which we had to restart the slave.
> >
> > SEVERE: Could not start SOLR. Check solr/home property
> > java.lang.RuntimeException: java.io.FileNotFoundException: no segments*
> file
> > found in org.apache.lucene.store.FSDirectory@
> /opt/solr/solr_slave/solr/data/index:
> > files: null
>
> It looks like your index was removed out from under you. Perhaps this is
> due to the failed snapshot install?
>
> Can you replicate the problem? Stopping the slave and deleting the index
> directory and then restarting it should resolve it for now.
>
> >
> > I searched on forums but couldn't find any relevant info which could have
> > possibly caused the issue.
> >
> > In snapinstaller logs, following failed logs were observed:
> >
> > 2010/01/11 04:20:06 started by solr
> > 2010/01/11 04:20:06 command:
> > /opt/solr/solr_slave/solr/solr/bin/snapinstaller
> > 2010/01/11 04:20:07 installing snapshot
> > /opt/solr/solr_slave/solr/data/snapshot.20100111041402
> > 2010/01/11 04:20:07 notifing Solr to open a new Searcher
> > 2010/01/11 04:20:07 failed to connect to Solr server
> > 2010/01/11 04:20:07 snapshot installed but Solr server has not open a new
> > Searcher
> > 2010/01/11 04:20:08 failed (elapsed time: 1 sec)
> >
> >
> > Configurations:
> > There are 2 search servers in a virtualized VMware environment. Each has
> 2
> > instances of Solr running on separates ports in tomcat.
> > Server 1: hosts 1 master(application 1), 1 slave (application 1)
> > Server 2: hosta 1 master (application 2), 1 slave (application 1)
> >
> > Both servers have 4 CPUs and 4 GB RAM.
> > Master
> > - 4GB RAM
> > - 1GB JVM Heap memory is allocated to Solr
> > Slave1/Slave2:
> > - 4GB RAM
> > - 2GB JVM Heap memory is allocated to Solr
> >
> > Can there be any possible reasons that solr/home property couldn't be
> found?
> >
> > Thanks
> > Dipti
>
>
Re: Could not start SOLR issue
Posted by Grant Ingersoll <gs...@apache.org>.
On Jan 11, 2010, at 1:38 AM, dipti khullar wrote:
> Hi
>
> We are running master/slave Solr 1.3 version on production since about 5
> months.
>
> Yesterday, we faced following issue on one of the slaves for the first time
> because of which we had to restart the slave.
>
> SEVERE: Could not start SOLR. Check solr/home property
> java.lang.RuntimeException: java.io.FileNotFoundException: no segments* file
> found in org.apache.lucene.store.FSDirectory@/opt/solr/solr_slave/solr/data/index:
> files: null
It looks like your index was removed out from under you. Perhaps this is due to the failed snapshot install?
Can you replicate the problem? Stopping the slave and deleting the index directory and then restarting it should resolve it for now.
>
> I searched on forums but couldn't find any relevant info which could have
> possibly caused the issue.
>
> In snapinstaller logs, following failed logs were observed:
>
> 2010/01/11 04:20:06 started by solr
> 2010/01/11 04:20:06 command:
> /opt/solr/solr_slave/solr/solr/bin/snapinstaller
> 2010/01/11 04:20:07 installing snapshot
> /opt/solr/solr_slave/solr/data/snapshot.20100111041402
> 2010/01/11 04:20:07 notifing Solr to open a new Searcher
> 2010/01/11 04:20:07 failed to connect to Solr server
> 2010/01/11 04:20:07 snapshot installed but Solr server has not open a new
> Searcher
> 2010/01/11 04:20:08 failed (elapsed time: 1 sec)
>
>
> Configurations:
> There are 2 search servers in a virtualized VMware environment. Each has 2
> instances of Solr running on separates ports in tomcat.
> Server 1: hosts 1 master(application 1), 1 slave (application 1)
> Server 2: hosta 1 master (application 2), 1 slave (application 1)
>
> Both servers have 4 CPUs and 4 GB RAM.
> Master
> - 4GB RAM
> - 1GB JVM Heap memory is allocated to Solr
> Slave1/Slave2:
> - 4GB RAM
> - 2GB JVM Heap memory is allocated to Solr
>
> Can there be any possible reasons that solr/home property couldn't be found?
>
> Thanks
> Dipti