You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by David Whalen <dw...@enr-corp.com> on 2007/07/30 20:18:34 UTC
Please help! Solr 1.1 HTTP server stops responding
Guys:
Can anyone help me? Things are getting serious at my
company and heads are going to roll.
I need to figure out why solr just suddenly stops responding
without any warning.
DW
> -----Original Message-----
> From: David Whalen [mailto:dwhalen@enr-corp.com]
> Sent: Friday, July 27, 2007 10:49 AM
> To: solr-user@lucene.apache.org
> Subject: RE: Solr 1.1 HTTP server stops responding
>
> We're using Jetty. I don't know what version though. To my
> knowledge, Solr is the only thing running inside it.
>
> Yes, we cannot get to the admin pages either. Nothing on port
> 8983 responds.
>
> So maybe it's actually Jetty that's messing me up? How can I
> make sure of that?
>
> Thanks for the help!
>
> DW
>
>
> > -----Original Message-----
> > From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> > Sent: Friday, July 27, 2007 10:40 AM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Solr 1.1 HTTP server stops responding
> >
> > Solr runs as a webapp (think .war file) inside a servlet container
> > (e.g. Tomcat, Jetty, Resin...). It could be that the
> servlet contan
> > itself has a bug that prevents it from responding properly after a
> > while. If you have other webapps in the same container, do
> they still
> > respond? Can you got to
> > *any* of Solr's "pages" (e.g. admin page)? Anything in
> container or
> > Solr logs?
> >
> > Otis
> > --
> > Lucene Consulting - http://lucene-consulting.com/
> >
> >
> >
> > ----- Original Message ----
> > From: David Whalen <dw...@enr-corp.com>
> > To: solr-user@lucene.apache.org
> > Sent: Friday, July 27, 2007 4:21:18 PM
> > Subject: RE: Solr 1.1 HTTP server stops responding
> >
> > Hi Otis.
> >
> > I'm filling-in for the guy that installed the software for us (now
> > he's long gone), so I'm just getting familiar with all of
> this. Can
> > you elaborate on what you mean?
> >
> > DW
> >
> >
> > > -----Original Message-----
> > > From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> > > Sent: Friday, July 27, 2007 10:01 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: Solr 1.1 HTTP server stops responding
> > >
> > > Hi David,
> > >
> > > Have you ruled out your servlet container as the source
> of this bug?
> > >
> > > Otis
> > >
> > >
> > > ----- Original Message ----
> > > From: David Whalen <dw...@enr-corp.com>
> > > To: solr-user@lucene.apache.org
> > > Sent: Friday, July 27, 2007 3:06:42 PM
> > > Subject: Solr 1.1 HTTP server stops responding
> > >
> > > Hi All.
> > >
> > > We're running Solr 1.1 and we're seeing intermittent cases
> > where Solr
> > > stops responding to HTTP requests. It seems like the
> > listener on port
> > > 8983 just doesn't respond.
> > >
> > > We stop and restart Solr and everything works fine for a
> few hours,
> > > and then the problem returns. We can't seem to point to
> any single
> > > factor that would lead to this problem, and I'm hoping to
> get some
> > > hints on how to diagnose it.
> > >
> > > Here's what I can tell you now, and I can provide more info by
> > > request:
> > >
> > > 1) The query load (via /solr/select) isn't that high.
> > Maybe 20 or 30
> > > requests per minute tops.
> > >
> > > 2) The insert load (via /solr/update) is very high. We
> > commit almost
> > > 500,000 documents per day. We also trim out the same
> > number however,
> > > so the net number of documents should stay around 20 million.
> > >
> > > 3) We do see Out of Memory errors sometimes, especially
> when making
> > > facet queries (which we do most of the time).
> > >
> > > We think solr is great, and we want to keep using it, but
> > the downtime
> > > makes the product (and us) look bad, so we need to solve
> this soon.
> > >
> > > Thanks in advance for your help!
> > >
> > > DW
> > >
> > >
> > >
> > >
> > > No virus found in this incoming message.
> > > Checked by AVG Free Edition.
> > > Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
> > > Date: 7/27/2007 6:08 AM
> > >
> > >
> >
> >
> >
> >
> > No virus found in this incoming message.
> > Checked by AVG Free Edition.
> > Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
> > Date: 7/27/2007 6:08 AM
> >
> >
>
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
> Date: 7/27/2007 6:08 AM
>
>
Re: Please help! Solr 1.1 HTTP server stops responding
Posted by Yonik Seeley <yo...@apache.org>.
On 7/30/07, David Whalen <dw...@enr-corp.com> wrote:
> Yonik:
>
> > If that's not the problem, you could decrease memory usage
> > due to faceting by upgrading to Solr 1.2 and using
> > facet.enum.cache.minDf
>
> Is it hard to upgrade from 1.1 to 1.2? We were considering
> making that change if it wouldn't cost us a lot of downtime.
It should be simple and backward compatible... but it's always best to
verify such things in a test environment first.
If you don't use index distribution you can shut down the server,
replace the solr war with the new version, and restart. You could
also start with the Solr 1.2 example and copy over your config files
and index directory.
> can you help me understand what "using facet.enum.cache.minDf"
> means? Is that a setting in the config file?
http://wiki.apache.org/solr/SimpleFacetParameters
It's a query parameter that a client can specify, or it can be
specified as a default in the handler (add it to any handler you
use... dismax or standard).
-Yonik
RE: Please help! Solr 1.1 HTTP server stops responding
Posted by David Whalen <dw...@enr-corp.com>.
Yonik:
> If that's not the problem, you could decrease memory usage
> due to faceting by upgrading to Solr 1.2 and using
> facet.enum.cache.minDf
Is it hard to upgrade from 1.1 to 1.2? We were considering
making that change if it wouldn't cost us a lot of downtime.
can you help me understand what "using facet.enum.cache.minDf"
means? Is that a setting in the config file?
Dave W
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf
> Of Yonik Seeley
> Sent: Monday, July 30, 2007 3:29 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Please help! Solr 1.1 HTTP server stops responding
>
> Grep for PERFORMANCE in the logs to make sure that you aren't
> running into a scenario where more than one searcher is
> warming in the background.
>
> If that's not the problem, you could decrease memory usage
> due to faceting by upgrading to Solr 1.2 and using
> facet.enum.cache.minDf
>
> -Yonik
>
> On 7/30/07, Kevin Holmes <kh...@enr-corp.com> wrote:
> > Just got this:
> >
> >
> >
> > Jul 30, 2007 3:02:14 PM org.apache.solr.core.SolrException log
> > SEVERE: java.lang.OutOfMemoryError: Java heap space
> >
> > Jul 30, 2007 3:02:30 PM org.apache.solr.core.SolrException log
> > SEVERE: java.lang.OutOfMemoryError: Java heap space
> >
> >
> >
> >
> >
> > Kevin Holmes
> > eNR Services, Inc.
> > 20 Glover Ave. 2nd Floor
> > Norwalk, CT. 06851
> > 203-849-7248
> > kholmes@enr-corp.com
> >
> >
> > -----Original Message-----
> > From: yseeley@gmail.com [mailto:yseeley@gmail.com] On
> Behalf Of Yonik
> > Seeley
> > Sent: Monday, July 30, 2007 2:55 PM
> > To: solr-user@lucene.apache.org
> > Subject: Re: Please help! Solr 1.1 HTTP server stops responding
> >
> > On 7/30/07, David Whalen <dw...@enr-corp.com> wrote:
> > > We increased the heap size to 1500M and that didn't seem to help.
> > > In fact, the crashes seem to occur more now than ever. We're
> > > constantly restarting solr just to get a response.
> > >
> > > I don't know enough to know where the log files are to
> answer your
> > > question
> >
> > Me neither ;-)
> > Solr's example app that uses Jetty just has logging going to stdout
> > (the console) to make it clear and visible to new users
> when an error
> > happens. Hopefully you've configured Jetty to log to files, or at
> > least redirected Jetty's stdout/stderr to a file.
> > You need to look around and try and find those log files.
> > If you find them, one thing to look for would be "WARNING"
> in the log
> > files. Another thing to look for would be "Exception" or "Memory"
> >
> > > So maybe it's actually Jetty that's messing me up? How
> can I make
> > > sure of that?
> >
> > Perhaps point your browser at http://localhost:8983/ and see if you
> > get any reponse at all.
> >
> > -Yonik
> >
>
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.476 / Virus Database: 269.10.25/926 - Release
> Date: 7/29/2007 11:14 PM
>
>
Re: Please help! Solr 1.1 HTTP server stops responding
Posted by Yonik Seeley <yo...@apache.org>.
Grep for PERFORMANCE in the logs to make sure that you aren't running
into a scenario where more than one searcher is warming in the
background.
If that's not the problem, you could decrease memory usage due to
faceting by upgrading to Solr 1.2 and using facet.enum.cache.minDf
-Yonik
On 7/30/07, Kevin Holmes <kh...@enr-corp.com> wrote:
> Just got this:
>
>
>
> Jul 30, 2007 3:02:14 PM org.apache.solr.core.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
>
> Jul 30, 2007 3:02:30 PM org.apache.solr.core.SolrException log
> SEVERE: java.lang.OutOfMemoryError: Java heap space
>
>
>
>
>
> Kevin Holmes
> eNR Services, Inc.
> 20 Glover Ave. 2nd Floor
> Norwalk, CT. 06851
> 203-849-7248
> kholmes@enr-corp.com
>
>
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
> Seeley
> Sent: Monday, July 30, 2007 2:55 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Please help! Solr 1.1 HTTP server stops responding
>
> On 7/30/07, David Whalen <dw...@enr-corp.com> wrote:
> > We increased the heap size to 1500M and that didn't seem to help.
> > In fact, the crashes seem to occur more now than ever. We're
> > constantly restarting solr just to get a response.
> >
> > I don't know enough to know where the log files are to answer
> > your question
>
> Me neither ;-)
> Solr's example app that uses Jetty just has logging going to stdout
> (the console) to make it clear and visible to new users when an error
> happens. Hopefully you've configured Jetty to log to files, or at
> least redirected Jetty's stdout/stderr to a file.
> You need to look around and try and find those log files.
> If you find them, one thing to look for would be "WARNING" in the log
> files. Another thing to look for would be "Exception" or "Memory"
>
> > So maybe it's actually Jetty that's messing me up? How can I
> > make sure of that?
>
> Perhaps point your browser at http://localhost:8983/ and see if you
> get any reponse at all.
>
> -Yonik
>
Re: Please help! Solr 1.1 HTTP server stops responding
Posted by Yonik Seeley <yo...@apache.org>.
On 7/30/07, Kevin Holmes <kh...@enr-corp.com> wrote:
> Jul 30, 2007 3:05:03 PM org.apache.solr.core.SolrException log
> SEVERE: java.io.IOException: Lock obtain timed out:
> SimpleFSLock@/tmp/lucene-f4cca35f5bee7bbcd8238c7ef8697193-write.lock
> at org.apache.lucene.store.Lock.obtain(Lock.java:69)
That's a stale lock left over from the JVM restart.
You can configure Solr to remove these locks on startup (see solrconfig.xml)
-Yonik
RE: Please help! Solr 1.1 HTTP server stops responding
Posted by Kevin Holmes <kh...@enr-corp.com>.
This might be relevant too?
Jul 30, 2007 3:05:22 PM org.apache.solr.core.SolrException log
SEVERE: Error during auto-warming of
key:org.apache.solr.search.QueryResultKey@27641eb6:java.lang.OutOfMemory
Error: Java heap space
Jul 30, 2007 3:05:25 PM org.apache.solr.core.SolrException log
SEVERE: Error during auto-warming of
key:org.apache.solr.search.QueryResultKey@d9f8dcc5:java.lang.OutOfMemory
Error: Java heap space
Jul 30, 2007 3:05:27 PM org.apache.solr.core.SolrException log
SEVERE: Error during auto-warming of
key:org.apache.solr.search.QueryResultKey@6c2447f0:java.lang.OutOfMemory
Error: Java heap space
Jul 30, 2007 3:05:30 PM org.apache.solr.core.SolrException log
SEVERE: Error during auto-warming of
key:org.apache.solr.search.QueryResultKey@67e0124:java.lang.OutOfMemoryE
rror: Java heap space
Jul 30, 2007 3:05:33 PM org.apache.solr.core.SolrException log
SEVERE: Error during auto-warming of
key:org.apache.solr.search.QueryResultKey@690dac0d:java.lang.OutOfMemory
Error: Java heap space
Jul 30, 2007 3:05:36 PM org.apache.solr.core.SolrException log
SEVERE: Error during auto-warming of
key:org.apache.solr.search.QueryResultKey@269fb7bb:java.lang.OutOfMemory
Error: Java heap space
RE: Please help! Solr 1.1 HTTP server stops responding
Posted by Kevin Holmes <kh...@enr-corp.com>.
Just got this:
Jul 30, 2007 3:02:14 PM org.apache.solr.core.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
Jul 30, 2007 3:02:30 PM org.apache.solr.core.SolrException log
SEVERE: java.lang.OutOfMemoryError: Java heap space
Kevin Holmes
eNR Services, Inc.
20 Glover Ave. 2nd Floor
Norwalk, CT. 06851
203-849-7248
kholmes@enr-corp.com
-----Original Message-----
From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf Of Yonik
Seeley
Sent: Monday, July 30, 2007 2:55 PM
To: solr-user@lucene.apache.org
Subject: Re: Please help! Solr 1.1 HTTP server stops responding
On 7/30/07, David Whalen <dw...@enr-corp.com> wrote:
> We increased the heap size to 1500M and that didn't seem to help.
> In fact, the crashes seem to occur more now than ever. We're
> constantly restarting solr just to get a response.
>
> I don't know enough to know where the log files are to answer
> your question
Me neither ;-)
Solr's example app that uses Jetty just has logging going to stdout
(the console) to make it clear and visible to new users when an error
happens. Hopefully you've configured Jetty to log to files, or at
least redirected Jetty's stdout/stderr to a file.
You need to look around and try and find those log files.
If you find them, one thing to look for would be "WARNING" in the log
files. Another thing to look for would be "Exception" or "Memory"
> So maybe it's actually Jetty that's messing me up? How can I
> make sure of that?
Perhaps point your browser at http://localhost:8983/ and see if you
get any reponse at all.
-Yonik
RE: Please help! Solr 1.1 HTTP server stops responding
Posted by Kevin Holmes <kh...@enr-corp.com>.
These might be relevant too:
Jul 30, 2007 3:05:03 PM org.apache.solr.core.SolrException log
SEVERE: java.io.IOException: Lock obtain timed out:
SimpleFSLock@/tmp/lucene-f4cca35f5bee7bbcd8238c7ef8697193-write.lock
at org.apache.lucene.store.Lock.obtain(Lock.java:69)
at
org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:258)
at
org.apache.lucene.index.IndexWriter.<init>(IndexWriter.java:208)
at
org.apache.solr.update.SolrIndexWriter.<init>(SolrIndexWriter.java:66)
at
org.apache.solr.update.UpdateHandler.createMainIndexWriter(UpdateHandler
.java:119)
Kevin Holmes
eNR Services, Inc.
20 Glover Ave. 2nd Floor
Norwalk, CT. 06851
203-849-7248
kholmes@enr-corp.com
Re: Please help! Solr 1.1 HTTP server stops responding
Posted by Yonik Seeley <yo...@apache.org>.
On 7/30/07, David Whalen <dw...@enr-corp.com> wrote:
> We increased the heap size to 1500M and that didn't seem to help.
> In fact, the crashes seem to occur more now than ever. We're
> constantly restarting solr just to get a response.
>
> I don't know enough to know where the log files are to answer
> your question
Me neither ;-)
Solr's example app that uses Jetty just has logging going to stdout
(the console) to make it clear and visible to new users when an error
happens. Hopefully you've configured Jetty to log to files, or at
least redirected Jetty's stdout/stderr to a file.
You need to look around and try and find those log files.
If you find them, one thing to look for would be "WARNING" in the log
files. Another thing to look for would be "Exception" or "Memory"
> So maybe it's actually Jetty that's messing me up? How can I
> make sure of that?
Perhaps point your browser at http://localhost:8983/ and see if you
get any reponse at all.
-Yonik
Re: Please help! Solr 1.1 HTTP server stops responding
Posted by Jeff Rodenburg <je...@gmail.com>.
Not sure if this would help you, but we encountered java heap OOM issues
with 1.1 earlier this year. We patched solr with the latest bits at the
time, which included a lucene memory fix for java heap OOM issues. (
http://issues.apache.org/jira/browse/LUCENE-754)
Different servlet container (Tomcat 5.5) and we're running JRE 5 v9.
After applying the update to the solr bits that included the patch mentioned
above, OOM has never re-appeared.
-- j
On 7/30/07, Mike Klaas <mi...@gmail.com> wrote:
>
> On 30-Jul-07, at 11:35 AM, David Whalen wrote:
>
> > Hi Yonik!
> >
> > I'm glad to finally get to talk to you. We're all very impressed
> > with solr and when it's running it's really great.
> >
> > We increased the heap size to 1500M and that didn't seem to help.
> > In fact, the crashes seem to occur more now than ever. We're
> > constantly restarting solr just to get a response.
>
> How much memory is on the system, and is anything else running? How
> large is the resulting index?
>
> If you're willing for some queries to take longer after a commit,
> reducing/eliminating the autoWarmCount for your queryCache and
> facetCache should decrease the peak memory usage (as Solr as two
> copies of the cache open at that point). Setting it to zero could up
> the halve the peak memory usage (at the cost of loss of performance
> after commits).
>
> As yonik suggested, check for PERFORMANCE warnings too--you may have
> more than two Searchers open at once.
>
> -Mike
>
>
>
Re: Please help! Solr 1.1 HTTP server stops responding
Posted by Mike Klaas <mi...@gmail.com>.
On 30-Jul-07, at 11:35 AM, David Whalen wrote:
> Hi Yonik!
>
> I'm glad to finally get to talk to you. We're all very impressed
> with solr and when it's running it's really great.
>
> We increased the heap size to 1500M and that didn't seem to help.
> In fact, the crashes seem to occur more now than ever. We're
> constantly restarting solr just to get a response.
How much memory is on the system, and is anything else running? How
large is the resulting index?
If you're willing for some queries to take longer after a commit,
reducing/eliminating the autoWarmCount for your queryCache and
facetCache should decrease the peak memory usage (as Solr as two
copies of the cache open at that point). Setting it to zero could up
the halve the peak memory usage (at the cost of loss of performance
after commits).
As yonik suggested, check for PERFORMANCE warnings too--you may have
more than two Searchers open at once.
-Mike
RE: Please help! Solr 1.1 HTTP server stops responding
Posted by Kevin Holmes <kh...@enr-corp.com>.
debiandos:~# curl -i
http://localhost:8983/solr/select/?q=superduperobscuretestingstring
HTTP/1.1 200 OK
Date: Mon, 30 Jul 2007 19:20:40 GMT
Server: Jetty/5.1.11RC0 (Linux/2.6.18-4-686 i386 java/1.5.0_11
Content-Type: text/xml; charset=UTF-8
Content-Length: 272
<?xml version="1.0" encoding="UTF-8"?>
<response>
<lst name="responseHeader"><int name="status">0</int><int
name="QTime">121</int><lst name="params"><str
name="q">superduperobscuretestingstring</str></lst></lst><result
name="response" numFound="0" start="0"/>
</response>
Next, how is Jetty being started?
cd /home/jason/code/apache-solr-1.1.0-incubating/enrsolr;
java -Xmx1500m -jar start.jar
Where is its jetty.xml
/home/jason/code/apache-solr-1.1.0-incubating/enrsolr/etc/jetty.xml
configuration file? What does that file specify for RequestLog?
<!-- Uncomment for request logging.
<Set name="RequestLog">
<New class="org.mortbay.http.NCSARequestLog">
<Arg><SystemProperty name="jetty.home"
default="."/>/logs/yyyy_mm_dd.request.log</Arg>
<Set name="retainDays">90</Set>
<Set name="append">true</Set>
<Set name="extended">false</Set>
<Set name="LogTimeZone">GMT</Set>
</New>
</Set>
-->
My point is that I can't predict how it's started on your machine.
You need to find out yourself.
On Linux:
- "ps -ef | grep java"
root 10175 10174 11 15:17 pts/1 00:00:56 java -Xmx1500m -jar
start.jar
Re: Please help! Solr 1.1 HTTP server stops responding
Posted by Matt Kangas <ka...@gmail.com>.
David,
If "nothing on port 8983 responds", your servlet container is
certainly the first thing that should be checked, because that is
what's listening on port 8983.
First, let's need to figure out what version of Jetty you're using
and how it is started -- which will lead you to the log files, if it
is producing any. When Jetty/Solr is running correctly, try fetching
any page from that host using "curl -I".
Example: here's what I see on my laptop, with Solr running inside Jetty
> shaft:R curl -I http://localhost:8983/
> HTTP/1.1 404 Not Found
> Content-Type: text/html; charset=iso-8859-1
> Content-Length: 1287
> Server: Jetty(6.1.3)
From this, I know it's Jetty 6.1.3.
Next, how is Jetty being started? Where is its jetty.xml
configuration file? What does that file specify for RequestLog?
On my laptop, I'm manually starting it via "java -jar start.jar".
On my work hosts, "java -jar start.jar" is being run from daemontools
(unlikely in your case)
Or, Jetty can be invoked without the "start.jar" shortcut. That's
just the default way of starting Solr.
My point is that I can't predict how it's started on your machine.
You need to find out yourself.
On Linux:
- "ps -ef | grep java"
- look at that list, see which java process is the relevant one
- take the parent PID of that process and run "ps -p <value>" to see
what process started it
- repeat until you find the script or program that started Jetty, and
the path to jetty.xml
If the process actually was "java -jar start.jar", then look for an
"etc" subdir in the current working directory for that process.
HTH,
--Matt Kangas
(stepping in to help with what seems to be a panicked-newbie
question...)
On Jul 30, 2007, at 2:35 PM, David Whalen wrote:
> Hi Yonik!
>
> I'm glad to finally get to talk to you. We're all very impressed
> with solr and when it's running it's really great.
>
> We increased the heap size to 1500M and that didn't seem to help.
> In fact, the crashes seem to occur more now than ever. We're
> constantly restarting solr just to get a response.
>
> I don't know enough to know where the log files are to answer
> your question (again, I'm filling in for the guy that set us
> up with all this). Can I ask for your patience so we can figure
> this out?
>
> Thanks!
>
> Dave W
>
>
>> -----Original Message-----
>> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf
>> Of Yonik Seeley
>> Sent: Monday, July 30, 2007 2:23 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Please help! Solr 1.1 HTTP server stops responding
>>
>> It may be related to the out-of-memory errors you were seeing.
>> severe errors like that should never be ignored.
>> Do you see any other warning or severe errors in your logs?
>>
>> -Yonik
>>
>> On 7/30/07, David Whalen <dw...@enr-corp.com> wrote:
>>> Guys:
>>>
>>> Can anyone help me? Things are getting serious at my company and
>>> heads are going to roll.
>>>
>>> I need to figure out why solr just suddenly stops
>> responding without
>>> any warning.
>>>
>>> DW
>>>
>>>
>>>> -----Original Message-----
>>>> From: David Whalen [mailto:dwhalen@enr-corp.com]
>>>> Sent: Friday, July 27, 2007 10:49 AM
>>>> To: solr-user@lucene.apache.org
>>>> Subject: RE: Solr 1.1 HTTP server stops responding
>>>>
>>>> We're using Jetty. I don't know what version though. To my
>>>> knowledge, Solr is the only thing running inside it.
>>>>
>>>> Yes, we cannot get to the admin pages either. Nothing on port
>>>> 8983 responds.
>>>>
>>>> So maybe it's actually Jetty that's messing me up? How
>> can I make
>>>> sure of that?
>>>>
>>>> Thanks for the help!
>>>>
>>>> DW
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
>>>>> Sent: Friday, July 27, 2007 10:40 AM
>>>>> To: solr-user@lucene.apache.org
>>>>> Subject: Re: Solr 1.1 HTTP server stops responding
>>>>>
>>>>> Solr runs as a webapp (think .war file) inside a
>> servlet container
>>>>> (e.g. Tomcat, Jetty, Resin...). It could be that the
>>>> servlet contan
>>>>> itself has a bug that prevents it from responding
>> properly after a
>>>>> while. If you have other webapps in the same container, do
>>>> they still
>>>>> respond? Can you got to
>>>>> *any* of Solr's "pages" (e.g. admin page)? Anything in
>>>> container or
>>>>> Solr logs?
>>>>>
>>>>> Otis
>>>>> --
>>>>> Lucene Consulting - http://lucene-consulting.com/
>>>>>
>>>>>
>>>>>
>>>>> ----- Original Message ----
>>>>> From: David Whalen <dw...@enr-corp.com>
>>>>> To: solr-user@lucene.apache.org
>>>>> Sent: Friday, July 27, 2007 4:21:18 PM
>>>>> Subject: RE: Solr 1.1 HTTP server stops responding
>>>>>
>>>>> Hi Otis.
>>>>>
>>>>> I'm filling-in for the guy that installed the software
>> for us (now
>>>>> he's long gone), so I'm just getting familiar with all of
>>>> this. Can
>>>>> you elaborate on what you mean?
>>>>>
>>>>> DW
>>>>>
>>>>>
>>>>>> -----Original Message-----
>>>>>> From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
>>>>>> Sent: Friday, July 27, 2007 10:01 AM
>>>>>> To: solr-user@lucene.apache.org
>>>>>> Subject: Re: Solr 1.1 HTTP server stops responding
>>>>>>
>>>>>> Hi David,
>>>>>>
>>>>>> Have you ruled out your servlet container as the source
>>>> of this bug?
>>>>>>
>>>>>> Otis
>>>>>>
>>>>>>
>>>>>> ----- Original Message ----
>>>>>> From: David Whalen <dw...@enr-corp.com>
>>>>>> To: solr-user@lucene.apache.org
>>>>>> Sent: Friday, July 27, 2007 3:06:42 PM
>>>>>> Subject: Solr 1.1 HTTP server stops responding
>>>>>>
>>>>>> Hi All.
>>>>>>
>>>>>> We're running Solr 1.1 and we're seeing intermittent cases
>>>>> where Solr
>>>>>> stops responding to HTTP requests. It seems like the
>>>>> listener on port
>>>>>> 8983 just doesn't respond.
>>>>>>
>>>>>> We stop and restart Solr and everything works fine for a
>>>> few hours,
>>>>>> and then the problem returns. We can't seem to point to
>>>> any single
>>>>>> factor that would lead to this problem, and I'm hoping to
>>>> get some
>>>>>> hints on how to diagnose it.
>>>>>>
>>>>>> Here's what I can tell you now, and I can provide more info by
>>>>>> request:
>>>>>>
>>>>>> 1) The query load (via /solr/select) isn't that high.
>>>>> Maybe 20 or 30
>>>>>> requests per minute tops.
>>>>>>
>>>>>> 2) The insert load (via /solr/update) is very high. We
>>>>> commit almost
>>>>>> 500,000 documents per day. We also trim out the same
>>>>> number however,
>>>>>> so the net number of documents should stay around 20 million.
>>>>>>
>>>>>> 3) We do see Out of Memory errors sometimes, especially
>>>> when making
>>>>>> facet queries (which we do most of the time).
>>>>>>
>>>>>> We think solr is great, and we want to keep using it, but
>>>>> the downtime
>>>>>> makes the product (and us) look bad, so we need to solve
>>>> this soon.
>>>>>>
>>>>>> Thanks in advance for your help!
>>>>>>
>>>>>> DW
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> No virus found in this incoming message.
>>>>>> Checked by AVG Free Edition.
>>>>>> Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
>>>>>> Date: 7/27/2007 6:08 AM
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> No virus found in this incoming message.
>>>>> Checked by AVG Free Edition.
>>>>> Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
>>>>> Date: 7/27/2007 6:08 AM
>>>>>
>>>>>
>>>>
>>>> No virus found in this incoming message.
>>>> Checked by AVG Free Edition.
>>>> Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
>>>> Date: 7/27/2007 6:08 AM
>>>>
>>>>
>>>
>>
>> No virus found in this incoming message.
>> Checked by AVG Free Edition.
>> Version: 7.5.476 / Virus Database: 269.10.25/926 - Release
>> Date: 7/29/2007 11:14 PM
>>
>>
--
Matt Kangas / kangas@gmail.com
RE: Please help! Solr 1.1 HTTP server stops responding
Posted by David Whalen <dw...@enr-corp.com>.
Hi Yonik!
I'm glad to finally get to talk to you. We're all very impressed
with solr and when it's running it's really great.
We increased the heap size to 1500M and that didn't seem to help.
In fact, the crashes seem to occur more now than ever. We're
constantly restarting solr just to get a response.
I don't know enough to know where the log files are to answer
your question (again, I'm filling in for the guy that set us
up with all this). Can I ask for your patience so we can figure
this out?
Thanks!
Dave W
> -----Original Message-----
> From: yseeley@gmail.com [mailto:yseeley@gmail.com] On Behalf
> Of Yonik Seeley
> Sent: Monday, July 30, 2007 2:23 PM
> To: solr-user@lucene.apache.org
> Subject: Re: Please help! Solr 1.1 HTTP server stops responding
>
> It may be related to the out-of-memory errors you were seeing.
> severe errors like that should never be ignored.
> Do you see any other warning or severe errors in your logs?
>
> -Yonik
>
> On 7/30/07, David Whalen <dw...@enr-corp.com> wrote:
> > Guys:
> >
> > Can anyone help me? Things are getting serious at my company and
> > heads are going to roll.
> >
> > I need to figure out why solr just suddenly stops
> responding without
> > any warning.
> >
> > DW
> >
> >
> > > -----Original Message-----
> > > From: David Whalen [mailto:dwhalen@enr-corp.com]
> > > Sent: Friday, July 27, 2007 10:49 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: RE: Solr 1.1 HTTP server stops responding
> > >
> > > We're using Jetty. I don't know what version though. To my
> > > knowledge, Solr is the only thing running inside it.
> > >
> > > Yes, we cannot get to the admin pages either. Nothing on port
> > > 8983 responds.
> > >
> > > So maybe it's actually Jetty that's messing me up? How
> can I make
> > > sure of that?
> > >
> > > Thanks for the help!
> > >
> > > DW
> > >
> > >
> > > > -----Original Message-----
> > > > From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> > > > Sent: Friday, July 27, 2007 10:40 AM
> > > > To: solr-user@lucene.apache.org
> > > > Subject: Re: Solr 1.1 HTTP server stops responding
> > > >
> > > > Solr runs as a webapp (think .war file) inside a
> servlet container
> > > > (e.g. Tomcat, Jetty, Resin...). It could be that the
> > > servlet contan
> > > > itself has a bug that prevents it from responding
> properly after a
> > > > while. If you have other webapps in the same container, do
> > > they still
> > > > respond? Can you got to
> > > > *any* of Solr's "pages" (e.g. admin page)? Anything in
> > > container or
> > > > Solr logs?
> > > >
> > > > Otis
> > > > --
> > > > Lucene Consulting - http://lucene-consulting.com/
> > > >
> > > >
> > > >
> > > > ----- Original Message ----
> > > > From: David Whalen <dw...@enr-corp.com>
> > > > To: solr-user@lucene.apache.org
> > > > Sent: Friday, July 27, 2007 4:21:18 PM
> > > > Subject: RE: Solr 1.1 HTTP server stops responding
> > > >
> > > > Hi Otis.
> > > >
> > > > I'm filling-in for the guy that installed the software
> for us (now
> > > > he's long gone), so I'm just getting familiar with all of
> > > this. Can
> > > > you elaborate on what you mean?
> > > >
> > > > DW
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> > > > > Sent: Friday, July 27, 2007 10:01 AM
> > > > > To: solr-user@lucene.apache.org
> > > > > Subject: Re: Solr 1.1 HTTP server stops responding
> > > > >
> > > > > Hi David,
> > > > >
> > > > > Have you ruled out your servlet container as the source
> > > of this bug?
> > > > >
> > > > > Otis
> > > > >
> > > > >
> > > > > ----- Original Message ----
> > > > > From: David Whalen <dw...@enr-corp.com>
> > > > > To: solr-user@lucene.apache.org
> > > > > Sent: Friday, July 27, 2007 3:06:42 PM
> > > > > Subject: Solr 1.1 HTTP server stops responding
> > > > >
> > > > > Hi All.
> > > > >
> > > > > We're running Solr 1.1 and we're seeing intermittent cases
> > > > where Solr
> > > > > stops responding to HTTP requests. It seems like the
> > > > listener on port
> > > > > 8983 just doesn't respond.
> > > > >
> > > > > We stop and restart Solr and everything works fine for a
> > > few hours,
> > > > > and then the problem returns. We can't seem to point to
> > > any single
> > > > > factor that would lead to this problem, and I'm hoping to
> > > get some
> > > > > hints on how to diagnose it.
> > > > >
> > > > > Here's what I can tell you now, and I can provide more info by
> > > > > request:
> > > > >
> > > > > 1) The query load (via /solr/select) isn't that high.
> > > > Maybe 20 or 30
> > > > > requests per minute tops.
> > > > >
> > > > > 2) The insert load (via /solr/update) is very high. We
> > > > commit almost
> > > > > 500,000 documents per day. We also trim out the same
> > > > number however,
> > > > > so the net number of documents should stay around 20 million.
> > > > >
> > > > > 3) We do see Out of Memory errors sometimes, especially
> > > when making
> > > > > facet queries (which we do most of the time).
> > > > >
> > > > > We think solr is great, and we want to keep using it, but
> > > > the downtime
> > > > > makes the product (and us) look bad, so we need to solve
> > > this soon.
> > > > >
> > > > > Thanks in advance for your help!
> > > > >
> > > > > DW
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > No virus found in this incoming message.
> > > > > Checked by AVG Free Edition.
> > > > > Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
> > > > > Date: 7/27/2007 6:08 AM
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > >
> > > > No virus found in this incoming message.
> > > > Checked by AVG Free Edition.
> > > > Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
> > > > Date: 7/27/2007 6:08 AM
> > > >
> > > >
> > >
> > > No virus found in this incoming message.
> > > Checked by AVG Free Edition.
> > > Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
> > > Date: 7/27/2007 6:08 AM
> > >
> > >
> >
>
> No virus found in this incoming message.
> Checked by AVG Free Edition.
> Version: 7.5.476 / Virus Database: 269.10.25/926 - Release
> Date: 7/29/2007 11:14 PM
>
>
Re: Please help! Solr 1.1 HTTP server stops responding
Posted by Yonik Seeley <yo...@apache.org>.
It may be related to the out-of-memory errors you were seeing.
severe errors like that should never be ignored.
Do you see any other warning or severe errors in your logs?
-Yonik
On 7/30/07, David Whalen <dw...@enr-corp.com> wrote:
> Guys:
>
> Can anyone help me? Things are getting serious at my
> company and heads are going to roll.
>
> I need to figure out why solr just suddenly stops responding
> without any warning.
>
> DW
>
>
> > -----Original Message-----
> > From: David Whalen [mailto:dwhalen@enr-corp.com]
> > Sent: Friday, July 27, 2007 10:49 AM
> > To: solr-user@lucene.apache.org
> > Subject: RE: Solr 1.1 HTTP server stops responding
> >
> > We're using Jetty. I don't know what version though. To my
> > knowledge, Solr is the only thing running inside it.
> >
> > Yes, we cannot get to the admin pages either. Nothing on port
> > 8983 responds.
> >
> > So maybe it's actually Jetty that's messing me up? How can I
> > make sure of that?
> >
> > Thanks for the help!
> >
> > DW
> >
> >
> > > -----Original Message-----
> > > From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> > > Sent: Friday, July 27, 2007 10:40 AM
> > > To: solr-user@lucene.apache.org
> > > Subject: Re: Solr 1.1 HTTP server stops responding
> > >
> > > Solr runs as a webapp (think .war file) inside a servlet container
> > > (e.g. Tomcat, Jetty, Resin...). It could be that the
> > servlet contan
> > > itself has a bug that prevents it from responding properly after a
> > > while. If you have other webapps in the same container, do
> > they still
> > > respond? Can you got to
> > > *any* of Solr's "pages" (e.g. admin page)? Anything in
> > container or
> > > Solr logs?
> > >
> > > Otis
> > > --
> > > Lucene Consulting - http://lucene-consulting.com/
> > >
> > >
> > >
> > > ----- Original Message ----
> > > From: David Whalen <dw...@enr-corp.com>
> > > To: solr-user@lucene.apache.org
> > > Sent: Friday, July 27, 2007 4:21:18 PM
> > > Subject: RE: Solr 1.1 HTTP server stops responding
> > >
> > > Hi Otis.
> > >
> > > I'm filling-in for the guy that installed the software for us (now
> > > he's long gone), so I'm just getting familiar with all of
> > this. Can
> > > you elaborate on what you mean?
> > >
> > > DW
> > >
> > >
> > > > -----Original Message-----
> > > > From: Otis Gospodnetic [mailto:otis_gospodnetic@yahoo.com]
> > > > Sent: Friday, July 27, 2007 10:01 AM
> > > > To: solr-user@lucene.apache.org
> > > > Subject: Re: Solr 1.1 HTTP server stops responding
> > > >
> > > > Hi David,
> > > >
> > > > Have you ruled out your servlet container as the source
> > of this bug?
> > > >
> > > > Otis
> > > >
> > > >
> > > > ----- Original Message ----
> > > > From: David Whalen <dw...@enr-corp.com>
> > > > To: solr-user@lucene.apache.org
> > > > Sent: Friday, July 27, 2007 3:06:42 PM
> > > > Subject: Solr 1.1 HTTP server stops responding
> > > >
> > > > Hi All.
> > > >
> > > > We're running Solr 1.1 and we're seeing intermittent cases
> > > where Solr
> > > > stops responding to HTTP requests. It seems like the
> > > listener on port
> > > > 8983 just doesn't respond.
> > > >
> > > > We stop and restart Solr and everything works fine for a
> > few hours,
> > > > and then the problem returns. We can't seem to point to
> > any single
> > > > factor that would lead to this problem, and I'm hoping to
> > get some
> > > > hints on how to diagnose it.
> > > >
> > > > Here's what I can tell you now, and I can provide more info by
> > > > request:
> > > >
> > > > 1) The query load (via /solr/select) isn't that high.
> > > Maybe 20 or 30
> > > > requests per minute tops.
> > > >
> > > > 2) The insert load (via /solr/update) is very high. We
> > > commit almost
> > > > 500,000 documents per day. We also trim out the same
> > > number however,
> > > > so the net number of documents should stay around 20 million.
> > > >
> > > > 3) We do see Out of Memory errors sometimes, especially
> > when making
> > > > facet queries (which we do most of the time).
> > > >
> > > > We think solr is great, and we want to keep using it, but
> > > the downtime
> > > > makes the product (and us) look bad, so we need to solve
> > this soon.
> > > >
> > > > Thanks in advance for your help!
> > > >
> > > > DW
> > > >
> > > >
> > > >
> > > >
> > > > No virus found in this incoming message.
> > > > Checked by AVG Free Edition.
> > > > Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
> > > > Date: 7/27/2007 6:08 AM
> > > >
> > > >
> > >
> > >
> > >
> > >
> > > No virus found in this incoming message.
> > > Checked by AVG Free Edition.
> > > Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
> > > Date: 7/27/2007 6:08 AM
> > >
> > >
> >
> > No virus found in this incoming message.
> > Checked by AVG Free Edition.
> > Version: 7.5.476 / Virus Database: 269.10.22/922 - Release
> > Date: 7/27/2007 6:08 AM
> >
> >
>