You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by alexander sulz <a....@digiconcept.net> on 2011/08/02 12:01:35 UTC
Solr 3.3 crashes after ~18 hours?
Hello folks,
I'm using the latest stable Solr release -> 3.3 and I encounter strange
phenomena with it.
After about 19 hours it just crashes, but I can't find anything in the
logs, no exceptions, no warnings,
no suspicious info entries..
I have an index-job running from 6am to 8pm every 10 minutes. After each
job there is a commit.
An optimize-job is done twice a day at 12:15pm and 9:15pm.
Does anyone have an idea what could possibly be wrong or where to look
for further debug info?
regards and thank you
alex
Re: Solr 3.3 crashes after ~18 hours?
Posted by Markus Jelsma <ma...@openindex.io>.
Strange, anything out of the ordinary in the syslog?
On Tuesday 02 August 2011 12:01:35 alexander sulz wrote:
> Hello folks,
>
> I'm using the latest stable Solr release -> 3.3 and I encounter strange
> phenomena with it.
> After about 19 hours it just crashes, but I can't find anything in the
> logs, no exceptions, no warnings,
> no suspicious info entries..
>
> I have an index-job running from 6am to 8pm every 10 minutes. After each
> job there is a commit.
> An optimize-job is done twice a day at 12:15pm and 9:15pm.
>
> Does anyone have an idea what could possibly be wrong or where to look
> for further debug info?
>
> regards and thank you
> alex
--
Markus Jelsma - CTO - Openindex
http://www.linkedin.com/in/markus17
050-8536620 / 06-50258350
Re: Solr 3.3 crashes after ~18 hours?
Posted by Manish Bafna <ma...@gmail.com>.
Check out Physcial memory/virtual memory usage.
RAM usage might be less but Physical memory usage goes up as you index more
documents.
It might be because of MMapDirectory which used MappedByteBuffer.
On Thu, Aug 4, 2011 at 7:38 PM, Yonik Seeley <yo...@lucidimagination.com>wrote:
> On Thu, Aug 4, 2011 at 8:09 AM, alexander sulz <a....@digiconcept.net>
> wrote:
> > Thank you for the many replies!
> >
> > Like I said, I couldn't find anything in logs created by solr.
> > I just had a look at the /var/logs/messages and there wasn't anything
> > either.
> >
> > What I mean by crash is that the process is still there and http GET
> pings
> > would return 200
> > but when i try visiting /solr/admin, I'd get a blank page! The server
> > ignores any incoming updates or commits,
>
> "ignores" means what? The request hangs? If so, could you get a thread
> dump?
>
> Do queries work (like /solr/select?q=*:*) ?
>
> > thous throwing no errors, no 503's.. It's like the server has a blackout
> and
> > stares blankly into space.
>
> Are you using a different servlet container than what is shipped with solr?
> If you did start with the solr "example" server, what jetty
> configuration changes have you made?
>
> -Yonik
> http://www.lucidimagination.com
>
Re: Solr 3.3 crashes after ~18 hours?
Posted by Stephen Duncan Jr <st...@gmail.com>.
I know it seems like my problem may not be the same as the original
poster, but in investigating this, I did find this Jetty issue that
may be related: http://jira.codehaus.org/browse/JETTY-1377
Stephen Duncan Jr
www.stephenduncanjr.com
On Thu, Aug 4, 2011 at 1:54 PM, Stephen Duncan Jr
<st...@gmail.com> wrote:
> On Thu, Aug 4, 2011 at 10:08 AM, Yonik Seeley
> <yo...@lucidimagination.com> wrote:
>>
>> "ignores" means what? The request hangs? If so, could you get a thread dump?
>>
>> Do queries work (like /solr/select?q=*:*) ?
>>
>>> thous throwing no errors, no 503's.. It's like the server has a blackout and
>>> stares blankly into space.
>>
>> Are you using a different servlet container than what is shipped with solr?
>> If you did start with the solr "example" server, what jetty
>> configuration changes have you made?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>
> We're seeing something similar here. Not sure exactly what the
> circumstances are, but occasionally our Solr 3.3 test instance is
> hanging, nothing seems to be happening for several minutes. It does
> seem to be happening while data is being added and continuous queries
> are being sent. It also may be related to an optimize happening (we
> attempt to optimize after adding all the new data from our database).
> The last log message is:
>
> 2011-08-04 13:46:56,418 [qtp30604342-451] INFO
> org.apache.solr.core.SolrCore - [report] webapp= path=/update
> params={optimize=true&waitSearcher=true&maxSegments=1&waitFlush=true&wt=javabin&version=2}
> status=0 QTime=109109
>
> Here is our thread dump:
>
>
> 2011-08-04 13:47:16
> Full thread dump Java HotSpot(TM) Client VM (20.1-b02 mixed mode):
>
> "RMI TCP Connection(13)-172.16.10.102" daemon prio=6 tid=0x47a4a400
> nid=0x1384 runnable [0x4861f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.SocketInputStream.socketRead0(Native Method)
> at java.net.SocketInputStream.read(SocketInputStream.java:129)
> at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
> at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
> - locked <0x183a55a0> (a java.io.BufferedInputStream)
> at java.io.FilterInputStream.read(FilterInputStream.java:66)
> at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
> at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - <0x183a7c68> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
>
> "qtp30604342-451" prio=6 tid=0x475c4800 nid=0x1a58 waiting on
> condition [0x4897f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x18214c08> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "qtp30604342-450" prio=6 tid=0x47ad1c00 nid=0x1ca4 waiting on
> condition [0x49d2f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x18214c08> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "qtp30604342-449" prio=6 tid=0x47a57c00 nid=0xb2c waiting on condition
> [0x49c2f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x18214c08> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "JMX server connection timeout 438" daemon prio=6 tid=0x47604c00
> nid=0x1c88 in Object.wait() [0x48c7f000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
> - locked <0x18331dc8> (a [I)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "RMI Scheduler(0)" daemon prio=6 tid=0x48fd0400 nid=0x608 waiting on
> condition [0x48b7f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x18331e58> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "RMI TCP Accept-0" daemon prio=6 tid=0x48ffd400 nid=0x1e4c runnable [0x4841f000]
> java.lang.Thread.State: RUNNABLE
> at java.net.PlainSocketImpl.socketAccept(Native Method)
> at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
> - locked <0x18331fc8> (a java.net.SocksSocketImpl)
> at java.net.ServerSocket.implAccept(ServerSocket.java:462)
> at java.net.ServerSocket.accept(ServerSocket.java:430)
> at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
> at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "qtp30604342-424" prio=6 tid=0x48f96400 nid=0x1214 waiting on
> condition [0x49a2f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x18214c08> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "qtp30604342-423" prio=6 tid=0x46d46400 nid=0x1df4 waiting on
> condition [0x4992f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x18214c08> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "qtp30604342-92" prio=6 tid=0x478d7400 nid=0x15e4 waiting on condition
> [0x49b2f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x18214c08> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "pool-4-thread-1" prio=6 tid=0x46d00800 nid=0x1bf0 waiting on
> condition [0x4887f000]
> java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x181a6628> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
> at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
> at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "qtp30604342-14 Selector0 SelectChannelConnector@127.0.0.1:8080
> STARTED" prio=6 tid=0x47a0c800 nid=0x1e60 runnable [0x4821f000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.WindowsSelectorImpl.discardUrgentData(Native Method)
> at sun.nio.ch.WindowsSelectorImpl.access$2200(WindowsSelectorImpl.java:32)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.processFDSet(WindowsSelectorImpl.java:336)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.processSelectedKeys(WindowsSelectorImpl.java:299)
> at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$2900(WindowsSelectorImpl.java:255)
> at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:471)
> at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:149)
> at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
> - locked <0x18217d98> (a sun.nio.ch.Util$2)
> - locked <0x18217da8> (a java.util.Collections$UnmodifiableSet)
> - locked <0x18217d20> (a sun.nio.ch.WindowsSelectorImpl)
> at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:88)
> at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:475)
> at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228)
> at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:257)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "qtp30604342-12 Acceptor0 SelectChannelConnector@127.0.0.1:8080
> STARTED" prio=6 tid=0x479f8800 nid=0x1f58 runnable [0x4801f000]
> java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
> at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
> - locked <0x18223600> (a java.lang.Object)
> at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
> at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830)
> at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "pool-3-thread-1" prio=6 tid=0x479ec400 nid=0x18f4 waiting on
> condition [0x47f1f000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x182132a0> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "pool-1-thread-1" prio=6 tid=0x479a4400 nid=0x970 waiting on condition
> [0x47e1f000]
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x17fdd938> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
> at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
> at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
> at java.lang.Thread.run(Thread.java:662)
>
> Locked ownable synchronizers:
> - None
>
> "HashSessionScavenger-0" daemon prio=6 tid=0x478e4400 nid=0x154c in
> Object.wait() [0x47d0f000]
> java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.util.TimerThread.mainLoop(Timer.java:509)
> - locked <0x17e57de8> (a java.util.TaskQueue)
> at java.util.TimerThread.run(Timer.java:462)
>
> Locked ownable synchronizers:
> - None
>
> "Low Memory Detector" daemon prio=6 tid=0x46cf0000 nid=0xcfc runnable
> [0x00000000]
> java.lang.Thread.State: RUNNABLE
>
> Locked ownable synchronizers:
> - None
>
> "C1 CompilerThread0" daemon prio=10 tid=0x46cea000 nid=0x1494 waiting
> on condition [0x00000000]
> java.lang.Thread.State: RUNNABLE
>
> Locked ownable synchronizers:
> - None
>
> "Attach Listener" daemon prio=10 tid=0x46ce8400 nid=0x1024 waiting on
> condition [0x00000000]
> java.lang.Thread.State: RUNNABLE
>
> Locked ownable synchronizers:
> - None
>
> "Signal Dispatcher" daemon prio=10 tid=0x46ce7000 nid=0x13f8 runnable
> [0x00000000]
> java.lang.Thread.State: RUNNABLE
>
> Locked ownable synchronizers:
> - None
>
> "Finalizer" daemon prio=8 tid=0x46cd6000 nid=0xb44 in Object.wait() [0x4705f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
> - locked <0x17df02c0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
>
> Locked ownable synchronizers:
> - None
>
> "Reference Handler" daemon prio=10 tid=0x46cd4c00 nid=0xc90 in
> Object.wait() [0x46f5f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:485)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
> - locked <0x17df0350> (a java.lang.ref.Reference$Lock)
>
> Locked ownable synchronizers:
> - None
>
> "main" prio=6 tid=0x003e6400 nid=0x880 in Object.wait() [0x0061f000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x17e5ea80> (a java.lang.Object)
> at java.lang.Object.wait(Object.java:485)
> at org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:382)
> - locked <0x17e5ea80> (a java.lang.Object)
> at org.eclipse.jetty.server.Server.join(Server.java:399)
> at <ourapp>.App.main(App.java:42)
>
> Locked ownable synchronizers:
> - None
>
> "VM Thread" prio=10 tid=0x46cd1000 nid=0x1a74 runnable
>
> "VM Periodic Task Thread" prio=10 tid=0x46cfac00 nid=0x1430 waiting on
> condition
>
> JNI global references: 1665
>
> ---
> Stephen Duncan Jr
> www.stephenduncanjr.com
>
Re: Solr 3.3 crashes after ~18 hours?
Posted by Stephen Duncan Jr <st...@gmail.com>.
On Thu, Aug 4, 2011 at 10:08 AM, Yonik Seeley
<yo...@lucidimagination.com> wrote:
>
> "ignores" means what? The request hangs? If so, could you get a thread dump?
>
> Do queries work (like /solr/select?q=*:*) ?
>
>> thous throwing no errors, no 503's.. It's like the server has a blackout and
>> stares blankly into space.
>
> Are you using a different servlet container than what is shipped with solr?
> If you did start with the solr "example" server, what jetty
> configuration changes have you made?
>
> -Yonik
> http://www.lucidimagination.com
>
We're seeing something similar here. Not sure exactly what the
circumstances are, but occasionally our Solr 3.3 test instance is
hanging, nothing seems to be happening for several minutes. It does
seem to be happening while data is being added and continuous queries
are being sent. It also may be related to an optimize happening (we
attempt to optimize after adding all the new data from our database).
The last log message is:
2011-08-04 13:46:56,418 [qtp30604342-451] INFO
org.apache.solr.core.SolrCore - [report] webapp= path=/update
params={optimize=true&waitSearcher=true&maxSegments=1&waitFlush=true&wt=javabin&version=2}
status=0 QTime=109109
Here is our thread dump:
2011-08-04 13:47:16
Full thread dump Java HotSpot(TM) Client VM (20.1-b02 mixed mode):
"RMI TCP Connection(13)-172.16.10.102" daemon prio=6 tid=0x47a4a400
nid=0x1384 runnable [0x4861f000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:129)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
- locked <0x183a55a0> (a java.io.BufferedInputStream)
at java.io.FilterInputStream.read(FilterInputStream.java:66)
at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:517)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:790)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:649)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- <0x183a7c68> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
"qtp30604342-451" prio=6 tid=0x475c4800 nid=0x1a58 waiting on
condition [0x4897f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x18214c08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"qtp30604342-450" prio=6 tid=0x47ad1c00 nid=0x1ca4 waiting on
condition [0x49d2f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x18214c08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"qtp30604342-449" prio=6 tid=0x47a57c00 nid=0xb2c waiting on condition
[0x49c2f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x18214c08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"JMX server connection timeout 438" daemon prio=6 tid=0x47604c00
nid=0x1c88 in Object.wait() [0x48c7f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at com.sun.jmx.remote.internal.ServerCommunicatorAdmin$Timeout.run(ServerCommunicatorAdmin.java:150)
- locked <0x18331dc8> (a [I)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"RMI Scheduler(0)" daemon prio=6 tid=0x48fd0400 nid=0x608 waiting on
condition [0x48b7f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x18331e58> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"RMI TCP Accept-0" daemon prio=6 tid=0x48ffd400 nid=0x1e4c runnable [0x4841f000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:408)
- locked <0x18331fc8> (a java.net.SocksSocketImpl)
at java.net.ServerSocket.implAccept(ServerSocket.java:462)
at java.net.ServerSocket.accept(ServerSocket.java:430)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:34)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:369)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:341)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"qtp30604342-424" prio=6 tid=0x48f96400 nid=0x1214 waiting on
condition [0x49a2f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x18214c08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"qtp30604342-423" prio=6 tid=0x46d46400 nid=0x1df4 waiting on
condition [0x4992f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x18214c08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"qtp30604342-92" prio=6 tid=0x478d7400 nid=0x15e4 waiting on condition
[0x49b2f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x18214c08> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
at org.eclipse.jetty.util.thread.QueuedThreadPool.idleJobPoll(QueuedThreadPool.java:512)
at org.eclipse.jetty.util.thread.QueuedThreadPool.access$600(QueuedThreadPool.java:38)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:558)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"pool-4-thread-1" prio=6 tid=0x46d00800 nid=0x1bf0 waiting on
condition [0x4887f000]
java.lang.Thread.State: TIMED_WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x181a6628> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:198)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2025)
at java.util.concurrent.DelayQueue.take(DelayQueue.java:164)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:609)
at java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:602)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"qtp30604342-14 Selector0 SelectChannelConnector@127.0.0.1:8080
STARTED" prio=6 tid=0x47a0c800 nid=0x1e60 runnable [0x4821f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.WindowsSelectorImpl.discardUrgentData(Native Method)
at sun.nio.ch.WindowsSelectorImpl.access$2200(WindowsSelectorImpl.java:32)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.processFDSet(WindowsSelectorImpl.java:336)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.processSelectedKeys(WindowsSelectorImpl.java:299)
at sun.nio.ch.WindowsSelectorImpl$SubSelector.access$2900(WindowsSelectorImpl.java:255)
at sun.nio.ch.WindowsSelectorImpl.updateSelectedKeys(WindowsSelectorImpl.java:471)
at sun.nio.ch.WindowsSelectorImpl.doSelect(WindowsSelectorImpl.java:149)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)
- locked <0x18217d98> (a sun.nio.ch.Util$2)
- locked <0x18217da8> (a java.util.Collections$UnmodifiableSet)
- locked <0x18217d20> (a sun.nio.ch.WindowsSelectorImpl)
at sun.nio.ch.SelectorImpl.selectNow(SelectorImpl.java:88)
at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:475)
at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:228)
at org.eclipse.jetty.server.nio.SelectChannelConnector$1.run(SelectChannelConnector.java:257)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"qtp30604342-12 Acceptor0 SelectChannelConnector@127.0.0.1:8080
STARTED" prio=6 tid=0x479f8800 nid=0x1f58 runnable [0x4801f000]
java.lang.Thread.State: RUNNABLE
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:152)
- locked <0x18223600> (a java.lang.Object)
at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:92)
at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:830)
at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:529)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"pool-3-thread-1" prio=6 tid=0x479ec400 nid=0x18f4 waiting on
condition [0x47f1f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x182132a0> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"pool-1-thread-1" prio=6 tid=0x479a4400 nid=0x970 waiting on condition
[0x47e1f000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x17fdd938> (a
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- None
"HashSessionScavenger-0" daemon prio=6 tid=0x478e4400 nid=0x154c in
Object.wait() [0x47d0f000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.util.TimerThread.mainLoop(Timer.java:509)
- locked <0x17e57de8> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:462)
Locked ownable synchronizers:
- None
"Low Memory Detector" daemon prio=6 tid=0x46cf0000 nid=0xcfc runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"C1 CompilerThread0" daemon prio=10 tid=0x46cea000 nid=0x1494 waiting
on condition [0x00000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Attach Listener" daemon prio=10 tid=0x46ce8400 nid=0x1024 waiting on
condition [0x00000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Signal Dispatcher" daemon prio=10 tid=0x46ce7000 nid=0x13f8 runnable
[0x00000000]
java.lang.Thread.State: RUNNABLE
Locked ownable synchronizers:
- None
"Finalizer" daemon prio=8 tid=0x46cd6000 nid=0xb44 in Object.wait() [0x4705f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:118)
- locked <0x17df02c0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:134)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)
Locked ownable synchronizers:
- None
"Reference Handler" daemon prio=10 tid=0x46cd4c00 nid=0xc90 in
Object.wait() [0x46f5f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:485)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x17df0350> (a java.lang.ref.Reference$Lock)
Locked ownable synchronizers:
- None
"main" prio=6 tid=0x003e6400 nid=0x880 in Object.wait() [0x0061f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x17e5ea80> (a java.lang.Object)
at java.lang.Object.wait(Object.java:485)
at org.eclipse.jetty.util.thread.QueuedThreadPool.join(QueuedThreadPool.java:382)
- locked <0x17e5ea80> (a java.lang.Object)
at org.eclipse.jetty.server.Server.join(Server.java:399)
at <ourapp>.App.main(App.java:42)
Locked ownable synchronizers:
- None
"VM Thread" prio=10 tid=0x46cd1000 nid=0x1a74 runnable
"VM Periodic Task Thread" prio=10 tid=0x46cfac00 nid=0x1430 waiting on
condition
JNI global references: 1665
---
Stephen Duncan Jr
www.stephenduncanjr.com
Re: Solr 3.3 crashes after ~18 hours?
Posted by cowwoc <co...@bbs.darktech.org>.
I believe this issue is related to this Jetty bug report:
https://bugs.eclipse.org/bugs/show_bug.cgi?id=357318
Gili
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-3-3-crashes-after-18-hours-tp3218496p3655937.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr 3.3 crashes after ~18 hours?
Posted by alexander sulz <a....@digiconcept.net>.
Am 19.08.2011 16:43, schrieb Yonik Seeley:
> On Fri, Aug 19, 2011 at 10:36 AM, alexander sulz<a....@digiconcept.net> wrote:
>> using lsof I think I pinned down the problem: too many open files!
>> I already doubled from 512 to 1024 once but it seems there are many SOCKETS
>> involved,
>> which are listed as "can't identify protocol", instead of "real files".
>> over time, the list grows and grows with these entries until.. it "crashs".
>> So Ive read several times the fix for this problem is to set the limit to a
>> ridiculous high number but
>> that seems a little bit of a crude fix. Why so many open sockets in the
>> first place?
> What are you using as a client to talk to solr?
> You need to look at both the update side and the query side.
> Using persistent connections is the best all-around, but if not, be
> sure to close the connections in the client.
>
> -Yonik
> http://www.lucidimagination.com
I use PHP to talk to solr, this one to be exact
http://code.google.com/p/solr-php-client/ version r22 i guess.
I'll try updating it and see what happens..
Re: Solr 3.3 crashes after ~18 hours?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Aug 19, 2011 at 10:36 AM, alexander sulz <a....@digiconcept.net> wrote:
> using lsof I think I pinned down the problem: too many open files!
> I already doubled from 512 to 1024 once but it seems there are many SOCKETS
> involved,
> which are listed as "can't identify protocol", instead of "real files".
> over time, the list grows and grows with these entries until.. it "crashs".
> So Ive read several times the fix for this problem is to set the limit to a
> ridiculous high number but
> that seems a little bit of a crude fix. Why so many open sockets in the
> first place?
What are you using as a client to talk to solr?
You need to look at both the update side and the query side.
Using persistent connections is the best all-around, but if not, be
sure to close the connections in the client.
-Yonik
http://www.lucidimagination.com
Re: Solr 3.3 crashes after ~18 hours?
Posted by alexander sulz <a....@digiconcept.net>.
Am 19.08.2011 15:48, schrieb alexander sulz:
> Am 10.08.2011 17:11, schrieb Yonik Seeley:
>> On Wed, Aug 10, 2011 at 11:00 AM, alexander
>> sulz<a....@digiconcept.net> wrote:
>>> Okay, with this command it hangs.
>> It doesn't look like a hang from this thread dump. It doesn't look
>> like any solr requests are executing at the time the dump was taken.
>>
>> Did you do this from the command line?
>> curl "http://localhost:8983/solr/update?commit=true"
>>
>> Are you saying that the curl command just hung and never returned?
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>> Also: I managed to get a Thread Dump (attached).
>>>
>>> regards
>>>
>>> Am 05.08.2011 15:08, schrieb Yonik Seeley:
>>>> On Fri, Aug 5, 2011 at 7:33 AM, alexander sulz<a....@digiconcept.net>
>>>> wrote:
>>>>> Usually you get a XML-Response when doing commits or optimize, in
>>>>> this
>>>>> case
>>>>> I get nothing
>>>>> in return, but the site ( http://[...]/solr/update?optimize=true )
>>>>> DOESN'T
>>>>> load forever or anything.
>>>>> It doesn't hang! I just get a blank page / empty response.
>>>> Sounds like you are doing it from a browser?
>>>> Can you try it from the command line? It should give back some sort
>>>> of response (or hang waiting for a response).
>>>>
>>>> curl "http://localhost:8983/solr/update?commit=true"
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>>
>>>>
>>>>> I use the stuff in the example folder, the only changes i made was
>>>>> enable
>>>>> logging and changing the port to 8985.
>>>>> I'll try getting a thread dump if it happens again!
>>>>> So far its looking good with having allocated more memory to it.
>>>>>
>>>>> Am 04.08.2011 16:08, schrieb Yonik Seeley:
>>>>>> On Thu, Aug 4, 2011 at 8:09 AM, alexander
>>>>>> sulz<a....@digiconcept.net>
>>>>>> wrote:
>>>>>>> Thank you for the many replies!
>>>>>>>
>>>>>>> Like I said, I couldn't find anything in logs created by solr.
>>>>>>> I just had a look at the /var/logs/messages and there wasn't
>>>>>>> anything
>>>>>>> either.
>>>>>>>
>>>>>>> What I mean by crash is that the process is still there and http
>>>>>>> GET
>>>>>>> pings
>>>>>>> would return 200
>>>>>>> but when i try visiting /solr/admin, I'd get a blank page! The
>>>>>>> server
>>>>>>> ignores any incoming updates or commits,
>>>>>> "ignores" means what? The request hangs? If so, could you get a
>>>>>> thread
>>>>>> dump?
>>>>>>
>>>>>> Do queries work (like /solr/select?q=*:*) ?
>>>>>>
>>>>>>> thous throwing no errors, no 503's.. It's like the server has a
>>>>>>> blackout
>>>>>>> and
>>>>>>> stares blankly into space.
>>>>>> Are you using a different servlet container than what is shipped
>>>>>> with
>>>>>> solr?
>>>>>> If you did start with the solr "example" server, what jetty
>>>>>> configuration changes have you made?
>>>>>>
>>>>>> -Yonik
>>>>>> http://www.lucidimagination.com
>>>
> Sigh it happened again, but I have a clue: before the crash I was
> deleting some entries but haven't optimized afterwards, then, when I
> tried indexing something, solr "crashed" again (responsive but just
> blank/empty returns).
>
> I've just tried it again (doing the curl command while solr is its
> "zombie state")
> and i get the following reply from curl: "curl: (52) Empty reply from
> server"
>
> Also, I updated my Java so the HotSpot version is now 20.1-b3
using lsof I think I pinned down the problem: too many open files!
I already doubled from 512 to 1024 once but it seems there are many
SOCKETS involved,
which are listed as "can't identify protocol", instead of "real files".
over time, the list grows and grows with these entries until.. it "crashs".
So Ive read several times the fix for this problem is to set the limit
to a ridiculous high number but
that seems a little bit of a crude fix. Why so many open sockets in the
first place?
Re: Solr 3.3 crashes after ~18 hours?
Posted by alexander sulz <a....@digiconcept.net>.
Am 10.08.2011 17:11, schrieb Yonik Seeley:
> On Wed, Aug 10, 2011 at 11:00 AM, alexander sulz<a....@digiconcept.net> wrote:
>> Okay, with this command it hangs.
> It doesn't look like a hang from this thread dump. It doesn't look
> like any solr requests are executing at the time the dump was taken.
>
> Did you do this from the command line?
> curl "http://localhost:8983/solr/update?commit=true"
>
> Are you saying that the curl command just hung and never returned?
>
> -Yonik
> http://www.lucidimagination.com
>
>> Also: I managed to get a Thread Dump (attached).
>>
>> regards
>>
>> Am 05.08.2011 15:08, schrieb Yonik Seeley:
>>> On Fri, Aug 5, 2011 at 7:33 AM, alexander sulz<a....@digiconcept.net>
>>> wrote:
>>>> Usually you get a XML-Response when doing commits or optimize, in this
>>>> case
>>>> I get nothing
>>>> in return, but the site ( http://[...]/solr/update?optimize=true )
>>>> DOESN'T
>>>> load forever or anything.
>>>> It doesn't hang! I just get a blank page / empty response.
>>> Sounds like you are doing it from a browser?
>>> Can you try it from the command line? It should give back some sort
>>> of response (or hang waiting for a response).
>>>
>>> curl "http://localhost:8983/solr/update?commit=true"
>>>
>>> -Yonik
>>> http://www.lucidimagination.com
>>>
>>>
>>>> I use the stuff in the example folder, the only changes i made was enable
>>>> logging and changing the port to 8985.
>>>> I'll try getting a thread dump if it happens again!
>>>> So far its looking good with having allocated more memory to it.
>>>>
>>>> Am 04.08.2011 16:08, schrieb Yonik Seeley:
>>>>> On Thu, Aug 4, 2011 at 8:09 AM, alexander sulz<a....@digiconcept.net>
>>>>> wrote:
>>>>>> Thank you for the many replies!
>>>>>>
>>>>>> Like I said, I couldn't find anything in logs created by solr.
>>>>>> I just had a look at the /var/logs/messages and there wasn't anything
>>>>>> either.
>>>>>>
>>>>>> What I mean by crash is that the process is still there and http GET
>>>>>> pings
>>>>>> would return 200
>>>>>> but when i try visiting /solr/admin, I'd get a blank page! The server
>>>>>> ignores any incoming updates or commits,
>>>>> "ignores" means what? The request hangs? If so, could you get a thread
>>>>> dump?
>>>>>
>>>>> Do queries work (like /solr/select?q=*:*) ?
>>>>>
>>>>>> thous throwing no errors, no 503's.. It's like the server has a
>>>>>> blackout
>>>>>> and
>>>>>> stares blankly into space.
>>>>> Are you using a different servlet container than what is shipped with
>>>>> solr?
>>>>> If you did start with the solr "example" server, what jetty
>>>>> configuration changes have you made?
>>>>>
>>>>> -Yonik
>>>>> http://www.lucidimagination.com
>>
Sigh it happened again, but I have a clue: before the crash I was
deleting some entries but haven't optimized afterwards, then, when I
tried indexing something, solr "crashed" again (responsive but just
blank/empty returns).
I've just tried it again (doing the curl command while solr is its
"zombie state")
and i get the following reply from curl: "curl: (52) Empty reply from
server"
Also, I updated my Java so the HotSpot version is now 20.1-b3
Re: Solr 3.3 crashes after ~18 hours?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Wed, Aug 10, 2011 at 11:00 AM, alexander sulz <a....@digiconcept.net> wrote:
> Okay, with this command it hangs.
It doesn't look like a hang from this thread dump. It doesn't look
like any solr requests are executing at the time the dump was taken.
Did you do this from the command line?
curl "http://localhost:8983/solr/update?commit=true"
Are you saying that the curl command just hung and never returned?
-Yonik
http://www.lucidimagination.com
> Also: I managed to get a Thread Dump (attached).
>
> regards
>
> Am 05.08.2011 15:08, schrieb Yonik Seeley:
>>
>> On Fri, Aug 5, 2011 at 7:33 AM, alexander sulz<a....@digiconcept.net>
>> wrote:
>>>
>>> Usually you get a XML-Response when doing commits or optimize, in this
>>> case
>>> I get nothing
>>> in return, but the site ( http://[...]/solr/update?optimize=true )
>>> DOESN'T
>>> load forever or anything.
>>> It doesn't hang! I just get a blank page / empty response.
>>
>> Sounds like you are doing it from a browser?
>> Can you try it from the command line? It should give back some sort
>> of response (or hang waiting for a response).
>>
>> curl "http://localhost:8983/solr/update?commit=true"
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>> I use the stuff in the example folder, the only changes i made was enable
>>> logging and changing the port to 8985.
>>> I'll try getting a thread dump if it happens again!
>>> So far its looking good with having allocated more memory to it.
>>>
>>> Am 04.08.2011 16:08, schrieb Yonik Seeley:
>>>>
>>>> On Thu, Aug 4, 2011 at 8:09 AM, alexander sulz<a....@digiconcept.net>
>>>> wrote:
>>>>>
>>>>> Thank you for the many replies!
>>>>>
>>>>> Like I said, I couldn't find anything in logs created by solr.
>>>>> I just had a look at the /var/logs/messages and there wasn't anything
>>>>> either.
>>>>>
>>>>> What I mean by crash is that the process is still there and http GET
>>>>> pings
>>>>> would return 200
>>>>> but when i try visiting /solr/admin, I'd get a blank page! The server
>>>>> ignores any incoming updates or commits,
>>>>
>>>> "ignores" means what? The request hangs? If so, could you get a thread
>>>> dump?
>>>>
>>>> Do queries work (like /solr/select?q=*:*) ?
>>>>
>>>>> thous throwing no errors, no 503's.. It's like the server has a
>>>>> blackout
>>>>> and
>>>>> stares blankly into space.
>>>>
>>>> Are you using a different servlet container than what is shipped with
>>>> solr?
>>>> If you did start with the solr "example" server, what jetty
>>>> configuration changes have you made?
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>
>
>
Re: Solr 3.3 crashes after ~18 hours?
Posted by alexander sulz <a....@digiconcept.net>.
Okay, with this command it hangs.
Also: I managed to get a Thread Dump (attached).
regards
Am 05.08.2011 15:08, schrieb Yonik Seeley:
> On Fri, Aug 5, 2011 at 7:33 AM, alexander sulz<a....@digiconcept.net> wrote:
>> Usually you get a XML-Response when doing commits or optimize, in this case
>> I get nothing
>> in return, but the site ( http://[...]/solr/update?optimize=true ) DOESN'T
>> load forever or anything.
>> It doesn't hang! I just get a blank page / empty response.
> Sounds like you are doing it from a browser?
> Can you try it from the command line? It should give back some sort
> of response (or hang waiting for a response).
>
> curl "http://localhost:8983/solr/update?commit=true"
>
> -Yonik
> http://www.lucidimagination.com
>
>
>> I use the stuff in the example folder, the only changes i made was enable
>> logging and changing the port to 8985.
>> I'll try getting a thread dump if it happens again!
>> So far its looking good with having allocated more memory to it.
>>
>> Am 04.08.2011 16:08, schrieb Yonik Seeley:
>>> On Thu, Aug 4, 2011 at 8:09 AM, alexander sulz<a....@digiconcept.net>
>>> wrote:
>>>> Thank you for the many replies!
>>>>
>>>> Like I said, I couldn't find anything in logs created by solr.
>>>> I just had a look at the /var/logs/messages and there wasn't anything
>>>> either.
>>>>
>>>> What I mean by crash is that the process is still there and http GET
>>>> pings
>>>> would return 200
>>>> but when i try visiting /solr/admin, I'd get a blank page! The server
>>>> ignores any incoming updates or commits,
>>> "ignores" means what? The request hangs? If so, could you get a thread
>>> dump?
>>>
>>> Do queries work (like /solr/select?q=*:*) ?
>>>
>>>> thous throwing no errors, no 503's.. It's like the server has a blackout
>>>> and
>>>> stares blankly into space.
>>> Are you using a different servlet container than what is shipped with
>>> solr?
>>> If you did start with the solr "example" server, what jetty
>>> configuration changes have you made?
>>>
>>> -Yonik
>>> http://www.lucidimagination.com
>>
Re: Solr 3.3 crashes after ~18 hours?
Posted by Bernd Fehling <be...@uni-bielefeld.de>.
Hi, googling "hotspot server 19.1-b02" shows that you are not alone
with hanging threads and crashes. And not only with solr.
Maybe try another JAVA?
Bernd
Am 10.08.2011 17:00, schrieb alexander sulz:
> Okay, with this command it hangs.
> Also: I managed to get a Thread Dump (attached).
>
> regards
>
> Am 05.08.2011 15:08, schrieb Yonik Seeley:
>> On Fri, Aug 5, 2011 at 7:33 AM, alexander sulz<a....@digiconcept.net> wrote:
>>> Usually you get a XML-Response when doing commits or optimize, in this case
>>> I get nothing
>>> in return, but the site ( http://[...]/solr/update?optimize=true ) DOESN'T
>>> load forever or anything.
>>> It doesn't hang! I just get a blank page / empty response.
>> Sounds like you are doing it from a browser?
>> Can you try it from the command line? It should give back some sort
>> of response (or hang waiting for a response).
>>
>> curl "http://localhost:8983/solr/update?commit=true"
>>
>> -Yonik
>> http://www.lucidimagination.com
>>
>>
>>> I use the stuff in the example folder, the only changes i made was enable
>>> logging and changing the port to 8985.
>>> I'll try getting a thread dump if it happens again!
>>> So far its looking good with having allocated more memory to it.
>>>
>>> Am 04.08.2011 16:08, schrieb Yonik Seeley:
>>>> On Thu, Aug 4, 2011 at 8:09 AM, alexander sulz<a....@digiconcept.net>
>>>> wrote:
>>>>> Thank you for the many replies!
>>>>>
>>>>> Like I said, I couldn't find anything in logs created by solr.
>>>>> I just had a look at the /var/logs/messages and there wasn't anything
>>>>> either.
>>>>>
>>>>> What I mean by crash is that the process is still there and http GET
>>>>> pings
>>>>> would return 200
>>>>> but when i try visiting /solr/admin, I'd get a blank page! The server
>>>>> ignores any incoming updates or commits,
>>>> "ignores" means what? The request hangs? If so, could you get a thread
>>>> dump?
>>>>
>>>> Do queries work (like /solr/select?q=*:*) ?
>>>>
>>>>> thous throwing no errors, no 503's.. It's like the server has a blackout
>>>>> and
>>>>> stares blankly into space.
>>>> Are you using a different servlet container than what is shipped with
>>>> solr?
>>>> If you did start with the solr "example" server, what jetty
>>>> configuration changes have you made?
>>>>
>>>> -Yonik
>>>> http://www.lucidimagination.com
>>>
>
--
*************************************************************
Bernd Fehling Universitätsbibliothek Bielefeld
Dipl.-Inform. (FH) Universitätsstr. 25
Tel. +49 521 106-4060 Fax. +49 521 106-4052
bernd.fehling@uni-bielefeld.de 33615 Bielefeld
BASE - Bielefeld Academic Search Engine - www.base-search.net
*************************************************************
Re: Solr 3.3 crashes after ~18 hours?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Fri, Aug 5, 2011 at 7:33 AM, alexander sulz <a....@digiconcept.net> wrote:
> Usually you get a XML-Response when doing commits or optimize, in this case
> I get nothing
> in return, but the site ( http://[...]/solr/update?optimize=true ) DOESN'T
> load forever or anything.
> It doesn't hang! I just get a blank page / empty response.
Sounds like you are doing it from a browser?
Can you try it from the command line? It should give back some sort
of response (or hang waiting for a response).
curl "http://localhost:8983/solr/update?commit=true"
-Yonik
http://www.lucidimagination.com
> I use the stuff in the example folder, the only changes i made was enable
> logging and changing the port to 8985.
> I'll try getting a thread dump if it happens again!
> So far its looking good with having allocated more memory to it.
>
> Am 04.08.2011 16:08, schrieb Yonik Seeley:
>>
>> On Thu, Aug 4, 2011 at 8:09 AM, alexander sulz<a....@digiconcept.net>
>> wrote:
>>>
>>> Thank you for the many replies!
>>>
>>> Like I said, I couldn't find anything in logs created by solr.
>>> I just had a look at the /var/logs/messages and there wasn't anything
>>> either.
>>>
>>> What I mean by crash is that the process is still there and http GET
>>> pings
>>> would return 200
>>> but when i try visiting /solr/admin, I'd get a blank page! The server
>>> ignores any incoming updates or commits,
>>
>> "ignores" means what? The request hangs? If so, could you get a thread
>> dump?
>>
>> Do queries work (like /solr/select?q=*:*) ?
>>
>>> thous throwing no errors, no 503's.. It's like the server has a blackout
>>> and
>>> stares blankly into space.
>>
>> Are you using a different servlet container than what is shipped with
>> solr?
>> If you did start with the solr "example" server, what jetty
>> configuration changes have you made?
>>
>> -Yonik
>> http://www.lucidimagination.com
>
>
Re: Solr 3.3 crashes after ~18 hours?
Posted by alexander sulz <a....@digiconcept.net>.
Usually you get a XML-Response when doing commits or optimize, in this
case I get nothing
in return, but the site ( http://[...]/solr/update?optimize=true )
DOESN'T load forever or anything.
It doesn't hang! I just get a blank page / empty response.
I use the stuff in the example folder, the only changes i made was
enable logging and changing the port to 8985.
I'll try getting a thread dump if it happens again!
So far its looking good with having allocated more memory to it.
Am 04.08.2011 16:08, schrieb Yonik Seeley:
> On Thu, Aug 4, 2011 at 8:09 AM, alexander sulz<a....@digiconcept.net> wrote:
>> Thank you for the many replies!
>>
>> Like I said, I couldn't find anything in logs created by solr.
>> I just had a look at the /var/logs/messages and there wasn't anything
>> either.
>>
>> What I mean by crash is that the process is still there and http GET pings
>> would return 200
>> but when i try visiting /solr/admin, I'd get a blank page! The server
>> ignores any incoming updates or commits,
> "ignores" means what? The request hangs? If so, could you get a thread dump?
>
> Do queries work (like /solr/select?q=*:*) ?
>
>> thous throwing no errors, no 503's.. It's like the server has a blackout and
>> stares blankly into space.
> Are you using a different servlet container than what is shipped with solr?
> If you did start with the solr "example" server, what jetty
> configuration changes have you made?
>
> -Yonik
> http://www.lucidimagination.com
Re: Solr 3.3 crashes after ~18 hours?
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Thu, Aug 4, 2011 at 8:09 AM, alexander sulz <a....@digiconcept.net> wrote:
> Thank you for the many replies!
>
> Like I said, I couldn't find anything in logs created by solr.
> I just had a look at the /var/logs/messages and there wasn't anything
> either.
>
> What I mean by crash is that the process is still there and http GET pings
> would return 200
> but when i try visiting /solr/admin, I'd get a blank page! The server
> ignores any incoming updates or commits,
"ignores" means what? The request hangs? If so, could you get a thread dump?
Do queries work (like /solr/select?q=*:*) ?
> thous throwing no errors, no 503's.. It's like the server has a blackout and
> stares blankly into space.
Are you using a different servlet container than what is shipped with solr?
If you did start with the solr "example" server, what jetty
configuration changes have you made?
-Yonik
http://www.lucidimagination.com
Re: Solr 3.3 crashes after ~18 hours?
Posted by alexander sulz <a....@digiconcept.net>.
Thank you for the many replies!
Like I said, I couldn't find anything in logs created by solr.
I just had a look at the /var/logs/messages and there wasn't anything
either.
What I mean by crash is that the process is still there and http GET
pings would return 200
but when i try visiting /solr/admin, I'd get a blank page! The server
ignores any incoming updates or commits,
thous throwing no errors, no 503's.. It's like the server has a blackout
and stares blankly into space.
I just gave allocated more memory like proposed and will keep an eye on
it if the problem still persists.
Thank you guys, you are awesome.
Am 02.08.2011 15:23, schrieb François Schiettecatte:
> Assuming you are running on Linux, you might want to check /var/log/messages too (the location might vary), I think the kernel logs forced process termination there. I recall that the kernel will usually picks the process consuming the most memory, there may be other factors involved too.
>
> François
>
> On Aug 2, 2011, at 9:04 AM, wakemaster 39 wrote:
>
>> Monitor your memory usage. I use to encounter a problem like this before
>> where nothing was in the logs and the process was just gone.
>>
>> Turned out my system was out odd memory and swap got used up because of
>> another process which then forced the kernel to start killing off processes.
>> Google OOM linux and you will find plenty of other programs and people with
>> a similar problem.
>>
>> Cameron
>> On Aug 2, 2011 6:02 AM, "alexander sulz"<a....@digiconcept.net> wrote:
>>> Hello folks,
>>>
>>> I'm using the latest stable Solr release -> 3.3 and I encounter strange
>>> phenomena with it.
>>> After about 19 hours it just crashes, but I can't find anything in the
>>> logs, no exceptions, no warnings,
>>> no suspicious info entries..
>>>
>>> I have an index-job running from 6am to 8pm every 10 minutes. After each
>>> job there is a commit.
>>> An optimize-job is done twice a day at 12:15pm and 9:15pm.
>>>
>>> Does anyone have an idea what could possibly be wrong or where to look
>>> for further debug info?
>>>
>>> regards and thank you
>>> alex
Re: Solr 3.3 crashes after ~18 hours?
Posted by François Schiettecatte <fs...@gmail.com>.
Assuming you are running on Linux, you might want to check /var/log/messages too (the location might vary), I think the kernel logs forced process termination there. I recall that the kernel will usually picks the process consuming the most memory, there may be other factors involved too.
François
On Aug 2, 2011, at 9:04 AM, wakemaster 39 wrote:
> Monitor your memory usage. I use to encounter a problem like this before
> where nothing was in the logs and the process was just gone.
>
> Turned out my system was out odd memory and swap got used up because of
> another process which then forced the kernel to start killing off processes.
> Google OOM linux and you will find plenty of other programs and people with
> a similar problem.
>
> Cameron
> On Aug 2, 2011 6:02 AM, "alexander sulz" <a....@digiconcept.net> wrote:
>> Hello folks,
>>
>> I'm using the latest stable Solr release -> 3.3 and I encounter strange
>> phenomena with it.
>> After about 19 hours it just crashes, but I can't find anything in the
>> logs, no exceptions, no warnings,
>> no suspicious info entries..
>>
>> I have an index-job running from 6am to 8pm every 10 minutes. After each
>> job there is a commit.
>> An optimize-job is done twice a day at 12:15pm and 9:15pm.
>>
>> Does anyone have an idea what could possibly be wrong or where to look
>> for further debug info?
>>
>> regards and thank you
>> alex
Re: Solr 3.3 crashes after ~18 hours?
Posted by wakemaster 39 <wa...@z33k.com>.
Monitor your memory usage. I use to encounter a problem like this before
where nothing was in the logs and the process was just gone.
Turned out my system was out odd memory and swap got used up because of
another process which then forced the kernel to start killing off processes.
Google OOM linux and you will find plenty of other programs and people with
a similar problem.
Cameron
On Aug 2, 2011 6:02 AM, "alexander sulz" <a....@digiconcept.net> wrote:
> Hello folks,
>
> I'm using the latest stable Solr release -> 3.3 and I encounter strange
> phenomena with it.
> After about 19 hours it just crashes, but I can't find anything in the
> logs, no exceptions, no warnings,
> no suspicious info entries..
>
> I have an index-job running from 6am to 8pm every 10 minutes. After each
> job there is a commit.
> An optimize-job is done twice a day at 12:15pm and 9:15pm.
>
> Does anyone have an idea what could possibly be wrong or where to look
> for further debug info?
>
> regards and thank you
> alex
Re: Solr 3.3 crashes after ~18 hours?
Posted by Pranav Prakash <pr...@gmail.com>.
What do you mean by it just crashes? Does the process stops execution? Does
it takes too long to respond which might result in lots of 503s in your
application? Does the system run out of resources?
Are you indexing and serving from the same server? It happened once with us
that Solr was performing commit and then optimize while the load from app
server was at its peak. This caused slow response from search server, which
caused requests getting stacked up at app server and causing 503s. Could you
look if you have a similar syndrome?
*Pranav Prakash*
"temet nosce"
Twitter <http://twitter.com/pranavprakash> | Blog <http://blog.myblive.com> |
Google <http://www.google.com/profiles/pranny>
On Tue, Aug 2, 2011 at 15:31, alexander sulz <a....@digiconcept.net> wrote:
> Hello folks,
>
> I'm using the latest stable Solr release -> 3.3 and I encounter strange
> phenomena with it.
> After about 19 hours it just crashes, but I can't find anything in the
> logs, no exceptions, no warnings,
> no suspicious info entries..
>
> I have an index-job running from 6am to 8pm every 10 minutes. After each
> job there is a commit.
> An optimize-job is done twice a day at 12:15pm and 9:15pm.
>
> Does anyone have an idea what could possibly be wrong or where to look for
> further debug info?
>
> regards and thank you
> alex
>