You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Kevin Osborn <ke...@cbsi.com> on 2013/09/03 23:15:34 UTC
Solr Cloud hangs when replicating updates
I was having problems updating SolrCloud with a large batch of records. The
records are coming in bursts with lulls between updates.
At first, I just tried large updates of 100,000 records at a time.
Eventually, this caused Solr to hang. When hung, I can still query Solr.
But I cannot do any deletes or other updates to the index.
At first, my updates were going as SolrJ CSV posts. I have also tried local
file updates and had similar results. I finally slowed things down to just
use SolrJ's Update feature, which is basically just JavaBin. I am also
sending over just 100 at a time in 10 threads. Again, it eventually hung.
Sometimes, Solr hangs in the first couple of chunks. Other times, it hangs
right away.
These are my commit settings:
<autoCommit>
<maxTime>15000</maxTime>
<maxDocs>5000</maxDocs>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>30000</maxTime>
</autoSoftCommit>
I have tried quite a few variations with the same results. I also tried
various JVM settings with the same results. The only variable seems to be
that reducing the cluster size from 2 to 1 is the only thing that helps.
I also did a jstack trace. I did not see any explicit deadlocks, but I did
see quite a few threads in WAITING or TIMED_WAITING. It is typically
something like this:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074039a450> (a
java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
at
org.apache.solr.util.AdjustableSemaphore.acquire(AdjustableSemaphore.java:61)
at
org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:418)
at
org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:368)
at
org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:300)
at
org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:139)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:474)
at
org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:395)
at
org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:44)
at
org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:364)
at org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:31)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
It basically appears that Solr gets stuck while trying to acquire a
semaphore that never becomes available.
Anyone have any ideas? This is definitely causing major problems for us.
--
*KEVIN OSBORN*
LEAD SOFTWARE ENGINEER
CNET Content Solutions
OFFICE 949.399.8714
CELL 949.310.4677 SKYPE osbornk
5 Park Plaza, Suite 600, Irvine, CA 92614
[image: CNET Content Solutions]
RE: Solr Cloud hangs when replicating updates
Posted by Greg Walters <gw...@sherpaanalytics.com>.
Kevin,
Take a look at http://lucene.472066.n3.nabble.com/updating-docs-in-solr-cloud-hangs-td4067388.html and https://issues.apache.org/jira/browse/SOLR-4816. I had the same issue that you're reporting for a while then I applied the patch from SOLR-4816 to my clients and the problems went away. If you don't feel like applying the patch it looks like it should be included in the release of version 4.5. Also note that the problem happens more frequently when the replication factor is greater than 1.
Thanks,
Greg
-----Original Message-----
From: kevin.osborn@cbsinteractive.com [mailto:kevin.osborn@cbsinteractive.com] On Behalf Of Kevin Osborn
Sent: Tuesday, September 03, 2013 4:16 PM
To: solr-user
Subject: Solr Cloud hangs when replicating updates
I was having problems updating SolrCloud with a large batch of records. The records are coming in bursts with lulls between updates.
At first, I just tried large updates of 100,000 records at a time.
Eventually, this caused Solr to hang. When hung, I can still query Solr.
But I cannot do any deletes or other updates to the index.
At first, my updates were going as SolrJ CSV posts. I have also tried local file updates and had similar results. I finally slowed things down to just use SolrJ's Update feature, which is basically just JavaBin. I am also sending over just 100 at a time in 10 threads. Again, it eventually hung.
Sometimes, Solr hangs in the first couple of chunks. Other times, it hangs right away.
These are my commit settings:
<autoCommit>
<maxTime>15000</maxTime>
<maxDocs>5000</maxDocs>
<openSearcher>false</openSearcher>
</autoCommit>
<autoSoftCommit>
<maxTime>30000</maxTime>
</autoSoftCommit>
I have tried quite a few variations with the same results. I also tried various JVM settings with the same results. The only variable seems to be that reducing the cluster size from 2 to 1 is the only thing that helps.
I also did a jstack trace. I did not see any explicit deadlocks, but I did see quite a few threads in WAITING or TIMED_WAITING. It is typically something like this:
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for <0x000000074039a450> (a
java.util.concurrent.Semaphore$NonfairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
at
org.apache.solr.util.AdjustableSemaphore.acquire(AdjustableSemaphore.java:61)
at
org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:418)
at
org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:368)
at
org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:300)
at
org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:139)
at
org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:474)
at
org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:395)
at
org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:44)
at
org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:364)
at org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:31)
at
org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
at
org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
at
org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
at
org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
at
org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
at
org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
at
org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
at
org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
at
org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
at
org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
at
org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
at
org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
at
org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
at
org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
at
org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
It basically appears that Solr gets stuck while trying to acquire a semaphore that never becomes available.
Anyone have any ideas? This is definitely causing major problems for us.
--
*KEVIN OSBORN*
LEAD SOFTWARE ENGINEER
CNET Content Solutions
OFFICE 949.399.8714
CELL 949.310.4677 SKYPE osbornk
5 Park Plaza, Suite 600, Irvine, CA 92614
[image: CNET Content Solutions]
Re: Solr Cloud hangs when replicating updates
Posted by mewmewball <me...@gmail.com>.
Hey guys,
We just hit a deadlock similar to this one on 4.5, and it seems to be
related to leaked connections probably due to
https://issues.apache.org/jira/browse/SOLR-4327. We're going to apply the
suggested change to add "method.abort()" in the finally block and see if it
fixes things.
Jessica
--
View this message in context: http://lucene.472066.n3.nabble.com/Solr-Cloud-hangs-when-replicating-updates-tp4088083p4095061.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Solr Cloud hangs when replicating updates
Posted by Kevin Osborn <ke...@cbsi.com>.
Thanks a ton Mark. I have tried SOLR-4816 and it didn't help. But I will
try Mark's patch next week, and see what happens.
-Kevin
On Thu, Sep 5, 2013 at 4:46 AM, Erick Erickson <er...@gmail.com>wrote:
> If you run into this again, try a jstack trace. You should see
> evidence of being stuck in SolrCmdDistributor on a variable
> called "semaphore"... On current 4x this is around line 420.
>
> If you're using SolrJ, then SOLR-4816 is another thing to try.
>
> But Mark's patch would be best of all to test, If that doesn't
> fix it then the jstack suggestion would at least tell us if it's
> the issue we think it is.
>
> FWIW,
> Erick
>
>
> On Wed, Sep 4, 2013 at 12:51 PM, Mark Miller <ma...@gmail.com>
> wrote:
>
> > It would be great if you could give this patch a try:
> > http://pastebin.com/raw.php?i=aaRWwSGP
> >
> > - Mark
> >
> >
> > On Wed, Sep 4, 2013 at 8:31 AM, Kevin Osborn <ke...@cbsi.com>
> > wrote:
> >
> > > Thanks. If there is anything I can do to help you resolve this issue,
> let
> > > me know.
> > >
> > > -Kevin
> > >
> > >
> > > On Wed, Sep 4, 2013 at 7:51 AM, Mark Miller <ma...@gmail.com>
> > wrote:
> > >
> > > > Ill look at fixing the root issue for 4.5. I've been putting it off
> for
> > > > way to long.
> > > >
> > > > Mark
> > > >
> > > > Sent from my iPhone
> > > >
> > > > On Sep 3, 2013, at 2:15 PM, Kevin Osborn <ke...@cbsi.com>
> > wrote:
> > > >
> > > > > I was having problems updating SolrCloud with a large batch of
> > records.
> > > > The
> > > > > records are coming in bursts with lulls between updates.
> > > > >
> > > > > At first, I just tried large updates of 100,000 records at a time.
> > > > > Eventually, this caused Solr to hang. When hung, I can still query
> > > Solr.
> > > > > But I cannot do any deletes or other updates to the index.
> > > > >
> > > > > At first, my updates were going as SolrJ CSV posts. I have also
> tried
> > > > local
> > > > > file updates and had similar results. I finally slowed things down
> to
> > > > just
> > > > > use SolrJ's Update feature, which is basically just JavaBin. I am
> > also
> > > > > sending over just 100 at a time in 10 threads. Again, it eventually
> > > hung.
> > > > >
> > > > > Sometimes, Solr hangs in the first couple of chunks. Other times,
> it
> > > > hangs
> > > > > right away.
> > > > >
> > > > > These are my commit settings:
> > > > >
> > > > > <autoCommit>
> > > > > <maxTime>15000</maxTime>
> > > > > <maxDocs>5000</maxDocs>
> > > > > <openSearcher>false</openSearcher>
> > > > > </autoCommit>
> > > > > <autoSoftCommit>
> > > > > <maxTime>30000</maxTime>
> > > > > </autoSoftCommit>
> > > > >
> > > > > I have tried quite a few variations with the same results. I also
> > tried
> > > > > various JVM settings with the same results. The only variable seems
> > to
> > > be
> > > > > that reducing the cluster size from 2 to 1 is the only thing that
> > > helps.
> > > > >
> > > > > I also did a jstack trace. I did not see any explicit deadlocks,
> but
> > I
> > > > did
> > > > > see quite a few threads in WAITING or TIMED_WAITING. It is
> typically
> > > > > something like this:
> > > > >
> > > > > java.lang.Thread.State: WAITING (parking)
> > > > > at sun.misc.Unsafe.park(Native Method)
> > > > > - parking to wait for <0x000000074039a450> (a
> > > > > java.util.concurrent.Semaphore$NonfairSync)
> > > > > at
> > > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> > > > > at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> > > > > at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> > > > > at
> > > > >
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> > > > > at
> java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.util.AdjustableSemaphore.acquire(AdjustableSemaphore.java:61)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:418)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:368)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:300)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:139)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:474)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:395)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:44)
> > > > > at
> > > > >
> > >
> org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:364)
> > > > > at
> > > > org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:31)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > > > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> > > > > at
> > > > >
> > >
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> > > > > at
> > > > >
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> > > > >
> > > > > It basically appears that Solr gets stuck while trying to acquire a
> > > > > semaphore that never becomes available.
> > > > >
> > > > > Anyone have any ideas? This is definitely causing major problems
> for
> > > us.
> > > > >
> > > > > --
> > > > > *KEVIN OSBORN*
> > > > > LEAD SOFTWARE ENGINEER
> > > > > CNET Content Solutions
> > > > > OFFICE 949.399.8714
> > > > > CELL 949.310.4677 SKYPE osbornk
> > > > > 5 Park Plaza, Suite 600, Irvine, CA 92614
> > > > > [image: CNET Content Solutions]
> > > >
> > >
> > >
> > >
> > > --
> > > *KEVIN OSBORN*
> > > LEAD SOFTWARE ENGINEER
> > > CNET Content Solutions
> > > OFFICE 949.399.8714
> > > CELL 949.310.4677 SKYPE osbornk
> > > 5 Park Plaza, Suite 600, Irvine, CA 92614
> > > [image: CNET Content Solutions]
> > >
> >
> >
> >
> > --
> > - Mark
> >
>
--
*KEVIN OSBORN*
LEAD SOFTWARE ENGINEER
CNET Content Solutions
OFFICE 949.399.8714
CELL 949.310.4677 SKYPE osbornk
5 Park Plaza, Suite 600, Irvine, CA 92614
[image: CNET Content Solutions]
Re: Solr Cloud hangs when replicating updates
Posted by Erick Erickson <er...@gmail.com>.
If you run into this again, try a jstack trace. You should see
evidence of being stuck in SolrCmdDistributor on a variable
called "semaphore"... On current 4x this is around line 420.
If you're using SolrJ, then SOLR-4816 is another thing to try.
But Mark's patch would be best of all to test, If that doesn't
fix it then the jstack suggestion would at least tell us if it's
the issue we think it is.
FWIW,
Erick
On Wed, Sep 4, 2013 at 12:51 PM, Mark Miller <ma...@gmail.com> wrote:
> It would be great if you could give this patch a try:
> http://pastebin.com/raw.php?i=aaRWwSGP
>
> - Mark
>
>
> On Wed, Sep 4, 2013 at 8:31 AM, Kevin Osborn <ke...@cbsi.com>
> wrote:
>
> > Thanks. If there is anything I can do to help you resolve this issue, let
> > me know.
> >
> > -Kevin
> >
> >
> > On Wed, Sep 4, 2013 at 7:51 AM, Mark Miller <ma...@gmail.com>
> wrote:
> >
> > > Ill look at fixing the root issue for 4.5. I've been putting it off for
> > > way to long.
> > >
> > > Mark
> > >
> > > Sent from my iPhone
> > >
> > > On Sep 3, 2013, at 2:15 PM, Kevin Osborn <ke...@cbsi.com>
> wrote:
> > >
> > > > I was having problems updating SolrCloud with a large batch of
> records.
> > > The
> > > > records are coming in bursts with lulls between updates.
> > > >
> > > > At first, I just tried large updates of 100,000 records at a time.
> > > > Eventually, this caused Solr to hang. When hung, I can still query
> > Solr.
> > > > But I cannot do any deletes or other updates to the index.
> > > >
> > > > At first, my updates were going as SolrJ CSV posts. I have also tried
> > > local
> > > > file updates and had similar results. I finally slowed things down to
> > > just
> > > > use SolrJ's Update feature, which is basically just JavaBin. I am
> also
> > > > sending over just 100 at a time in 10 threads. Again, it eventually
> > hung.
> > > >
> > > > Sometimes, Solr hangs in the first couple of chunks. Other times, it
> > > hangs
> > > > right away.
> > > >
> > > > These are my commit settings:
> > > >
> > > > <autoCommit>
> > > > <maxTime>15000</maxTime>
> > > > <maxDocs>5000</maxDocs>
> > > > <openSearcher>false</openSearcher>
> > > > </autoCommit>
> > > > <autoSoftCommit>
> > > > <maxTime>30000</maxTime>
> > > > </autoSoftCommit>
> > > >
> > > > I have tried quite a few variations with the same results. I also
> tried
> > > > various JVM settings with the same results. The only variable seems
> to
> > be
> > > > that reducing the cluster size from 2 to 1 is the only thing that
> > helps.
> > > >
> > > > I also did a jstack trace. I did not see any explicit deadlocks, but
> I
> > > did
> > > > see quite a few threads in WAITING or TIMED_WAITING. It is typically
> > > > something like this:
> > > >
> > > > java.lang.Thread.State: WAITING (parking)
> > > > at sun.misc.Unsafe.park(Native Method)
> > > > - parking to wait for <0x000000074039a450> (a
> > > > java.util.concurrent.Semaphore$NonfairSync)
> > > > at
> > > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> > > > at
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> > > > at
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> > > > at
> > > >
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> > > > at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> > > > at
> > > >
> > >
> >
> org.apache.solr.util.AdjustableSemaphore.acquire(AdjustableSemaphore.java:61)
> > > > at
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:418)
> > > > at
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:368)
> > > > at
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:300)
> > > > at
> > > >
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:139)
> > > > at
> > > >
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:474)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:395)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:44)
> > > > at
> > > >
> > org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:364)
> > > > at
> > > org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:31)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> > > > at
> > > >
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
> > > > at
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
> > > > at
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
> > > > at
> > > >
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> > > > at
> > > >
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> > > > at
> > > >
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> > > >
> > > > It basically appears that Solr gets stuck while trying to acquire a
> > > > semaphore that never becomes available.
> > > >
> > > > Anyone have any ideas? This is definitely causing major problems for
> > us.
> > > >
> > > > --
> > > > *KEVIN OSBORN*
> > > > LEAD SOFTWARE ENGINEER
> > > > CNET Content Solutions
> > > > OFFICE 949.399.8714
> > > > CELL 949.310.4677 SKYPE osbornk
> > > > 5 Park Plaza, Suite 600, Irvine, CA 92614
> > > > [image: CNET Content Solutions]
> > >
> >
> >
> >
> > --
> > *KEVIN OSBORN*
> > LEAD SOFTWARE ENGINEER
> > CNET Content Solutions
> > OFFICE 949.399.8714
> > CELL 949.310.4677 SKYPE osbornk
> > 5 Park Plaza, Suite 600, Irvine, CA 92614
> > [image: CNET Content Solutions]
> >
>
>
>
> --
> - Mark
>
Re: Solr Cloud hangs when replicating updates
Posted by Mark Miller <ma...@gmail.com>.
It would be great if you could give this patch a try:
http://pastebin.com/raw.php?i=aaRWwSGP
- Mark
On Wed, Sep 4, 2013 at 8:31 AM, Kevin Osborn <ke...@cbsi.com> wrote:
> Thanks. If there is anything I can do to help you resolve this issue, let
> me know.
>
> -Kevin
>
>
> On Wed, Sep 4, 2013 at 7:51 AM, Mark Miller <ma...@gmail.com> wrote:
>
> > Ill look at fixing the root issue for 4.5. I've been putting it off for
> > way to long.
> >
> > Mark
> >
> > Sent from my iPhone
> >
> > On Sep 3, 2013, at 2:15 PM, Kevin Osborn <ke...@cbsi.com> wrote:
> >
> > > I was having problems updating SolrCloud with a large batch of records.
> > The
> > > records are coming in bursts with lulls between updates.
> > >
> > > At first, I just tried large updates of 100,000 records at a time.
> > > Eventually, this caused Solr to hang. When hung, I can still query
> Solr.
> > > But I cannot do any deletes or other updates to the index.
> > >
> > > At first, my updates were going as SolrJ CSV posts. I have also tried
> > local
> > > file updates and had similar results. I finally slowed things down to
> > just
> > > use SolrJ's Update feature, which is basically just JavaBin. I am also
> > > sending over just 100 at a time in 10 threads. Again, it eventually
> hung.
> > >
> > > Sometimes, Solr hangs in the first couple of chunks. Other times, it
> > hangs
> > > right away.
> > >
> > > These are my commit settings:
> > >
> > > <autoCommit>
> > > <maxTime>15000</maxTime>
> > > <maxDocs>5000</maxDocs>
> > > <openSearcher>false</openSearcher>
> > > </autoCommit>
> > > <autoSoftCommit>
> > > <maxTime>30000</maxTime>
> > > </autoSoftCommit>
> > >
> > > I have tried quite a few variations with the same results. I also tried
> > > various JVM settings with the same results. The only variable seems to
> be
> > > that reducing the cluster size from 2 to 1 is the only thing that
> helps.
> > >
> > > I also did a jstack trace. I did not see any explicit deadlocks, but I
> > did
> > > see quite a few threads in WAITING or TIMED_WAITING. It is typically
> > > something like this:
> > >
> > > java.lang.Thread.State: WAITING (parking)
> > > at sun.misc.Unsafe.park(Native Method)
> > > - parking to wait for <0x000000074039a450> (a
> > > java.util.concurrent.Semaphore$NonfairSync)
> > > at
> > java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> > > at
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> > > at
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> > > at
> > >
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> > > at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> > > at
> > >
> >
> org.apache.solr.util.AdjustableSemaphore.acquire(AdjustableSemaphore.java:61)
> > > at
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:418)
> > > at
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:368)
> > > at
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:300)
> > > at
> > >
> >
> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:139)
> > > at
> > >
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:474)
> > > at
> > >
> >
> org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:395)
> > > at
> > >
> >
> org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:44)
> > > at
> > >
> org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:364)
> > > at
> > org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:31)
> > > at
> > >
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> > > at
> > >
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> > > at
> > >
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
> > > at
> > >
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> > > at
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> > > at
> > >
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> > > at
> > >
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
> > > at
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> > > at
> > >
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> > > at
> > >
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> > > at
> > >
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> > >
> > > It basically appears that Solr gets stuck while trying to acquire a
> > > semaphore that never becomes available.
> > >
> > > Anyone have any ideas? This is definitely causing major problems for
> us.
> > >
> > > --
> > > *KEVIN OSBORN*
> > > LEAD SOFTWARE ENGINEER
> > > CNET Content Solutions
> > > OFFICE 949.399.8714
> > > CELL 949.310.4677 SKYPE osbornk
> > > 5 Park Plaza, Suite 600, Irvine, CA 92614
> > > [image: CNET Content Solutions]
> >
>
>
>
> --
> *KEVIN OSBORN*
> LEAD SOFTWARE ENGINEER
> CNET Content Solutions
> OFFICE 949.399.8714
> CELL 949.310.4677 SKYPE osbornk
> 5 Park Plaza, Suite 600, Irvine, CA 92614
> [image: CNET Content Solutions]
>
--
- Mark
Re: Solr Cloud hangs when replicating updates
Posted by Kevin Osborn <ke...@cbsi.com>.
Thanks. If there is anything I can do to help you resolve this issue, let
me know.
-Kevin
On Wed, Sep 4, 2013 at 7:51 AM, Mark Miller <ma...@gmail.com> wrote:
> Ill look at fixing the root issue for 4.5. I've been putting it off for
> way to long.
>
> Mark
>
> Sent from my iPhone
>
> On Sep 3, 2013, at 2:15 PM, Kevin Osborn <ke...@cbsi.com> wrote:
>
> > I was having problems updating SolrCloud with a large batch of records.
> The
> > records are coming in bursts with lulls between updates.
> >
> > At first, I just tried large updates of 100,000 records at a time.
> > Eventually, this caused Solr to hang. When hung, I can still query Solr.
> > But I cannot do any deletes or other updates to the index.
> >
> > At first, my updates were going as SolrJ CSV posts. I have also tried
> local
> > file updates and had similar results. I finally slowed things down to
> just
> > use SolrJ's Update feature, which is basically just JavaBin. I am also
> > sending over just 100 at a time in 10 threads. Again, it eventually hung.
> >
> > Sometimes, Solr hangs in the first couple of chunks. Other times, it
> hangs
> > right away.
> >
> > These are my commit settings:
> >
> > <autoCommit>
> > <maxTime>15000</maxTime>
> > <maxDocs>5000</maxDocs>
> > <openSearcher>false</openSearcher>
> > </autoCommit>
> > <autoSoftCommit>
> > <maxTime>30000</maxTime>
> > </autoSoftCommit>
> >
> > I have tried quite a few variations with the same results. I also tried
> > various JVM settings with the same results. The only variable seems to be
> > that reducing the cluster size from 2 to 1 is the only thing that helps.
> >
> > I also did a jstack trace. I did not see any explicit deadlocks, but I
> did
> > see quite a few threads in WAITING or TIMED_WAITING. It is typically
> > something like this:
> >
> > java.lang.Thread.State: WAITING (parking)
> > at sun.misc.Unsafe.park(Native Method)
> > - parking to wait for <0x000000074039a450> (a
> > java.util.concurrent.Semaphore$NonfairSync)
> > at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> > at
> >
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> > at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> > at
> >
> org.apache.solr.util.AdjustableSemaphore.acquire(AdjustableSemaphore.java:61)
> > at
> >
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:418)
> > at
> >
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:368)
> > at
> >
> org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:300)
> > at
> >
> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:139)
> > at
> >
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:474)
> > at
> >
> org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:395)
> > at
> >
> org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:44)
> > at
> > org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:364)
> > at
> org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:31)
> > at
> >
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> > at
> >
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> > at
> >
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> > at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
> > at
> >
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> > at
> >
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> > at
> >
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> > at
> > org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> > at
> >
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> > at
> >
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> > at
> >
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> > at
> >
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> >
> > It basically appears that Solr gets stuck while trying to acquire a
> > semaphore that never becomes available.
> >
> > Anyone have any ideas? This is definitely causing major problems for us.
> >
> > --
> > *KEVIN OSBORN*
> > LEAD SOFTWARE ENGINEER
> > CNET Content Solutions
> > OFFICE 949.399.8714
> > CELL 949.310.4677 SKYPE osbornk
> > 5 Park Plaza, Suite 600, Irvine, CA 92614
> > [image: CNET Content Solutions]
>
--
*KEVIN OSBORN*
LEAD SOFTWARE ENGINEER
CNET Content Solutions
OFFICE 949.399.8714
CELL 949.310.4677 SKYPE osbornk
5 Park Plaza, Suite 600, Irvine, CA 92614
[image: CNET Content Solutions]
Re: Solr Cloud hangs when replicating updates
Posted by Mark Miller <ma...@gmail.com>.
Ill look at fixing the root issue for 4.5. I've been putting it off for way to long.
Mark
Sent from my iPhone
On Sep 3, 2013, at 2:15 PM, Kevin Osborn <ke...@cbsi.com> wrote:
> I was having problems updating SolrCloud with a large batch of records. The
> records are coming in bursts with lulls between updates.
>
> At first, I just tried large updates of 100,000 records at a time.
> Eventually, this caused Solr to hang. When hung, I can still query Solr.
> But I cannot do any deletes or other updates to the index.
>
> At first, my updates were going as SolrJ CSV posts. I have also tried local
> file updates and had similar results. I finally slowed things down to just
> use SolrJ's Update feature, which is basically just JavaBin. I am also
> sending over just 100 at a time in 10 threads. Again, it eventually hung.
>
> Sometimes, Solr hangs in the first couple of chunks. Other times, it hangs
> right away.
>
> These are my commit settings:
>
> <autoCommit>
> <maxTime>15000</maxTime>
> <maxDocs>5000</maxDocs>
> <openSearcher>false</openSearcher>
> </autoCommit>
> <autoSoftCommit>
> <maxTime>30000</maxTime>
> </autoSoftCommit>
>
> I have tried quite a few variations with the same results. I also tried
> various JVM settings with the same results. The only variable seems to be
> that reducing the cluster size from 2 to 1 is the only thing that helps.
>
> I also did a jstack trace. I did not see any explicit deadlocks, but I did
> see quite a few threads in WAITING or TIMED_WAITING. It is typically
> something like this:
>
> java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x000000074039a450> (a
> java.util.concurrent.Semaphore$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:994)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1303)
> at java.util.concurrent.Semaphore.acquire(Semaphore.java:317)
> at
> org.apache.solr.util.AdjustableSemaphore.acquire(AdjustableSemaphore.java:61)
> at
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:418)
> at
> org.apache.solr.update.SolrCmdDistributor.submit(SolrCmdDistributor.java:368)
> at
> org.apache.solr.update.SolrCmdDistributor.flushAdds(SolrCmdDistributor.java:300)
> at
> org.apache.solr.update.SolrCmdDistributor.distribAdd(SolrCmdDistributor.java:139)
> at
> org.apache.solr.update.processor.DistributedUpdateProcessor.processAdd(DistributedUpdateProcessor.java:474)
> at
> org.apache.solr.handler.loader.CSVLoaderBase.doAdd(CSVLoaderBase.java:395)
> at
> org.apache.solr.handler.loader.SingleThreadedCSVLoader.addDoc(CSVLoader.java:44)
> at
> org.apache.solr.handler.loader.CSVLoaderBase.load(CSVLoaderBase.java:364)
> at org.apache.solr.handler.loader.CSVLoader.load(CSVLoader.java:31)
> at
> org.apache.solr.handler.UpdateRequestHandler$1.load(UpdateRequestHandler.java:92)
> at
> org.apache.solr.handler.ContentStreamHandlerBase.handleRequestBody(ContentStreamHandlerBase.java:74)
> at
> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:135)
> at org.apache.solr.core.SolrCore.execute(SolrCore.java:1904)
> at
> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:659)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:362)
> at
> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:158)
> at
> org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1419)
> at
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:455)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
> at
> org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:533)
> at
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:231)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> at
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> at
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> at
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> at
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> at
> org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:255)
> at
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
>
> It basically appears that Solr gets stuck while trying to acquire a
> semaphore that never becomes available.
>
> Anyone have any ideas? This is definitely causing major problems for us.
>
> --
> *KEVIN OSBORN*
> LEAD SOFTWARE ENGINEER
> CNET Content Solutions
> OFFICE 949.399.8714
> CELL 949.310.4677 SKYPE osbornk
> 5 Park Plaza, Suite 600, Irvine, CA 92614
> [image: CNET Content Solutions]