You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by maephisto <my...@yahoo.com> on 2013/10/15 10:54:26 UTC
Debugging update request
As a followup to another thread, where I described how my SolrCloud sometimes
just stops accepting updates
http://lucene.472066.n3.nabble.com/Concurent-indexing-td4095409.html
I have a question, is there a way to debug or analyze the update request?
Verbose output or anything else?
It happens for me that when in the above situation, I'm using the post.sh
tool to post 1 doc and i get no feedback, it just hangs and waits.
10x!
--
View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Debugging update request
Posted by Chris Geeringh <ge...@gmail.com>.
I ran an import last night, and this morning my cloud wouldn't accept
updates. I'm running the latest 4.6 snapshot. I was importing with latest
solrj snapshot, and using java bin transport with CloudSolrServer.
The cluster had indexed ~1.3 million docs before no further updates were
accepted, querying still working.
I'll run jstack shortly and provide the results.
On Wednesday, October 16, 2013, michael.boom wrote:
> Thanks Erick!
>
> The version is 4.4.0.
>
> I'm posting 100k docs batches every 30-40 sec from each indexing client and
> sometimes two or more clients post in a very small timeframe. That's when i
> think the deadlock happens.
>
> I'll try to replicate the problem and check the thread dump.
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095821.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Debugging update request
Posted by "michael.boom" <my...@yahoo.com>.
Thanks Erick!
The version is 4.4.0.
I'm posting 100k docs batches every 30-40 sec from each indexing client and
sometimes two or more clients post in a very small timeframe. That's when i
think the deadlock happens.
I'll try to replicate the problem and check the thread dump.
--
View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095821.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Debugging update request
Posted by Erick Erickson <er...@gmail.com>.
@Michael:
Yep, that's the bit that's addressed by the two patches I referenced. If
you can try this with 4.5 (or the soon to be done 4.5.1), the problem
should go away.
@Chris:
I think you have a different issue. A very quick glance at your stack trace
doesn't really show anything outstanding. There are always a bunch of
threads waiting around for something to do that show up as "blocked". So
I'm pretty puzzled. Are your Solr logs showing anything when you try to
update after this occurs?
On Wed, Oct 16, 2013 at 11:32 AM, Chris Geeringh <ge...@gmail.com> wrote:
> Here is my jstack output... Lots of blocked threads.
>
> http://pastebin.com/1ktjBYbf
>
>
> On 16 October 2013 10:28, michael.boom <my...@yahoo.com> wrote:
>
> > I got the trace from jstack.
> > I found references to "semaphore" but not sure if this is what you meant.
> > Here's the trace:
> > http://pastebin.com/15QKAz7U
> >
> >
> >
> > --
> > View this message in context:
> >
> http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095847.html
> > Sent from the Solr - User mailing list archive at Nabble.com.
> >
>
Re: Debugging update request
Posted by Chris Geeringh <ge...@gmail.com>.
Here is my jstack output... Lots of blocked threads.
http://pastebin.com/1ktjBYbf
On 16 October 2013 10:28, michael.boom <my...@yahoo.com> wrote:
> I got the trace from jstack.
> I found references to "semaphore" but not sure if this is what you meant.
> Here's the trace:
> http://pastebin.com/15QKAz7U
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095847.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
Re: Debugging update request
Posted by "michael.boom" <my...@yahoo.com>.
I got the trace from jstack.
I found references to "semaphore" but not sure if this is what you meant.
Here's the trace:
http://pastebin.com/15QKAz7U
--
View this message in context: http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619p4095847.html
Sent from the Solr - User mailing list archive at Nabble.com.
Re: Debugging update request
Posted by Erick Erickson <er...@gmail.com>.
What version of Solr? There's a known issue
with distributed deadlock (since fixed) where
under heavy load from multiple clients to
multiple Solr nodes, you can get into a situation where
it will stop accepting updates, but _will_ continue
to serve queries.
If this describes your situation, dump the stack (jstack works)
and look for the word "semaphore" in the stack traces. If
Solr is waiting on that, that's your problem.
SOLR-4816 (4.5) should alleviate this if you're indexing
from SolrJ, and SOLR-5216 should fix it on the server.
Best,
Erick
On Tue, Oct 15, 2013 at 10:54 AM, maephisto <my...@yahoo.com> wrote:
> As a followup to another thread, where I described how my SolrCloud
> sometimes
> just stops accepting updates
> http://lucene.472066.n3.nabble.com/Concurent-indexing-td4095409.html
>
> I have a question, is there a way to debug or analyze the update request?
> Verbose output or anything else?
> It happens for me that when in the above situation, I'm using the post.sh
> tool to post 1 doc and i get no feedback, it just hangs and waits.
> 10x!
>
>
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Debugging-update-request-tp4095619.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>