You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "matt knecht (JIRA)" <ji...@apache.org> on 2013/04/10 17:30:16 UTC
[jira] [Comment Edited] (SOLR-4696) All threads become blocked
resulting in hang when bulk adding
[ https://issues.apache.org/jira/browse/SOLR-4696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627906#comment-13627906 ]
matt knecht edited comment on SOLR-4696 at 4/10/13 3:29 PM:
------------------------------------------------------------
Nagios shows that both solr nodes had a peak of about 50 open connections, 150 in TIME_WAIT and maybe a dozen in CLOSE_WAIT. Once the problem manifests, I see around 50 open, and 100 stuck in CLOSE_WAIT. Both nodes look nearly identical.
The host doing the insert/updates shows a peak of around 100 open connections and around 9000 in TIME_WAIT. Once the problem manifests all those TIME_WAIT connections clear. During one test, I gave tomcat a thread pool of 4096 and saw those eaten up as well.
On a successful load of all 350k documents, I see a similar pattern, perhaps a few less connections in TIME_WAIT.
The PHP script sends batches of 30 at a time for processing.
was (Author: mwk):
Nagios shows that both solr nodes had a peak of about 50 open connections, 150 in TIME_WAIT and maybe a dozen in CLOSE_WAIT. Once the problem manifests, I see around 50 open, and 100 stuck in CLOSE_WAIT. Both nodes look nearly identical.
The host doing the insert/updates shows a peak of around 100 open connections and around 9000 in TIME_WAIT. Once the problem manifests all those TIME_WAIT connections clear. During one test, I gave tomcat a thread pool of 4096 and saw those eaten up as well.
On a successful load of all 350k documents, I see a similar pattern, perhaps a few less connections in TIME_WAIT.
> All threads become blocked resulting in hang when bulk adding
> -------------------------------------------------------------
>
> Key: SOLR-4696
> URL: https://issues.apache.org/jira/browse/SOLR-4696
> Project: Solr
> Issue Type: Bug
> Components: SolrCloud
> Affects Versions: 4.1, 4.2, 4.2.1
> Environment: Ubuntu 12.04.2 LTS 3.5.0-27-generic
> Java HotSpot(TM) 64-Bit Server VM (build 23.7-b01, mixed mode)
> KVM, 4xCPU, 5GB RAM, 4GB heap.
> 4 cores, 2 shards, 2 nodes, tomcat7
> Reporter: matt knecht
> Labels: hang
> Attachments: screenshot-1.jpg, solrconfig.xml, solr.jstack.1, solr.jstack.2
>
>
> During a bulk load after about 150,000 documents load, thread usage spikes, solr no longer processes any documents. Any additional documents added result in a new thread until the pool is exhausted.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org