You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Johannes Goll <jo...@gmail.com> on 2011/06/13 03:10:52 UTC
Re: Huge performance drop in distributed search w/ shards on the same server/container
Hi Fred,
we are having similar issues of scaling Solr 3.1 distributed searches on a
single box with 18 cores. We use the StatsComponent which seems to be mainly
CPU bound. Using distributed searches resulted in a 9 fold decrease in
response time. However, sporadically, Jetty 6.1.2X (shipped with Solr 3.1.)
sporadically throws Socket connect exceptions when executing distributed
searches. Our next step is to switch from from jetty to tomcat. Did you
find a solution for improving the CPU utilization and requests per second
for your system?
Johannes
2011/5/26 pravesh <su...@yahoo.com>
> Do you really require multi-shards? Single core/shard will do for even
> millions of documents and the search will be faster than searching on
> multi-shards.
>
> Consider multi-shard when you cannot scale-up on a single
> shard/machine(e.g,
> CPU,RAM etc. becomes major block).
>
> Also read through the SOLR distributed search wiki to check on all tuning
> up
> required at application server(Tomcat) end, like maxHTTP request settings.
> For a single request in a multi-shard setup internal HTTP requests are made
> through all queried shards, so, make sure you set this parameter higher.
>
> --
> View this message in context:
> http://lucene.472066.n3.nabble.com/Huge-performance-drop-in-distributed-search-w-shards-on-the-same-server-container-tp2938421p2988464.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>
--
Johannes Goll
211 Curry Ford Lane
Gaithersburg, Maryland 20878
Re: Huge performance drop in distributed search w/ shards on the same server/container
Posted by Johannes Goll <jo...@gmail.com>.
I increased the maximum POST size and headerBufferSize to 10MB ; lowThreads
to 50, maxThreads to 100000 and lowResourceMaxIdleTime=15000. We tried
tomcat 6 using the following Connnector settings :
<Connector port="8989"
protocol="HTTP/1.1"
redirectPort="8443"
URIEncoding="UTF-8"
maxThreads="10000"
maxKeepAliveRequests="200"
maxHttpHeaderSize="10485760"
/>
I am getting the same exception as for jetty
SEVERE: org.apache.solr.common.SolrException:
org.apache.solr.client.solrj.SolrServerException: java.net.SocketException:
Connection reset
This seem to point towards a Solr specific issue (solrj.SolrServerException
during individual shard searches). I monitored the CPU utilization
executing sequential distributed searches and noticed that in the beginning
all CPUs are getting used for a short period of time (multiple lines for
shard searches are shown in the log with isShard=true arguments), then all
CPU except one become idle and the request is being processed by this one
CPU for the longest period of time.
I also noticed in the logs that while most of the individual shard searches
(isShard=true) have low QTimes (5-10), a minority has extreme QTimes
(104402-105126). All shards are fairly similar in size and content (1.2 M
documents) and the StatsComponent is being used
[stats=true&stats.field=weight&stats.facet=library_id]. Here library_id
equals the shard/core name.
Is there an internal timeout for gathering shard results or other fixed
resource limitation ?
Johannes
2011/6/13 Yonik Seeley <yo...@lucidimagination.com>
> On Sun, Jun 12, 2011 at 9:10 PM, Johannes Goll <jo...@gmail.com>
> wrote:
> > However, sporadically, Jetty 6.1.2X (shipped with Solr 3.1.)
> > sporadically throws Socket connect exceptions when executing distributed
> > searches.
>
> Are you using the exact jetty.xml that shipped with the solr example
> server,
> or did you make any modifications?
>
> -Yonik
> http://www.lucidimagination.com
>
--
Johannes Goll
211 Curry Ford Lane
Gaithersburg, Maryland 20878
Re: Huge performance drop in distributed search w/ shards on the same server/container
Posted by Yonik Seeley <yo...@lucidimagination.com>.
On Sun, Jun 12, 2011 at 9:10 PM, Johannes Goll <jo...@gmail.com> wrote:
> However, sporadically, Jetty 6.1.2X (shipped with Solr 3.1.)
> sporadically throws Socket connect exceptions when executing distributed
> searches.
Are you using the exact jetty.xml that shipped with the solr example server,
or did you make any modifications?
-Yonik
http://www.lucidimagination.com