You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Liu, Chang" <Ch...@disney.com> on 2010/05/22 02:05:31 UTC
questions on CommonsHttpSolrServer and MultiThreadedHttpConnectionManager
Hi All,
I have a question about using CommonsHttpSolrServer with
MultiThreadedHttpConnectionManager. We have a single
CommonsHttpSolrServer instance that handles all the solr requests. Once
we deployed our application to linux servers, we kept getting too many
files opened exception. After did some internet researches, we found
that use CommonsHttpSolrServer instead of SolrServer should the solution
that we are looking for. I was also hoping to get the overhead down. For
some queries that we are running, Actual qtime is 2 to 3 ms, but the the
call SOLRSERVER.query(query, SolrRequest.METHOD.POST) took more than 80
ms if not more. After we updated our application, we didn't see much
improvement on overhead issues. I am wondering if the changes we put in
place is functional at all. Any suggests/advices/tutorials will be
highly appreciated.
Here is the snippet of our codes that does initialization and query. If
we keep the code as it is, is the multithreadconnectionmanager active at
all or we need to call CommonsHttpSolrServer.request() to take
advantages of the connection pooling
HttpClient client = new HttpClient();
MultiThreadedHttpConnectionManager mgr = new
MultiThreadedHttpConnectionManager();
client.setHttpConnectionManager(mgr);
SOLRSERVER = new
CommonsHttpSolrServer(Constants.SOLRSERVERURL, client);
SOLRSERVER.setDefaultMaxConnectionsPerHost(10000);
SOLRSERVER.setMaxTotalConnections(10000);
SOLRSERVER.setMaxRetries(1);
SOLRSERVER.setSoTimeout(1000);
SolrQuery query = new SolrQuery(queryString);
QueryResponse response = SOLRSERVER.query(query,
SolrRequest.METHOD.POST);
Thanks
Chang
Re: questions on CommonsHttpSolrServer and MultiThreadedHttpConnectionManager
Posted by Simon Willnauer <si...@googlemail.com>.
Hey there,
check out Ryans link for explanation of your TooManyOpenFiles problem.
For the query time, make sure the Nagle alg is disabled - should be by
default though.
MultiThreadedHttpConnectionManager mgr = new
MultiThreadedHttpConnectionManager();
mgr.getParams().setTcpNoDelay(true)
You should also make sure you keep your HttpClient instance around and
share it between requests that will most likely improve your
performance. Don't know if you do it but I can not tell from the
snippet you send around.
I also think your default values for setDefaultMaxConnectionsPerHost
and setMaxTotalConnections is kind of high. I guess you can reduce
them dramatically to about 100 or 1000
simon
On Sat, May 22, 2010 at 2:05 AM, Liu, Chang <Ch...@disney.com> wrote:
> Hi All,
>
>
>
> I have a question about using CommonsHttpSolrServer with
> MultiThreadedHttpConnectionManager. We have a single CommonsHttpSolrServer
> instance that handles all the solr requests. Once we deployed our
> application to linux servers, we kept getting too many files opened
> exception. After did some internet researches, we found that use
> CommonsHttpSolrServer instead of SolrServer should the solution that we are
> looking for. I was also hoping to get the overhead down. For some queries
> that we are running, Actual qtime is 2 to 3 ms, but the the call
> SOLRSERVER.query(query, SolrRequest.METHOD.POST) took more than 80 ms if not
> more. After we updated our application, we didn’t see much improvement on
> overhead issues. I am wondering if the changes we put in place is functional
> at all. Any suggests/advices/tutorials will be highly appreciated.
>
>
>
> Here is the snippet of our codes that does initialization and query. If we
> keep the code as it is, is the multithreadconnectionmanager active at all or
> we need to call CommonsHttpSolrServer.request() to take advantages of the
> connection pooling
>
>
>
>
>
> HttpClient client = new HttpClient();
>
> MultiThreadedHttpConnectionManager mgr = new
> MultiThreadedHttpConnectionManager();
>
> client.setHttpConnectionManager(mgr);
>
> SOLRSERVER = new
> CommonsHttpSolrServer(Constants.SOLRSERVERURL, client);
>
> SOLRSERVER.setDefaultMaxConnectionsPerHost(10000);
>
> SOLRSERVER.setMaxTotalConnections(10000);
>
> SOLRSERVER.setMaxRetries(1);
>
> SOLRSERVER.setSoTimeout(1000);
>
>
>
>
>
>
>
> SolrQuery query = new SolrQuery(queryString);
>
> QueryResponse response = SOLRSERVER.query(query,
> SolrRequest.METHOD.POST);
>
>
>
>
>
>
>
> Thanks
>
> Chang
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org
Re: questions on CommonsHttpSolrServer and MultiThreadedHttpConnectionManager
Posted by Ryan McKinley <ry...@gmail.com>.
On Fri, May 21, 2010 at 8:05 PM, Liu, Chang <Ch...@disney.com> wrote:
> Hi All,
>
> Once we deployed our
> application to linux servers, we kept getting too many files opened
> exception.
More likely it has to do with the number of open files...
check:
http://wiki.apache.org/lucene-java/LuceneFAQ#Why_am_I_getting_an_IOException_that_says_.22Too_many_open_files.22.3F
FYI, this type of question may be better posted to solr-user@lucene.apache.org
ryan
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org