You are viewing a plain text version of this content. The canonical link for it is here.
Posted to solr-user@lucene.apache.org by Ian Holsman <li...@holsman.net> on 2008/11/14 01:49:37 UTC
solrj and CLOSE_WAIT's
Hi guys.
I'm running a little upload project that uploads documents into a solr
index. there is also a 2nd thread that runs a deleteby query and a
optimize every once and a while.
in an effort to reduce the probably of things being held onto I've made
everything local, but it is still collecting CLOSE_WAITs and FIN_WAIT2's
on the server side until it eventually runs out of file handles in a day
or two.
the following are the code snippets being used to call solr.
protected void doArchiveSolr() throws IOException, SolrServerException {
Calendar rightNow = Calendar.getInstance();
rightNow.add(Calendar.DATE, 31 * -1);
DateFormat f = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSS'Z'");
java.util.Date d = rightNow.getTime();
String s = "publish_date:[1976-03-06T23:59:59.999Z/YEAR TO " +
f.format(d) + "]";
logger.info("Archiver:" + s);
CommonsHttpSolrServer solrServer;
solrServer = new CommonsHttpSolrServer(solrURL);
solrServer.deleteByQuery(s);
solrServer.commit();
}
and this runs every X minutes.
it also has other local parts like
{
CommonsHttpSolrServer solrServer;
solrServer = new CommonsHttpSolrServer(solrURL);
solrServer.optimize();
}
and
{
CommonsHttpSolrServer solrServer;
UpdateResponse r;
solrServer = new CommonsHttpSolrServer(solrUrl);
solrServer.setSoTimeout(120000); // socket read
timeout 2minutes
solrServer.setConnectionTimeout(100);
solrServer.setDefaultMaxConnectionsPerHost(100);
solrServer.setMaxTotalConnections(100);
solrServer.setFollowRedirects(false); //
defaults to false
solrServer.setAllowCompression(false);
r = solrServer.add(docs);
r = solrServer.commit();
docs.clear();
}
Re: solrj and CLOSE_WAIT's
Posted by Ian Holsman <li...@holsman.net>.
Ryan McKinley wrote:
> not sure if it is something we can do better or part of HttpClient...
>
> From:
> http://www.nabble.com/CLOSE_WAIT-td19959428.html
>
> it seems to suggest you may want to call:
> con.closeIdleConnections(0L);
>
> But if you are creating a new MultiThreadedHttpConnectionManager for
> each request, is seems odd you would have to explicitly close the
> connection for each request.
>
> What happens if you try using a SimpleHttpConnectionManager rather
> then a MultiThreadedHttpConnectionManager? You can explicitly pass in:
>
I was thinking the same thing when i saw the other constructor.
I've modified the code to call the 'simple' version and will let it run
for an hour or three to make sure it works and doesn't exhibit the
behavior, so far it looks good and there are no CLOSE_WAITs (or
FIN_WAIT2's) showing up for longer than a couple of seconds. (according
to netstat -tn)
I'd petition we go back to the 'stupid' version by default that just
does what it is supposed to do, and leave the other one for 'experts'. I
can't even see how to tell the multi-threaded version to close itself
nicely ;(
> to:
> public CommonsHttpSolrServer(URL baseURL, HttpClient client,
> ResponseParser parser, boolean useMultiPartPost) {
>
> if that fixes things, it is a bit disturbing, but something we should
> look into.
>
> ryan
>
>
Re: solrj and CLOSE_WAIT's
Posted by Ian Holsman <li...@holsman.net>.
Ryan McKinley wrote:
> not sure if it is something we can do better or part of HttpClient...
>
> From:
> http://www.nabble.com/CLOSE_WAIT-td19959428.html
>
> it seems to suggest you may want to call:
> con.closeIdleConnections(0L);
>
> But if you are creating a new MultiThreadedHttpConnectionManager for
> each request, is seems odd you would have to explicitly close the
> connection for each request.
>
> What happens if you try using a SimpleHttpConnectionManager rather
> then a MultiThreadedHttpConnectionManager? You can explicitly pass in:
> new HttpClient( new SimpleHttpConnectionManager() )
> to:
> public CommonsHttpSolrServer(URL baseURL, HttpClient client,
> ResponseParser parser, boolean useMultiPartPost) {
>
> if that fixes things, it is a bit disturbing, but something we should
> look into.
>
> ryan
>
>
Hi Ryan.
the problem is with the HttpClient class.
the SimpleHttpConnectionManager by default hangs onto a connection "just
in case".
It needs to be called like:
CommonsHttpSolrServer solrServer;
SimpleHttpConnectionManager cm = new
SimpleHttpConnectionManager(true);
HttpClient httpClient = new HttpClient(cm);
solrServer = new CommonsHttpSolrServer(solrURL, httpClient);
solrServer.optimize();
cm.shutdown();
I'm not sure the 'shutdown' method is required, but the code now has
*NO* CLOSE_WAITS after running for 3-4 hours.
I'll stick up a jira with makes 'SimpleHttpConnectionManager(true)' the
default.
regards
Ian
Re: solrj and CLOSE_WAIT's
Posted by Ryan McKinley <ry...@gmail.com>.
not sure if it is something we can do better or part of HttpClient...
From:
http://www.nabble.com/CLOSE_WAIT-td19959428.html
it seems to suggest you may want to call:
con.closeIdleConnections(0L);
But if you are creating a new MultiThreadedHttpConnectionManager for
each request, is seems odd you would have to explicitly close the
connection for each request.
What happens if you try using a SimpleHttpConnectionManager rather
then a MultiThreadedHttpConnectionManager? You can explicitly pass in:
new HttpClient( new SimpleHttpConnectionManager() )
to:
public CommonsHttpSolrServer(URL baseURL, HttpClient client,
ResponseParser parser, boolean useMultiPartPost) {
if that fixes things, it is a bit disturbing, but something we should
look into.
ryan
On Nov 13, 2008, at 7:49 PM, Ian Holsman wrote:
> Hi guys.
>
> I'm running a little upload project that uploads documents into a
> solr index. there is also a 2nd thread that runs a deleteby query
> and a optimize every once and a while.
>
> in an effort to reduce the probably of things being held onto I've
> made everything local, but it is still collecting CLOSE_WAITs and
> FIN_WAIT2's on the server side until it eventually runs out of file
> handles in a day or two.
>
> the following are the code snippets being used to call solr.
>
> protected void doArchiveSolr() throws IOException,
> SolrServerException {
> Calendar rightNow = Calendar.getInstance();
> rightNow.add(Calendar.DATE, 31 * -1);
> DateFormat f = new SimpleDateFormat("yyyy-MM-
> dd'T'HH:mm:ss.SSS'Z'");
> java.util.Date d = rightNow.getTime();
>
> String s = "publish_date:[1976-03-06T23:59:59.999Z/YEAR TO " +
> f.format(d) + "]";
> logger.info("Archiver:" + s);
> CommonsHttpSolrServer solrServer;
> solrServer = new CommonsHttpSolrServer(solrURL);
> solrServer.deleteByQuery(s);
> solrServer.commit();
> }
>
> and this runs every X minutes.
> it also has other local parts like
> {
> CommonsHttpSolrServer solrServer;
> solrServer = new CommonsHttpSolrServer(solrURL);
> solrServer.optimize();
> }
>
>
> and
> {
> CommonsHttpSolrServer solrServer;
> UpdateResponse r;
>
> solrServer = new CommonsHttpSolrServer(solrUrl);
> solrServer.setSoTimeout(120000); // socket
> read timeout 2minutes
> solrServer.setConnectionTimeout(100);
> solrServer.setDefaultMaxConnectionsPerHost(100);
> solrServer.setMaxTotalConnections(100);
> solrServer.setFollowRedirects(false); //
> defaults to false
> solrServer.setAllowCompression(false);
>
> r = solrServer.add(docs);
>
> r = solrServer.commit();
> docs.clear();
>
> }
>
>