You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@lucene.apache.org by "Shawn Heisey (JIRA)" <ji...@apache.org> on 2015/10/14 20:36:07 UTC
[jira] [Comment Edited] (SOLR-6239) HttpSolrServer: connection
still allocated
[ https://issues.apache.org/jira/browse/SOLR-6239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14957250#comment-14957250 ]
Shawn Heisey edited comment on SOLR-6239 at 10/14/15 6:35 PM:
--------------------------------------------------------------
Current versions of Solr are using the 4.4.1 versions of the httpcomponents jars. Version 4.9 was released in June of 2014. SOLR-8040 describes my plans to upgrade to HC 4.5.1.
There is a potential hiccup -- although Solr is using some of the new HC object types, there are still a lot of objects and methods in use that were deprecated in HC 4.3. SOLR-5604 discusses that problem and the fact that a fix is not going to be trivial.
One thing you can do to bring Solr slightly closer to the modern era is to build your own HttpClient object, using HC-recommended methods, and use that when building Solr client objects. Here's code that I use to do this, where I share HttpClient objects among multiple HttpSolrClient objects:
{code}
/**
* Creates and initializes a new Core object.
*
* @param chain The name of the chain.
* @param shard The name of the shard.
* @param name The name of the core.
*/
public Core(String chain, String shard, String name)
{
_backgroundOptimizeUnderway.set(false);
/*
* If this is the first time an object of this type has been created,
* create the shared httpClient objects. One has increased connection
* properties, the other a very long socket timeout. Synchronized to
* ensure thread safety.
*/
synchronized (firstInstance)
{
if (firstInstance)
{
firstInstance = false;
/* Trying out a new way of creating clients */
RequestConfig rc = RequestConfig.custom()
.setConnectTimeout(15000)
.setSocketTimeout(Const.SOCKET_TIMEOUT)
.build();
httpClient = HttpClients.custom()
.setDefaultRequestConfig(rc)
.setMaxConnPerRoute(300)
.setMaxConnTotal(5000)
.disableAutomaticRetries()
.build();
RequestConfig orc = RequestConfig.custom()
.setConnectTimeout(15000)
.setSocketTimeout(Const.ONE_HOUR * 2)
.build();
optimizeHttpClient = HttpClients.custom()
.setDefaultRequestConfig(orc)
.disableAutomaticRetries()
.build();
}
}
_shard = shard;
_chain = chain;
_name = name;
_prefix = "shard." + chain + "." + shard + ".";
String serverHost = Static.cfg.get(_prefix + "host");
String serverPort = Static.cfg.get(_prefix + "port");
/*
* This try/catch structure does not throw any checked exceptions, or at
* least it didn't throw any when this comment was written. Leaving it
* here anyway just in case there might be any other kind of exception
* thrown.
*/
try
{
String serverBaseUrl = "http://" + serverHost + ":" + serverPort + "/solr/";
/*
* Check to see if we already have this host/port combination saved
* in the client maps. If we do, re-use them instead of creating new
* ones. If not, create new clients and put them in the list.
* Synchronized on one of the static Map objects to ensure thread
* safety.
*/
synchronized (clientMap)
{
HttpSolrClient tmpClient;
String hostPort = serverHost + serverPort;
if (clientMap.containsKey(hostPort))
{
_client = clientMap.get(hostPort);
}
else
{
tmpClient = new HttpSolrClient(serverBaseUrl, httpClient);
_client = tmpClient;
clientMap.put(hostPort, _client);
}
if (optimizeClientMap.containsKey(hostPort))
{
_optimizeClient = optimizeClientMap.get(hostPort);
}
else
{
tmpClient = new HttpSolrClient(serverBaseUrl, optimizeHttpClient);
_optimizeClient = tmpClient;
optimizeClientMap.put(hostPort, _optimizeClient);
}
}
}
catch (Exception e)
{
throw new RuntimeException(_prefix + _name + ": Failed to create client objects", e);
}
}
{code}
To help make sense of that code, here are thed important class field definitions:
{code}
/**
* A static boolean value indicating whether this is the first instance of
* this object. Also used for thread synchronization.
*/
private static Boolean firstInstance = true;
/**
* A static http client to use on Solr client objects.
*/
private static HttpClient httpClient = null;
/**
* A static http client to use on Solr client objects. This one is for doing
* optimizes, will use a much longer socket timeout.
*/
private static HttpClient optimizeHttpClient = null;
/**
* A static map of server-level Solr objects, so that instances living on
* the same server/port can share objects.
*/
private static final Map<String, SolrClient> clientMap = new HashMap<>();
/**
* A static map of server-level Solr objects for optimizes that have a
* larger socket timeout.
*/
private static final Map<String, SolrClient> optimizeClientMap = new HashMap<>();
/** The main SolrClient object. */
private SolrClient _client;
/** A SolrClient object using a longer socket timeout, for optimizes. */
private SolrClient _optimizeClient;
{code}
was (Author: elyograg):
Current versions of Solr are using the 4.4.1 versions of the httpcomponents jars. Version 4.9 was released in June of 2014. SOLR-8040 describes my plans to upgrade to HC 4.5.1.
There is a potential hiccup -- although Solr is using some of the new HC object types, there are still a lot of objects and methods in use that were deprecated in HC 4.2. SOLR-5604 discusses that problem and the fact that a fix is not going to be trivial.
One thing you can do to bring Solr slightly closer to the modern era is to build your own HttpClient object, using HC-recommended methods, and use that when building Solr client objects. Here's code that I use to do this, where I share HttpClient objects among multiple HttpSolrClient objects:
{code}
/**
* Creates and initializes a new Core object.
*
* @param chain The name of the chain.
* @param shard The name of the shard.
* @param name The name of the core.
*/
public Core(String chain, String shard, String name)
{
_backgroundOptimizeUnderway.set(false);
/*
* If this is the first time an object of this type has been created,
* create the shared httpClient objects. One has increased connection
* properties, the other a very long socket timeout. Synchronized to
* ensure thread safety.
*/
synchronized (firstInstance)
{
if (firstInstance)
{
firstInstance = false;
/* Trying out a new way of creating clients */
RequestConfig rc = RequestConfig.custom()
.setConnectTimeout(15000)
.setSocketTimeout(Const.SOCKET_TIMEOUT)
.build();
httpClient = HttpClients.custom()
.setDefaultRequestConfig(rc)
.setMaxConnPerRoute(300)
.setMaxConnTotal(5000)
.disableAutomaticRetries()
.build();
RequestConfig orc = RequestConfig.custom()
.setConnectTimeout(15000)
.setSocketTimeout(Const.ONE_HOUR * 2)
.build();
optimizeHttpClient = HttpClients.custom()
.setDefaultRequestConfig(orc)
.disableAutomaticRetries()
.build();
}
}
_shard = shard;
_chain = chain;
_name = name;
_prefix = "shard." + chain + "." + shard + ".";
String serverHost = Static.cfg.get(_prefix + "host");
String serverPort = Static.cfg.get(_prefix + "port");
/*
* This try/catch structure does not throw any checked exceptions, or at
* least it didn't throw any when this comment was written. Leaving it
* here anyway just in case there might be any other kind of exception
* thrown.
*/
try
{
String serverBaseUrl = "http://" + serverHost + ":" + serverPort + "/solr/";
/*
* Check to see if we already have this host/port combination saved
* in the client maps. If we do, re-use them instead of creating new
* ones. If not, create new clients and put them in the list.
* Synchronized on one of the static Map objects to ensure thread
* safety.
*/
synchronized (clientMap)
{
HttpSolrClient tmpClient;
String hostPort = serverHost + serverPort;
if (clientMap.containsKey(hostPort))
{
_client = clientMap.get(hostPort);
}
else
{
tmpClient = new HttpSolrClient(serverBaseUrl, httpClient);
_client = tmpClient;
clientMap.put(hostPort, _client);
}
if (optimizeClientMap.containsKey(hostPort))
{
_optimizeClient = optimizeClientMap.get(hostPort);
}
else
{
tmpClient = new HttpSolrClient(serverBaseUrl, optimizeHttpClient);
_optimizeClient = tmpClient;
optimizeClientMap.put(hostPort, _optimizeClient);
}
}
}
catch (Exception e)
{
throw new RuntimeException(_prefix + _name + ": Failed to create client objects", e);
}
}
{code}
To help make sense of that code, here are thed important class field definitions:
{code}
/**
* A static boolean value indicating whether this is the first instance of
* this object. Also used for thread synchronization.
*/
private static Boolean firstInstance = true;
/**
* A static http client to use on Solr client objects.
*/
private static HttpClient httpClient = null;
/**
* A static http client to use on Solr client objects. This one is for doing
* optimizes, will use a much longer socket timeout.
*/
private static HttpClient optimizeHttpClient = null;
/**
* A static map of server-level Solr objects, so that instances living on
* the same server/port can share objects.
*/
private static final Map<String, SolrClient> clientMap = new HashMap<>();
/**
* A static map of server-level Solr objects for optimizes that have a
* larger socket timeout.
*/
private static final Map<String, SolrClient> optimizeClientMap = new HashMap<>();
/** The main SolrClient object. */
private SolrClient _client;
/** A SolrClient object using a longer socket timeout, for optimizes. */
private SolrClient _optimizeClient;
{code}
> HttpSolrServer: connection still allocated
> ------------------------------------------
>
> Key: SOLR-6239
> URL: https://issues.apache.org/jira/browse/SOLR-6239
> Project: Solr
> Issue Type: Bug
> Components: clients - java
> Affects Versions: 4.9
> Reporter: Sergio Fernández
> Priority: Minor
>
> In scenarios where concurrency is aggressive, this exception could easily appear:
> {quote}
> org.apache.solr.client.solrj.impl.HttpSolrServer$RemoteSolrException: Invalid use of BasicClientConnManager: connection still allocated.
> Make sure to release the connection before allocating another one.
> at org.apache.solr.client.solrj.impl.HttpSolrServer.executeMethod(HttpSolrServer.java:554) ~[solr-solrj-4.9.0.jar:4.9.0 1604085 - rmuir - 2014-06-20 06:34:04]
> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:210) ~[solr-solrj-4.9.0.jar:4.9.0 1604085 - rmuir - 2014-06-20 06:34:04]
> at org.apache.solr.client.solrj.impl.HttpSolrServer.request(HttpSolrServer.java:206) ~[solr-solrj-4.9.0.jar:4.9.0 1604085 - rmuir - 2014-06-20 06:34:04]
> at org.apache.solr.client.solrj.request.AbstractUpdateRequest.process(AbstractUpdateRequest.java:124) ~[solr-solrj-4.9.0.jar:4.9.0 1604085 - rmuir - 2014-06-20 06:34:04]
> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:116) ~[solr-solrj-4.9.0.jar:4.9.0 1604085 - rmuir - 2014-06-20 06:34:04]
> at org.apache.solr.client.solrj.SolrServer.add(SolrServer.java:102) ~[solr-solrj-4.9.0.jar:4.9.0 1604085 - rmuir - 2014-06-20 06:34:04]
> {quote}
> I wonder if there is any solution for it?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org