You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hc.apache.org by Andrea Fabris <ca...@email.it> on 2004/05/06 15:10:53 UTC
MultiThreadedConnectionManager based on commons pool
Hello Everybody!
I'm Andrea from italy, and i'm very interested about the jakarta httpclient.
I tried it and i find it very easy to use, but i found (i wrote about it
some time ago) a bug in the multithredconnectionmanger.
It seems that the opened connections are tested for closure only when
they ahve to be given back from the pool to the client. In this way,
after some time where the connections are not used, the connections are
still open but the are in CLOSE_WAIT state: they are shut down when a
client request for a new connection form the pool, but the pool finds
the old conenction in a "CLOSE_WAIT" state, so it shuts down the
conenction and creates a new one.
Now, int this manner there could be a wate of resources, if nobody uses
connections for a long time.
I have seen that something has been done (in cvs) about auto closing
idle connection, but i still think that the multithreaded connection
manager needs a reimplementation.
I think that could be a great idea to reimplement the manager using
commons pool as a base.
Simply, the manager has to trak a list of open pools based on
GenericObjectPool: every pool in the manager (one for each host) must be
created using a factory that manages the connection life-cycle in the
pool (in the common pool framework).
I attach the code i've written, so you can take a look at it.
Regards
Andrea Fabris
Errore Apertura DB
Re: MultiThreadedConnectionManager based on commons pool
Posted by Andrea Fabris <ca...@email.it>.
On 20/07/2004 0.13, Oleg Kalnichevski wrote:
> Andrea,
>
> The code you contributed has been committed to the 2.0 branch. Many
> thanks for this contribution.
>
> http://cvs.apache.org/viewcvs.cgi/jakarta-commons/httpclient/src/contrib/org/apache/commons/httpclient/contrib/pool/Attic/
>
> I have applied some minor corrections to the source code to make it a
> little more compatible with the overall coding and formating guidelines.
> There's still potential for improvement: some additional javadocs would
> be nice; comments and debug logs should ideally be translated from
> Italian to English. So, feel free to submit incremental patches ;-)
>
> If you would like the authorship attribution for this code, please
> consider signing the CLA
>
> http://www.apache.org/licenses/
> http://www.apache.org/licenses/cla.txt
>
> Cheers,
>
> Oleg
Hi Oleg!
I'll check the code u have fixed, and fix all the translation/javadoc
problems.
I'm sorry i didn't have the time to contribute more to the problem (also
i have not tested estensivley the new pool in 3.0... it seems it has the
same problem of 2.0 release, eg: the pool tests for the closure of the
socket only when a socket is reclaimed from the pool, so mantaining some
socket in CLOSE_WAIT status), but i was very busy at work.
I hope to contribute as soon as possible.
Cheers
Andrea
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org
Re: MultiThreadedConnectionManager based on commons pool
Posted by Oleg Kalnichevski <ol...@apache.org>.
Andrea,
The code you contributed has been committed to the 2.0 branch. Many
thanks for this contribution.
http://cvs.apache.org/viewcvs.cgi/jakarta-commons/httpclient/src/contrib/org/apache/commons/httpclient/contrib/pool/Attic/
I have applied some minor corrections to the source code to make it a
little more compatible with the overall coding and formating guidelines.
There's still potential for improvement: some additional javadocs would
be nice; comments and debug logs should ideally be translated from
Italian to English. So, feel free to submit incremental patches ;-)
If you would like the authorship attribution for this code, please
consider signing the CLA
http://www.apache.org/licenses/
http://www.apache.org/licenses/cla.txt
Cheers,
Oleg
On Thu, 2004-05-06 at 15:10, Andrea Fabris wrote:
> Hello Everybody!
> I'm Andrea from italy, and i'm very interested about the jakarta httpclient.
> I tried it and i find it very easy to use, but i found (i wrote about it
> some time ago) a bug in the multithredconnectionmanger.
> It seems that the opened connections are tested for closure only when
> they ahve to be given back from the pool to the client. In this way,
> after some time where the connections are not used, the connections are
> still open but the are in CLOSE_WAIT state: they are shut down when a
> client request for a new connection form the pool, but the pool finds
> the old conenction in a "CLOSE_WAIT" state, so it shuts down the
> conenction and creates a new one.
> Now, int this manner there could be a wate of resources, if nobody uses
> connections for a long time.
> I have seen that something has been done (in cvs) about auto closing
> idle connection, but i still think that the multithreaded connection
> manager needs a reimplementation.
> I think that could be a great idea to reimplement the manager using
> commons pool as a base.
> Simply, the manager has to trak a list of open pools based on
> GenericObjectPool: every pool in the manager (one for each host) must be
> created using a factory that manages the connection life-cycle in the
> pool (in the common pool framework).
> I attach the code i've written, so you can take a look at it.
>
> Regards
> Andrea Fabris
>
> Errore Apertura DB
>
> ______________________________________________________________________
> package org.apache.commons.httpclient.pool;
>
> import org.apache.commons.pool.PoolableObjectFactory;
> import org.apache.commons.httpclient.HttpConnection;
> import org.apache.commons.httpclient.HostConfiguration;
> import org.apache.commons.logging.*;
>
> /**
> * <p>Title: </p>
> * <p>Description: </p>
> * <p>Copyright: Copyright (c) 2004</p>
> * <p>Company: </p>
> * @author not attributable
> * @version 1.0
> */
>
> public class PoolableHttpConnectionFactory implements PoolableObjectFactory {
>
> private HostConfiguration connConf;
> private HttpPoolConnectionManager manager;
> private static Log log = LogFactory.getLog(PoolableHttpConnectionFactory.class);
>
> public PoolableHttpConnectionFactory(HostConfiguration hc, HttpPoolConnectionManager manager) {
> connConf = hc;
> this.manager = manager;
> }
>
> /**
> * Create a new connection using the HostConfiguration passed in the Factory
> * @return a new connection
> */
> public Object makeObject() {
> log.debug("<PoolableHttpConnectionFactory - makeObject> Inizio metodo");
> HttpConnection conn = new HttpConnection(connConf);
> conn.setHttpConnectionManager(manager);
> log.debug("<PoolableHttpConnectionFactory - makeObject> Fine metodo");
> return conn;
> }
>
> public void destroyObject(Object in) {
> log.debug("<PoolableHttpConnectionFactory - destroyObject> Inizio metodo");
> if (in != null) {
> log.debug("<PoolableHttpConnectionFactory - destroyObject> Connection not null");
> HttpConnection conn = (HttpConnection) in;
> if (conn.isOpen()) {
> log.debug("<PoolableHttpConnectionFactory - destroyObject> Closing connection");
> conn.close();
> }
> in = null;
> log.debug("<PoolableHttpConnectionFactory - destroyObject> Fine metodo");
> }
> }
>
> public boolean validateObject(Object in) {
> log.debug("<PoolableHttpConnectionFactory - validateObject> Inizio metodo");
> boolean result = false;
> if (in != null) {
> HttpConnection conn = (HttpConnection) in;
> if (conn.isOpen())
> log.debug("<PoolableHttpConnectionFactory - validateObject> Connection opened: valid connection");
> result = true;
> }
> log.debug("<PoolableHttpConnectionFactory - validateObject> Inizio metodo");
> return result;
> }
>
> public void activateObject(Object in) {
> log.debug("<PoolableHttpConnectionFactory - activateObject> No action");
> }
>
> public void passivateObject(Object in){
> log.debug("<PoolableHttpConnectionFactory - passivateObject> No action");
> }
> }
>
> ______________________________________________________________________
> package org.apache.commons.httpclient.pool;
>
> import org.apache.commons.pool.impl.GenericObjectPool;
>
> /**
> * <p>Title: </p>
> * <p>Description: </p>
> * <p>Copyright: Copyright (c) 2004</p>
> * <p>Company: </p>
> * @author not attributable
> * @version 1.0
> */
>
> public class HttpPoolConnectionManagerConfiguration {
> private final long DEFAULT_TIMEOUT = 2000;
> private final int DEFAULT_MAX_CONNECTIONS_PERHOST = 20;
> private final long DEFAULT_EVICTION_TIME = 1000;
> private final int DEFAULT_NUM_TEST_PEREVICTION = 3;
> protected GenericObjectPool.Config poolconfig;
>
> /**
> * Default constructor
> * It initializes the pool configuration with default values
> */
> public HttpPoolConnectionManagerConfiguration() {
> poolconfig = new GenericObjectPool.Config();
> //pool configuration
> poolconfig.maxActive = DEFAULT_MAX_CONNECTIONS_PERHOST;
> //eviction thread
> poolconfig.minEvictableIdleTimeMillis = DEFAULT_TIMEOUT;
> poolconfig.whenExhaustedAction = GenericObjectPool.WHEN_EXHAUSTED_BLOCK;
> poolconfig.timeBetweenEvictionRunsMillis = DEFAULT_EVICTION_TIME;
> poolconfig.numTestsPerEvictionRun = DEFAULT_NUM_TEST_PEREVICTION;
> poolconfig.testWhileIdle = true;
> }
>
> /**
> * Sets the maximum number of connections that could be idle in the pool
> * @param max max idle connection in the pool
> */
> public void setMaxIdle(int max) {
> poolconfig.maxIdle = max;
> }
>
> /**
> * Sets the maximum number of connections that could be idle in the pool
> * @param min min idle connection pool
> */
> public void setMinIdle(int min) {
> poolconfig.minIdle = min;
> }
>
> /**
> * Sets if the connection has to be tested for validity before being returned to the pool
> * @param test true if the connection has to be tested before being returned to the pool
> */
> public void setTestOnReturn(boolean test) {
> poolconfig.testOnReturn = test;
> }
>
> /**
> * Sets if the connection has to be tested for validity before being borrowed from the pool
> * @param test true if the connection has to be tested before being borrowed from the pool
> */
> public void setTestOnBorrow(boolean test) {
> poolconfig.testOnBorrow = test;
> }
>
> /**
> * Sets how many milliseconds the client can wait for a connection from the pool
> * @param wait maximum wait time
> */
> public void setMaxWait(long wait) {
> poolconfig.maxWait = wait;
> }
>
> /**
> * Set the maximum number of connection allowed in the pool
> * @param max max connection in the pool
> */
> public void setMaxActive(int max) {
> poolconfig.maxActive = max;
> }
>
> /**
> * Sets how many milliseconds an dle connection could remain in the pool before being closed
> * @param timeout maximum idle time
> */
> public void setPoolIdleTime(long timeout) {
> poolconfig.minEvictableIdleTimeMillis = timeout;
> }
>
> /**
> * Sets the time (in milliseconds) between two runs of the check thred in the pool
> * @param time
> */
> public void setPoolCheckTime(long time) {
> poolconfig.timeBetweenEvictionRunsMillis = time;
> }
>
> /**
> * Sets how meny tests have to be perfomrmed during the eviction run
> * @param test int
> */
> public void setTestPerCheck(int test) {
> poolconfig.numTestsPerEvictionRun = test;
> }
> }
>
> ______________________________________________________________________
> package org.apache.commons.httpclient.pool;
>
> import java.util.*;
>
> import org.apache.commons.httpclient.*;
> import org.apache.commons.logging.*;
> import org.apache.commons.pool.impl.*;
>
> /**
> * <p>Title: </p>
> * <p>Description: </p>
> * <p>Copyright: Copyright (c) 2004</p>
> * <p>Company: </p>
> * @author not attributable
> * @version 1.0
> */
>
> public class HttpPoolConnectionManager
> implements HttpConnectionManager {
>
> private HashMap poolsMap = new HashMap();
> private static Log log = LogFactory.getLog(HttpPoolConnectionManager.class);
> private static int DEFAULT_MAX_CONNECTIONS = 100;
>
> HttpPoolConnectionManagerConfiguration config;
> private int maxConnections;
>
> public HttpPoolConnectionManager() {
> config = new HttpPoolConnectionManagerConfiguration();
> maxConnections = DEFAULT_MAX_CONNECTIONS;
> }
>
> /**
> * HttpPoolConnectionManager
> *
> * @param configuration HttpPoolConnectionManager
> */
> public HttpPoolConnectionManager(HttpPoolConnectionManagerConfiguration
> configuration) {
> config = configuration;
> maxConnections = DEFAULT_MAX_CONNECTIONS;
> }
>
> public HttpConnection getConnection(HostConfiguration hostConfiguration) {
> try {
> return getConnection(hostConfiguration, -1);
> } catch (HttpException ex) {
> log.error("<HttpPoolConnectionManager - getConnection> " + ex.toString());
> return null;
> }
> }
>
> public HttpConnection getConnection(HostConfiguration hostConfiguration,
> long timeout)
> throws HttpException {
> log.debug("<HttpConnectionManager - getConnection> Inizio metodo");
> //Ottengo la "chiave"
> String key = hostConfiguration.getHost() + ":" + hostConfiguration.getPort();
> log.debug("<HttpConnectionManager - getConnection> Pool [" + key + "]");
> GenericObjectPool pool = null;
> synchronized (this) {
> pool = (GenericObjectPool) poolsMap.get(key);
> if (pool == null) {
> //creo pool e ottengo connessione
> log.debug("<HttpConnectionManager - getConnection> Pool da creare");
> config.setMaxWait(timeout);
> GenericObjectPool newpool = new GenericObjectPool(new
> PoolableHttpConnectionFactory(hostConfiguration, this),
> config.poolconfig);
> poolsMap.put(key, newpool);
> log.debug("<HttpConnectionManager - getConnection> Fine metodo");
> try {
> return (HttpConnection) newpool.borrowObject();
> } catch (Exception ex) {
> log.error("<HttpConnectionManager - getConnection> " + ex.toString());
> throw new HttpException(ex.getMessage());
> }
> }
> }
> //pool in cache
> log.debug("<HttpConnectionManager - getConnection> Pool gi usato");
> log.debug("<HttpConnectionManager - getConnection> Fine metodo");
> try {
> return (HttpConnection) (pool).borrowObject();
> } catch (Exception ex) {
> log.error("<HttpConnectionManager - getConnection> " + ex.toString());
> throw new HttpException(ex.getMessage());
> }
> }
>
> public void releaseConnection(HttpConnection conn) {
> log.debug("<HttpConnectionManager - releaseConnection> Inizio metodo");
> String key = conn.getHost() + ":" + conn.getPort();
> log.debug("<HttpConnectionManager - releaseConnection> Pool [" + key + "]");
> if (poolsMap.containsKey(key)) {
> GenericObjectPool pool = (GenericObjectPool) poolsMap.get(key);
> try {
> pool.returnObject(conn);
> } catch (Exception ex) {
> log.error("<HttpPoolConnectionManager - releaseConnection> " +
> ex.toString());
> }
> }
> log.debug("<HttpConnectionManager - releaseConnection> Fine metodo");
> }
>
> public void finalize() {
> log.debug("<HttpPoolConnectionManager - finalize> Inizio metodo");
> if (this.poolsMap != null) {
> Iterator keyiter = this.poolsMap.keySet().iterator();
> while (keyiter.hasNext()) {
> String key = (String) keyiter.next();
> log.debug("<HttpPoolConnectionManager - finalize> Destroying [" + key +
> "]");
> GenericObjectPool pool = (GenericObjectPool)this.poolsMap.get(key);
> try {
> pool.close();
> this.poolsMap.put(key, null);
> } catch (Exception ex) {
> log.error("<HttpPoolConnectionManager - finalize> " + ex.toString());
> }
> }
> this.poolsMap = null;
> }
> log.debug("<HttpPoolConnectionManager - finalize> Fine metodo");
> }
>
> public void shutdown() {
> log.debug("<HttpPoolConnectionManager - shutdown> Inizio metodo");
> Iterator keyiter = this.poolsMap.keySet().iterator();
> while (keyiter.hasNext()) {
> String key = (String) keyiter.next();
> log.debug("<HttpPoolConnectionManager - shutdown> Destroying [" + key +
> "]");
> GenericObjectPool pool = (GenericObjectPool)this.poolsMap.get(key);
> try {
> pool.close();
> this.poolsMap.put(key, null);
> } catch (Exception ex) {
> log.error("<HttpPoolConnectionManager - shutdown> " + ex.toString());
> }
> }
> this.poolsMap = null;
> log.debug("<HttpPoolConnectionManager - shutdown> Fine metodo");
> }
> }
>
>
> ______________________________________________________________________
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
> For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commons-httpclient-dev-unsubscribe@jakarta.apache.org
For additional commands, e-mail: commons-httpclient-dev-help@jakarta.apache.org