You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Joan Balagueró <jo...@grupoventus.com> on 2018/11/07 18:30:07 UTC
RV: Migration from Async 4.1.3 to HttpClient 5
Hello Oleg,
We are finishing the migration and have the last questions:
1. If a connection is kept-alive for 30s at second 0, and after 10s is reused, this connection will die at second 30 or will survive until second 40?
2. Regarding the RetryHandler, below the method inherited from http 4.5 and modified to work with http5:
public boolean retryRequest(HttpRequest request, IOException exception, int executionCount, HttpContext context)
{
// Don't retry if max retries are reached.
if (executionCount > this.maxExecutionCount) return false;
// Don't retry if any of these exceptions occur.
if (exception instanceof InterruptedIOException || exception instanceof UnknownHostException || exception instanceof ConnectException || exception instanceof SSLException) return false;
// Retry of if this request is considered 'idempotent'.
return (!(request instanceof HttpEntityEnclosingRequest));
}
I understand the first two conditions are still ok (not sure if we have to add new exceptions on that list) but regarding the last condition,what would the equivalent condition be in Http5?
3. We have increased the response time of our backend (ip ended with '182') in order to exhaust the strict/lax pool. When this happens the pool starts to throw a DeadlineTimeoutException. At this moment the number of sockets in TIME_WAIT increases a lot until making the server unresponsive (probably exhausting the local ports):
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
99
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
101
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
98
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
25876
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
61507
[root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" | wc -l
97615
Is this the right behaviour? If Http5 cannot create new connections, so no new sockets are opened, why does the number of sockets in TIME_WAIT raise at those values?
Thanks,
Joan.
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: RV: Migration from Async 4.1.3 to HttpClient 5
Posted by Gary Gregory <ga...@gmail.com>.
On Fri, Nov 9, 2018 at 7:01 AM Oleg Kalnichevski <ol...@apache.org> wrote:
> On Fri, 2018-11-09 at 13:39 +0100, Joan Balagueró wrote:
> > Thanks Oleg. One more thing about the max connections with lax/strict
> > pool. Our code to modify the number of max connections is:
> >
> > public void setMaxConnections(int maxConnections)
> > {
> > this.phccm.setMaxTotal(maxConnections);
> > this.phccm.setDefaultMaxPerRoute(maxConnections);
> > }
> >
> > If I modify (on the fly) the max connections in a strict pool it
> > works. For example I set a very low value and I start to receive
> > DeadlineTimeoutException, when I set a higher value the error
> > disappears. If I print the pool.getMaxTotal() I get the right value.
> >
> > But this does not work with a lax pool. I set up a lax pool with max
> > connections = 1, and no DeadlineTimeoutException is thrown (with the
> > same load). When I print the maxTotal and defaultMaxPerRoute I get 0
> > and 1 (instead of 1 and 1).
> >
> > Is this a bug or am I missing something?
> >
>
> Max total is not enforced by the lax pool, only max per route.
>
Hi,
Can you make sure this is Javadoc'd in the right places if it is not
already?
Gary
>
> Oleg
>
>
> > Thanks,
> >
> > Joan.
> >
> >
> > -----Mensaje original-----
> > De: Oleg Kalnichevski [mailto:olegk@apache.org]
> > Enviado el: jueves, 8 de noviembre de 2018 11:04
> > Para: HttpClient User Discussion
> > Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5
> >
> > On Wed, 2018-11-07 at 19:30 +0100, Joan Balagueró wrote:
> > > Hello Oleg,
> > >
> > > We are finishing the migration and have the last questions:
> > >
> > > 1. If a connection is kept-alive for 30s at second 0, and after 10s
> > > is
> > > reused, this connection will die at second 30 or will survive
> > > until
> > > second 40?
> >
> > Keep-alive value is always relative to the last connection release.
> > If you want to limit the absolute connection life time please use set
> > a finite TTL (total time to live) value.
> >
> > >
> > > 2. Regarding the RetryHandler, below the method inherited from http
> > > 4.5 and modified to work with http5:
> > >
> >
> > I would recommend using DefaultHttpRequestRetryHandler shipped with
> > the library unless you have some application specific requirements.
> >
> >
>
> https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
> >
> > > public boolean retryRequest(HttpRequest request, IOException
> > > exception, int executionCount, HttpContext context) {
> > > // Don't retry if max retries are reached.
> > > if (executionCount > this.maxExecutionCount) return false;
> > >
> > > // Don't retry if any of these exceptions occur.
> > > if (exception instanceof InterruptedIOException || exception
> > > instanceof UnknownHostException || exception instanceof
> > > ConnectException || exception instanceof SSLException) return
> > > false;
> > >
> > > // Retry of if this request is considered 'idempotent'.
> > > return (!(request instanceof HttpEntityEnclosingRequest)); }
> > >
> > > I understand the first two conditions are still ok (not sure if we
> > > have to add new exceptions on that list) but regarding the last
> > > condition,what would the equivalent condition be in Http5?
> > >
> >
> > I would suggest the following:
> >
> >
>
> https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
> >
> >
> > >
> > > 3. We have increased the response time of our backend (ip ended
> > > with
> > > '182') in order to exhaust the strict/lax pool. When this happens
> > > the
> > > pool starts to throw a DeadlineTimeoutException. At this moment
> > > the
> > > number of sockets in TIME_WAIT increases a lot until making the
> > > server
> > > unresponsive (probably exhausting the local ports):
> > >
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > > |
> > > wc -l
> > > 99
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > > |
> > > wc -l
> > > 101
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > > |
> > > wc -l
> > > 98
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > > |
> > > wc -l
> > > 25876
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > > |
> > > wc -l
> > > 61507
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > > |
> > > wc -l
> > > 97615
> > >
> > > Is this the right behaviour? If Http5 cannot create new
> > > connections,
> > > so no new sockets are opened, why does the number of sockets in
> > > TIME_WAIT raise at those values?
> > >
> >
> > I believe it is. There is pretty good explanation of what the
> > TIME_WAIT state represents in our old wiki:
> >
> >
>
> https://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagementQuestions
> >
> > Oleg
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
Re: RV: Migration from Async 4.1.3 to HttpClient 5
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2018-11-09 at 15:19 +0100, Joan Balagueró wrote:
> Ok, so if I have a defaultMaxPerRoute = 1, and all requests I'm
> sending are using plain http to the same ip (without proxy) and only
> using 4 different ports (8080, 8081, 8082, 8083), than this means I
> have 1 max connection for ip:8080, 1 for ip:8081, 1 for ip:80802 and
> 1 for ip:80803?
> Joan.
>
Correct.
Oleg
> -----Mensaje original-----
> De: Oleg Kalnichevski [mailto:olegk@apache.org]
> Enviado el: viernes, 9 de noviembre de 2018 15:01
> Para: HttpClient User Discussion
> Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5
>
> On Fri, 2018-11-09 at 13:39 +0100, Joan Balagueró wrote:
> > Thanks Oleg. One more thing about the max connections with
> > lax/strict
> > pool. Our code to modify the number of max connections is:
> >
> > public void setMaxConnections(int maxConnections) {
> > this.phccm.setMaxTotal(maxConnections);
> > this.phccm.setDefaultMaxPerRoute(maxConnections);
> > }
> >
> > If I modify (on the fly) the max connections in a strict pool it
> > works. For example I set a very low value and I start to receive
> > DeadlineTimeoutException, when I set a higher value the error
> > disappears. If I print the pool.getMaxTotal() I get the right
> > value.
> >
> > But this does not work with a lax pool. I set up a lax pool with
> > max
> > connections = 1, and no DeadlineTimeoutException is thrown (with
> > the
> > same load). When I print the maxTotal and defaultMaxPerRoute I get
> > 0
> > and 1 (instead of 1 and 1).
> >
> > Is this a bug or am I missing something?
> >
>
> Max total is not enforced by the lax pool, only max per route.
>
> Oleg
>
>
> > Thanks,
> >
> > Joan.
> >
> >
> > -----Mensaje original-----
> > De: Oleg Kalnichevski [mailto:olegk@apache.org] Enviado el: jueves,
> > 8
> > de noviembre de 2018 11:04
> > Para: HttpClient User Discussion
> > Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5
> >
> > On Wed, 2018-11-07 at 19:30 +0100, Joan Balagueró wrote:
> > > Hello Oleg,
> > >
> > > We are finishing the migration and have the last questions:
> > >
> > > 1. If a connection is kept-alive for 30s at second 0, and after
> > > 10s
> > > is reused, this connection will die at second 30 or will survive
> > > until second 40?
> >
> > Keep-alive value is always relative to the last connection release.
> > If you want to limit the absolute connection life time please use
> > set
> > a finite TTL (total time to live) value.
> >
> > >
> > > 2. Regarding the RetryHandler, below the method inherited from
> > > http
> > > 4.5 and modified to work with http5:
> > >
> >
> > I would recommend using DefaultHttpRequestRetryHandler shipped
> > with
> > the library unless you have some application specific requirements.
> >
> >
>
>
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
> >
> > > public boolean retryRequest(HttpRequest request, IOException
> > > exception, int executionCount, HttpContext context) {
> > > // Don't retry if max retries are reached.
> > > if (executionCount > this.maxExecutionCount) return false;
> > >
> > > // Don't retry if any of these exceptions occur.
> > > if (exception instanceof InterruptedIOException || exception
> > > instanceof UnknownHostException || exception instanceof
> > > ConnectException || exception instanceof SSLException) return
> > > false;
> > >
> > > // Retry of if this request is considered 'idempotent'.
> > > return (!(request instanceof HttpEntityEnclosingRequest)); }
> > >
> > > I understand the first two conditions are still ok (not sure if
> > > we
> > > have to add new exceptions on that list) but regarding the last
> > > condition,what would the equivalent condition be in Http5?
> > >
> >
> > I would suggest the following:
> >
> >
>
>
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
> >
> >
> > >
> > > 3. We have increased the response time of our backend (ip ended
> > > with
> > > '182') in order to exhaust the strict/lax pool. When this
> > > happens
> > > the pool starts to throw a DeadlineTimeoutException. At this
> > > moment
> > > the number of sockets in TIME_WAIT increases a lot until making
> > > the
> > > server unresponsive (probably exhausting the local ports):
> > >
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep
> > > "179.182"
> > > >
> > >
> > > wc -l
> > > 99
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep
> > > "179.182"
> > > >
> > >
> > > wc -l
> > > 101
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep
> > > "179.182"
> > > >
> > >
> > > wc -l
> > > 98
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep
> > > "179.182"
> > > >
> > >
> > > wc -l
> > > 25876
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep
> > > "179.182"
> > > >
> > >
> > > wc -l
> > > 61507
> > > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep
> > > "179.182"
> > > >
> > >
> > > wc -l
> > > 97615
> > >
> > > Is this the right behaviour? If Http5 cannot create new
> > > connections,
> > > so no new sockets are opened, why does the number of sockets in
> > > TIME_WAIT raise at those values?
> > >
> >
> > I believe it is. There is pretty good explanation of what the
> > TIME_WAIT state represents in our old wiki:
> >
> >
>
>
https://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagementQuestions
> >
> > Oleg
> >
> >
> >
> > -----------------------------------------------------------------
> > ----
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail:
> > httpclient-users-help@hc.apache.org
> >
> >
> >
> >
> > -----------------------------------------------------------------
> > ----
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail:
> > httpclient-users-help@hc.apache.org
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: RV: Migration from Async 4.1.3 to HttpClient 5
Posted by Joan Balagueró <jo...@grupoventus.com>.
Ok, so if I have a defaultMaxPerRoute = 1, and all requests I'm sending are using plain http to the same ip (without proxy) and only using 4 different ports (8080, 8081, 8082, 8083), than this means I have 1 max connection for ip:8080, 1 for ip:8081, 1 for ip:80802 and 1 for ip:80803?
Joan.
-----Mensaje original-----
De: Oleg Kalnichevski [mailto:olegk@apache.org]
Enviado el: viernes, 9 de noviembre de 2018 15:01
Para: HttpClient User Discussion
Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5
On Fri, 2018-11-09 at 13:39 +0100, Joan Balagueró wrote:
> Thanks Oleg. One more thing about the max connections with lax/strict
> pool. Our code to modify the number of max connections is:
>
> public void setMaxConnections(int maxConnections) {
> this.phccm.setMaxTotal(maxConnections);
> this.phccm.setDefaultMaxPerRoute(maxConnections);
> }
>
> If I modify (on the fly) the max connections in a strict pool it
> works. For example I set a very low value and I start to receive
> DeadlineTimeoutException, when I set a higher value the error
> disappears. If I print the pool.getMaxTotal() I get the right value.
>
> But this does not work with a lax pool. I set up a lax pool with max
> connections = 1, and no DeadlineTimeoutException is thrown (with the
> same load). When I print the maxTotal and defaultMaxPerRoute I get 0
> and 1 (instead of 1 and 1).
>
> Is this a bug or am I missing something?
>
Max total is not enforced by the lax pool, only max per route.
Oleg
> Thanks,
>
> Joan.
>
>
> -----Mensaje original-----
> De: Oleg Kalnichevski [mailto:olegk@apache.org] Enviado el: jueves, 8
> de noviembre de 2018 11:04
> Para: HttpClient User Discussion
> Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5
>
> On Wed, 2018-11-07 at 19:30 +0100, Joan Balagueró wrote:
> > Hello Oleg,
> >
> > We are finishing the migration and have the last questions:
> >
> > 1. If a connection is kept-alive for 30s at second 0, and after 10s
> > is reused, this connection will die at second 30 or will survive
> > until second 40?
>
> Keep-alive value is always relative to the last connection release.
> If you want to limit the absolute connection life time please use set
> a finite TTL (total time to live) value.
>
> >
> > 2. Regarding the RetryHandler, below the method inherited from http
> > 4.5 and modified to work with http5:
> >
>
> I would recommend using DefaultHttpRequestRetryHandler shipped with
> the library unless you have some application specific requirements.
>
>
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
>
> > public boolean retryRequest(HttpRequest request, IOException
> > exception, int executionCount, HttpContext context) {
> > // Don't retry if max retries are reached.
> > if (executionCount > this.maxExecutionCount) return false;
> >
> > // Don't retry if any of these exceptions occur.
> > if (exception instanceof InterruptedIOException || exception
> > instanceof UnknownHostException || exception instanceof
> > ConnectException || exception instanceof SSLException) return false;
> >
> > // Retry of if this request is considered 'idempotent'.
> > return (!(request instanceof HttpEntityEnclosingRequest)); }
> >
> > I understand the first two conditions are still ok (not sure if we
> > have to add new exceptions on that list) but regarding the last
> > condition,what would the equivalent condition be in Http5?
> >
>
> I would suggest the following:
>
>
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
>
>
> >
> > 3. We have increased the response time of our backend (ip ended with
> > '182') in order to exhaust the strict/lax pool. When this happens
> > the pool starts to throw a DeadlineTimeoutException. At this moment
> > the number of sockets in TIME_WAIT increases a lot until making the
> > server unresponsive (probably exhausting the local ports):
> >
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 99
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 101
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 98
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 25876
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 61507
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 97615
> >
> > Is this the right behaviour? If Http5 cannot create new connections,
> > so no new sockets are opened, why does the number of sockets in
> > TIME_WAIT raise at those values?
> >
>
> I believe it is. There is pretty good explanation of what the
> TIME_WAIT state represents in our old wiki:
>
>
https://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagementQuestions
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: RV: Migration from Async 4.1.3 to HttpClient 5
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Fri, 2018-11-09 at 13:39 +0100, Joan Balagueró wrote:
> Thanks Oleg. One more thing about the max connections with lax/strict
> pool. Our code to modify the number of max connections is:
>
> public void setMaxConnections(int maxConnections)
> {
> this.phccm.setMaxTotal(maxConnections);
> this.phccm.setDefaultMaxPerRoute(maxConnections);
> }
>
> If I modify (on the fly) the max connections in a strict pool it
> works. For example I set a very low value and I start to receive
> DeadlineTimeoutException, when I set a higher value the error
> disappears. If I print the pool.getMaxTotal() I get the right value.
>
> But this does not work with a lax pool. I set up a lax pool with max
> connections = 1, and no DeadlineTimeoutException is thrown (with the
> same load). When I print the maxTotal and defaultMaxPerRoute I get 0
> and 1 (instead of 1 and 1).
>
> Is this a bug or am I missing something?
>
Max total is not enforced by the lax pool, only max per route.
Oleg
> Thanks,
>
> Joan.
>
>
> -----Mensaje original-----
> De: Oleg Kalnichevski [mailto:olegk@apache.org]
> Enviado el: jueves, 8 de noviembre de 2018 11:04
> Para: HttpClient User Discussion
> Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5
>
> On Wed, 2018-11-07 at 19:30 +0100, Joan Balagueró wrote:
> > Hello Oleg,
> >
> > We are finishing the migration and have the last questions:
> >
> > 1. If a connection is kept-alive for 30s at second 0, and after 10s
> > is
> > reused, this connection will die at second 30 or will survive
> > until
> > second 40?
>
> Keep-alive value is always relative to the last connection release.
> If you want to limit the absolute connection life time please use set
> a finite TTL (total time to live) value.
>
> >
> > 2. Regarding the RetryHandler, below the method inherited from http
> > 4.5 and modified to work with http5:
> >
>
> I would recommend using DefaultHttpRequestRetryHandler shipped with
> the library unless you have some application specific requirements.
>
>
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
>
> > public boolean retryRequest(HttpRequest request, IOException
> > exception, int executionCount, HttpContext context) {
> > // Don't retry if max retries are reached.
> > if (executionCount > this.maxExecutionCount) return false;
> >
> > // Don't retry if any of these exceptions occur.
> > if (exception instanceof InterruptedIOException || exception
> > instanceof UnknownHostException || exception instanceof
> > ConnectException || exception instanceof SSLException) return
> > false;
> >
> > // Retry of if this request is considered 'idempotent'.
> > return (!(request instanceof HttpEntityEnclosingRequest)); }
> >
> > I understand the first two conditions are still ok (not sure if we
> > have to add new exceptions on that list) but regarding the last
> > condition,what would the equivalent condition be in Http5?
> >
>
> I would suggest the following:
>
>
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
>
>
> >
> > 3. We have increased the response time of our backend (ip ended
> > with
> > '182') in order to exhaust the strict/lax pool. When this happens
> > the
> > pool starts to throw a DeadlineTimeoutException. At this moment
> > the
> > number of sockets in TIME_WAIT increases a lot until making the
> > server
> > unresponsive (probably exhausting the local ports):
> >
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 99
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 101
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 98
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 25876
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 61507
> > [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182"
> > |
> > wc -l
> > 97615
> >
> > Is this the right behaviour? If Http5 cannot create new
> > connections,
> > so no new sockets are opened, why does the number of sockets in
> > TIME_WAIT raise at those values?
> >
>
> I believe it is. There is pretty good explanation of what the
> TIME_WAIT state represents in our old wiki:
>
>
https://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagementQuestions
>
> Oleg
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
RE: RV: Migration from Async 4.1.3 to HttpClient 5
Posted by Joan Balagueró <jo...@grupoventus.com>.
Thanks Oleg. One more thing about the max connections with lax/strict pool. Our code to modify the number of max connections is:
public void setMaxConnections(int maxConnections)
{
this.phccm.setMaxTotal(maxConnections);
this.phccm.setDefaultMaxPerRoute(maxConnections);
}
If I modify (on the fly) the max connections in a strict pool it works. For example I set a very low value and I start to receive DeadlineTimeoutException, when I set a higher value the error disappears. If I print the pool.getMaxTotal() I get the right value.
But this does not work with a lax pool. I set up a lax pool with max connections = 1, and no DeadlineTimeoutException is thrown (with the same load). When I print the maxTotal and defaultMaxPerRoute I get 0 and 1 (instead of 1 and 1).
Is this a bug or am I missing something?
Thanks,
Joan.
-----Mensaje original-----
De: Oleg Kalnichevski [mailto:olegk@apache.org]
Enviado el: jueves, 8 de noviembre de 2018 11:04
Para: HttpClient User Discussion
Asunto: Re: RV: Migration from Async 4.1.3 to HttpClient 5
On Wed, 2018-11-07 at 19:30 +0100, Joan Balagueró wrote:
> Hello Oleg,
>
> We are finishing the migration and have the last questions:
>
> 1. If a connection is kept-alive for 30s at second 0, and after 10s is
> reused, this connection will die at second 30 or will survive until
> second 40?
Keep-alive value is always relative to the last connection release. If you want to limit the absolute connection life time please use set a finite TTL (total time to live) value.
>
> 2. Regarding the RetryHandler, below the method inherited from http
> 4.5 and modified to work with http5:
>
I would recommend using DefaultHttpRequestRetryHandler shipped with the library unless you have some application specific requirements.
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
> public boolean retryRequest(HttpRequest request, IOException
> exception, int executionCount, HttpContext context) {
> // Don't retry if max retries are reached.
> if (executionCount > this.maxExecutionCount) return false;
>
> // Don't retry if any of these exceptions occur.
> if (exception instanceof InterruptedIOException || exception
> instanceof UnknownHostException || exception instanceof
> ConnectException || exception instanceof SSLException) return false;
>
> // Retry of if this request is considered 'idempotent'.
> return (!(request instanceof HttpEntityEnclosingRequest)); }
>
> I understand the first two conditions are still ok (not sure if we
> have to add new exceptions on that list) but regarding the last
> condition,what would the equivalent condition be in Http5?
>
I would suggest the following:
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
>
> 3. We have increased the response time of our backend (ip ended with
> '182') in order to exhaust the strict/lax pool. When this happens the
> pool starts to throw a DeadlineTimeoutException. At this moment the
> number of sockets in TIME_WAIT increases a lot until making the server
> unresponsive (probably exhausting the local ports):
>
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 99
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 101
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 98
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 25876
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 61507
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 97615
>
> Is this the right behaviour? If Http5 cannot create new connections,
> so no new sockets are opened, why does the number of sockets in
> TIME_WAIT raise at those values?
>
I believe it is. There is pretty good explanation of what the TIME_WAIT state represents in our old wiki:
https://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagementQuestions
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: RV: Migration from Async 4.1.3 to HttpClient 5
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2018-11-07 at 19:30 +0100, Joan Balagueró wrote:
> Hello Oleg,
>
> We are finishing the migration and have the last questions:
>
> 1. If a connection is kept-alive for 30s at second 0, and after 10s
> is reused, this connection will die at second 30 or will survive
> until second 40?
Keep-alive value is always relative to the last connection release. If
you want to limit the absolute connection life time please use set a
finite TTL (total time to live) value.
>
> 2. Regarding the RetryHandler, below the method inherited from http
> 4.5 and modified to work with http5:
>
I would recommend using DefaultHttpRequestRetryHandler shipped with the
library unless you have some application specific requirements.
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
> public boolean retryRequest(HttpRequest request, IOException
> exception, int executionCount, HttpContext context)
> {
> // Don't retry if max retries are reached.
> if (executionCount > this.maxExecutionCount) return false;
>
> // Don't retry if any of these exceptions occur.
> if (exception instanceof InterruptedIOException || exception
> instanceof UnknownHostException || exception instanceof
> ConnectException || exception instanceof SSLException) return false;
>
> // Retry of if this request is considered 'idempotent'.
> return (!(request instanceof HttpEntityEnclosingRequest));
> }
>
> I understand the first two conditions are still ok (not sure if we
> have to add new exceptions on that list) but regarding the last
> condition,what would the equivalent condition be in Http5?
>
I would suggest the following:
https://github.com/apache/httpcomponents-client/blob/master/httpclient5/src/main/java/org/apache/hc/client5/http/impl/DefaultHttpRequestRetryHandler.java#L160
>
> 3. We have increased the response time of our backend (ip ended with
> '182') in order to exhaust the strict/lax pool. When this happens the
> pool starts to throw a DeadlineTimeoutException. At this moment the
> number of sockets in TIME_WAIT increases a lot until making the
> server unresponsive (probably exhausting the local ports):
>
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 99
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 101
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 98
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 25876
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 61507
> [root@ns3103538 ~]# netstat -anp | grep TIME_WAIT | grep "179.182" |
> wc -l
> 97615
>
> Is this the right behaviour? If Http5 cannot create new connections,
> so no new sockets are opened, why does the number of sockets in
> TIME_WAIT raise at those values?
>
I believe it is. There is pretty good explanation of what the TIME_WAIT
state represents in our old wiki:
https://wiki.apache.org/HttpComponents/FrequentlyAskedConnectionManagementQuestions
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org