You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by Benson Margulies <bi...@gmail.com> on 2016/06/20 17:57:41 UTC
Telling HC to make new connections all the time
We use the async http components library to connect various components
in a system that is deployed, in some cases, on AWS. When it's
deployed on AWS, there are ELBs involved.
Like some other people, we've having various trauma dealing with the
very dynamic environment of AWS ELBs, which includes constant DNS
changes and some other issues. So far, we've been making our HC
configuration more and more complex. I'm wondering if we should back
off and make it simpler -- make a new connection for each request.
Obviously, this will be somewhat slower and more expensive, but at the
moment we might prefer a slower and more expensive approach that works
to one that occasionally gets lost.
I appreciate that even making new connections all the time won't
compensate for all possible DNS problems -- if Java itself caches a
stale name->address translation, making a new connection will fail
just as effectively as anything else.
Does anyone out there have advice to share (other than running away from ELB)?
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Telling HC to make new connections all the time
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Tue, 2016-06-21 at 08:28 -0400, Benson Margulies wrote:
> Oleg, If I set a TTL, do I also need an IdleConnectionMonitorThread (
> http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html#d5e652
> )? thanks, benson.
>
TTL setting only ensures connections cannot get re-used beyond their TTL
but they can still be kept alive in the pool until the pool manager
discards them either while trying to find a valid connection or while
evicting expired ones. One might still need IdleConnectionMonitorThread
to ensure eviction of expired connections when the connection manager
becomes idle after a period of heavy activity.
Oleg
>
> On Tue, Jun 21, 2016 at 4:41 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> > On Mon, 2016-06-20 at 13:57 -0400, Benson Margulies wrote:
> >> We use the async http components library to connect various components
> >> in a system that is deployed, in some cases, on AWS. When it's
> >> deployed on AWS, there are ELBs involved.
> >>
> >> Like some other people, we've having various trauma dealing with the
> >> very dynamic environment of AWS ELBs, which includes constant DNS
> >> changes and some other issues. So far, we've been making our HC
> >> configuration more and more complex. I'm wondering if we should back
> >> off and make it simpler -- make a new connection for each request.
> >> Obviously, this will be somewhat slower and more expensive, but at the
> >> moment we might prefer a slower and more expensive approach that works
> >> to one that occasionally gets lost.
> >>
> >> I appreciate that even making new connections all the time won't
> >> compensate for all possible DNS problems -- if Java itself caches a
> >> stale name->address translation, making a new connection will fail
> >> just as effectively as anything else.
> >>
> >> Does anyone out there have advice to share (other than running away from ELB)?
> >>
> >
> > Hi Benson
> >
> > I personally would consider disabling connection persistence extreme and
> > unnecessary. Consider setting the TTL (total time to live) of persistent
> > connections to some low value (say, 10 seconds or even less) instead.
> > This should improve distribution of connections across the cluster of
> > servers behind a load balancer.
> >
> > Hope it helps
> >
> > Oleg
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Telling HC to make new connections all the time
Posted by Benson Margulies <bi...@gmail.com>.
Oleg, If I set a TTL, do I also need an IdleConnectionMonitorThread (
http://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html#d5e652
)? thanks, benson.
On Tue, Jun 21, 2016 at 4:41 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Mon, 2016-06-20 at 13:57 -0400, Benson Margulies wrote:
>> We use the async http components library to connect various components
>> in a system that is deployed, in some cases, on AWS. When it's
>> deployed on AWS, there are ELBs involved.
>>
>> Like some other people, we've having various trauma dealing with the
>> very dynamic environment of AWS ELBs, which includes constant DNS
>> changes and some other issues. So far, we've been making our HC
>> configuration more and more complex. I'm wondering if we should back
>> off and make it simpler -- make a new connection for each request.
>> Obviously, this will be somewhat slower and more expensive, but at the
>> moment we might prefer a slower and more expensive approach that works
>> to one that occasionally gets lost.
>>
>> I appreciate that even making new connections all the time won't
>> compensate for all possible DNS problems -- if Java itself caches a
>> stale name->address translation, making a new connection will fail
>> just as effectively as anything else.
>>
>> Does anyone out there have advice to share (other than running away from ELB)?
>>
>
> Hi Benson
>
> I personally would consider disabling connection persistence extreme and
> unnecessary. Consider setting the TTL (total time to live) of persistent
> connections to some low value (say, 10 seconds or even less) instead.
> This should improve distribution of connections across the cluster of
> servers behind a load balancer.
>
> Hope it helps
>
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Telling HC to make new connections all the time
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2016-06-20 at 13:57 -0400, Benson Margulies wrote:
> We use the async http components library to connect various components
> in a system that is deployed, in some cases, on AWS. When it's
> deployed on AWS, there are ELBs involved.
>
> Like some other people, we've having various trauma dealing with the
> very dynamic environment of AWS ELBs, which includes constant DNS
> changes and some other issues. So far, we've been making our HC
> configuration more and more complex. I'm wondering if we should back
> off and make it simpler -- make a new connection for each request.
> Obviously, this will be somewhat slower and more expensive, but at the
> moment we might prefer a slower and more expensive approach that works
> to one that occasionally gets lost.
>
> I appreciate that even making new connections all the time won't
> compensate for all possible DNS problems -- if Java itself caches a
> stale name->address translation, making a new connection will fail
> just as effectively as anything else.
>
> Does anyone out there have advice to share (other than running away from ELB)?
>
Hi Benson
I personally would consider disabling connection persistence extreme and
unnecessary. Consider setting the TTL (total time to live) of persistent
connections to some low value (say, 10 seconds or even less) instead.
This should improve distribution of connections across the cluster of
servers behind a load balancer.
Hope it helps
Oleg
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Telling HC to make new connections all the time
Posted by Benson Margulies <bi...@gmail.com>.
On Wed, Jun 22, 2016 at 11:26 AM, Oleg Kalnichevski <ol...@apache.org> wrote:
> On Wed, 2016-06-22 at 07:07 -0400, Benson Margulies wrote:
>> On Wed, Jun 22, 2016 at 3:05 AM, Stefan Magnus Landrø
>> <st...@gmail.com> wrote:
>> > Hi Benson,
>> >
>> > Have you considered implementing a custom
>> > https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html
>> > that handles DNS changes in a better way than the default implementation?
>> > You could probably use http://www.xbill.org/dnsjava/ and lookup names
>> > directly with the amazon NS?
>>
>> Thanks for reminding me to ask about that. Does HC call the resolver
>> as part of deciding whether to keep a connection or not?
>
> No, it does not, but one could make it do so by using a custom
> ConnectionReuseStrategy.
The javadoc for ConnectionReuseStrategy implies that it is called at
the end of the request. Am I misreading it?
>
> Oleg
>
>> The problem
>> as we see it is this: the DNS mapping for an ELB changes, and a minute
>> later the connections to it become useless but don't necessarily get
>> properly closed with a FIN by the ELB. I've written some goofy code
>> that notices when the address resolution changes for any of the ELB's
>> we care about, and when it does it calls 'closeIdleConnections' with a
>> time of 0. On the other hand, I don't think we're currently having any
>> problems getting InetAddress.getByName to keep up with Amazon, but
>> your suggestion would be safer.
>>
>>
>> >
>> > Stefan
>> >
>> > 2016-06-20 19:57 GMT+02:00 Benson Margulies <bi...@gmail.com>:
>> >
>> >> We use the async http components library to connect various components
>> >> in a system that is deployed, in some cases, on AWS. When it's
>> >> deployed on AWS, there are ELBs involved.
>> >>
>> >> Like some other people, we've having various trauma dealing with the
>> >> very dynamic environment of AWS ELBs, which includes constant DNS
>> >> changes and some other issues. So far, we've been making our HC
>> >> configuration more and more complex. I'm wondering if we should back
>> >> off and make it simpler -- make a new connection for each request.
>> >> Obviously, this will be somewhat slower and more expensive, but at the
>> >> moment we might prefer a slower and more expensive approach that works
>> >> to one that occasionally gets lost.
>> >>
>> >> I appreciate that even making new connections all the time won't
>> >> compensate for all possible DNS problems -- if Java itself caches a
>> >> stale name->address translation, making a new connection will fail
>> >> just as effectively as anything else.
>> >>
>> >> Does anyone out there have advice to share (other than running away from
>> >> ELB)?
>> >>
>> >> ---------------------------------------------------------------------
>> >> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> >> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>> >>
>> >>
>> >
>> >
>> > --
>> > BEKK Open
>> > http://open.bekk.no
>> >
>> > TesTcl - a unit test framework for iRules
>> > http://testcl.com
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Telling HC to make new connections all the time
Posted by Oleg Kalnichevski <ol...@apache.org>.
On Wed, 2016-06-22 at 07:07 -0400, Benson Margulies wrote:
> On Wed, Jun 22, 2016 at 3:05 AM, Stefan Magnus Landr�
> <st...@gmail.com> wrote:
> > Hi Benson,
> >
> > Have you considered implementing a custom
> > https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html
> > that handles DNS changes in a better way than the default implementation?
> > You could probably use http://www.xbill.org/dnsjava/ and lookup names
> > directly with the amazon NS?
>
> Thanks for reminding me to ask about that. Does HC call the resolver
> as part of deciding whether to keep a connection or not?
No, it does not, but one could make it do so by using a custom
ConnectionReuseStrategy.
Oleg
> The problem
> as we see it is this: the DNS mapping for an ELB changes, and a minute
> later the connections to it become useless but don't necessarily get
> properly closed with a FIN by the ELB. I've written some goofy code
> that notices when the address resolution changes for any of the ELB's
> we care about, and when it does it calls 'closeIdleConnections' with a
> time of 0. On the other hand, I don't think we're currently having any
> problems getting InetAddress.getByName to keep up with Amazon, but
> your suggestion would be safer.
>
>
> >
> > Stefan
> >
> > 2016-06-20 19:57 GMT+02:00 Benson Margulies <bi...@gmail.com>:
> >
> >> We use the async http components library to connect various components
> >> in a system that is deployed, in some cases, on AWS. When it's
> >> deployed on AWS, there are ELBs involved.
> >>
> >> Like some other people, we've having various trauma dealing with the
> >> very dynamic environment of AWS ELBs, which includes constant DNS
> >> changes and some other issues. So far, we've been making our HC
> >> configuration more and more complex. I'm wondering if we should back
> >> off and make it simpler -- make a new connection for each request.
> >> Obviously, this will be somewhat slower and more expensive, but at the
> >> moment we might prefer a slower and more expensive approach that works
> >> to one that occasionally gets lost.
> >>
> >> I appreciate that even making new connections all the time won't
> >> compensate for all possible DNS problems -- if Java itself caches a
> >> stale name->address translation, making a new connection will fail
> >> just as effectively as anything else.
> >>
> >> Does anyone out there have advice to share (other than running away from
> >> ELB)?
> >>
> >> ---------------------------------------------------------------------
> >> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> >> For additional commands, e-mail: httpclient-users-help@hc.apache.org
> >>
> >>
> >
> >
> > --
> > BEKK Open
> > http://open.bekk.no
> >
> > TesTcl - a unit test framework for iRules
> > http://testcl.com
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Telling HC to make new connections all the time
Posted by Benson Margulies <bi...@gmail.com>.
On Wed, Jun 22, 2016 at 3:05 AM, Stefan Magnus Landrø
<st...@gmail.com> wrote:
> Hi Benson,
>
> Have you considered implementing a custom
> https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html
> that handles DNS changes in a better way than the default implementation?
> You could probably use http://www.xbill.org/dnsjava/ and lookup names
> directly with the amazon NS?
Thanks for reminding me to ask about that. Does HC call the resolver
as part of deciding whether to keep a connection or not? The problem
as we see it is this: the DNS mapping for an ELB changes, and a minute
later the connections to it become useless but don't necessarily get
properly closed with a FIN by the ELB. I've written some goofy code
that notices when the address resolution changes for any of the ELB's
we care about, and when it does it calls 'closeIdleConnections' with a
time of 0. On the other hand, I don't think we're currently having any
problems getting InetAddress.getByName to keep up with Amazon, but
your suggestion would be safer.
>
> Stefan
>
> 2016-06-20 19:57 GMT+02:00 Benson Margulies <bi...@gmail.com>:
>
>> We use the async http components library to connect various components
>> in a system that is deployed, in some cases, on AWS. When it's
>> deployed on AWS, there are ELBs involved.
>>
>> Like some other people, we've having various trauma dealing with the
>> very dynamic environment of AWS ELBs, which includes constant DNS
>> changes and some other issues. So far, we've been making our HC
>> configuration more and more complex. I'm wondering if we should back
>> off and make it simpler -- make a new connection for each request.
>> Obviously, this will be somewhat slower and more expensive, but at the
>> moment we might prefer a slower and more expensive approach that works
>> to one that occasionally gets lost.
>>
>> I appreciate that even making new connections all the time won't
>> compensate for all possible DNS problems -- if Java itself caches a
>> stale name->address translation, making a new connection will fail
>> just as effectively as anything else.
>>
>> Does anyone out there have advice to share (other than running away from
>> ELB)?
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
>
> --
> BEKK Open
> http://open.bekk.no
>
> TesTcl - a unit test framework for iRules
> http://testcl.com
---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org
Re: Telling HC to make new connections all the time
Posted by Stefan Magnus Landrø <st...@gmail.com>.
Hi Benson,
Have you considered implementing a custom
https://hc.apache.org/httpcomponents-client-ga/httpclient/apidocs/org/apache/http/conn/DnsResolver.html
that handles DNS changes in a better way than the default implementation?
You could probably use http://www.xbill.org/dnsjava/ and lookup names
directly with the amazon NS?
Stefan
2016-06-20 19:57 GMT+02:00 Benson Margulies <bi...@gmail.com>:
> We use the async http components library to connect various components
> in a system that is deployed, in some cases, on AWS. When it's
> deployed on AWS, there are ELBs involved.
>
> Like some other people, we've having various trauma dealing with the
> very dynamic environment of AWS ELBs, which includes constant DNS
> changes and some other issues. So far, we've been making our HC
> configuration more and more complex. I'm wondering if we should back
> off and make it simpler -- make a new connection for each request.
> Obviously, this will be somewhat slower and more expensive, but at the
> moment we might prefer a slower and more expensive approach that works
> to one that occasionally gets lost.
>
> I appreciate that even making new connections all the time won't
> compensate for all possible DNS problems -- if Java itself caches a
> stale name->address translation, making a new connection will fail
> just as effectively as anything else.
>
> Does anyone out there have advice to share (other than running away from
> ELB)?
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>
--
BEKK Open
http://open.bekk.no
TesTcl - a unit test framework for iRules
http://testcl.com