You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Daniel Ferradal <df...@apache.org> on 2021/04/22 10:20:36 UTC
[users@httpd] mod_proxy_hcheck response timeout?
Hello,
I'm trying to examine a case in which we want a balancer member to be
set on fail if it can't responde in specific time to the hcheck
request
This example configuration:
<Proxy balancer://appbalancer>
# example member 1
BalancerMember http://ip:port/ hcmethod=HEAD hcuri=/ping hcpasses=1
hcfails=1 hcinterval=5 timeout=5 retry=30
# more realistic example member 2 and others
BalancerMember http://ip:port/ hcmethod=HEAD hcuri=/ping hcpasses=1
hcfails=1 hcinterval=5 timeout=60 retry=30
BalancerMember http://ip:port/ hcmethod=HEAD hcuri=/ping hcpasses=1
hcfails=1 hcinterval=5 timeout=60 retry=30
BalancerMember http://ip:port/ hcmethod=HEAD hcuri=/ping hcpasses=1
hcfails=1 hcinterval=5 timeout=60 retry=30
BalancerMember http://ip:port/ hcmethod=HEAD hcuri=/ping hcpasses=1
hcfails=1 hcinterval=5 timeout=60 retry=30
</Proxy>
ProxyPass /app balancer://appbalancer/app failontimeout=on
Let's assume the backend server 1 takes more than 5 seconds for the
/ping I would want to be set on fail and the Apache httpd server use
the other balancermember, if there are more members in the balancer,
for httpd to choose those other members which respond to the ping in a
timely fashion, but in my tests, if /ping reply goes above 5 seconds,
mod_proxy_hcheck is still saying it is ok.
Is my approach correct? As with most initial approaches to a specific
scenario, this may very well be the case... still bear with me.
Right now if the backend responds to the /ping in for example, 6
seconds, it is not being marked on fail, so as I understand any normal
request which lasts more than 5 seconds will, I don´t need hcheck for
that but this is just an example, there may be other more real
scenarios like the other members I have defined
for example Balancermember 2 we set timeout 60 seconds, app takes up
to 30-40 seconds to reply to specific requests but suppose we have
plenty of balancer members and may want to disable the member when it
degrades the /ping reply to more than 10 seconds because that means it
is getting degraded.
Is it possible to set a timeout for hchecks to set a member as failed?
Is my approach correct? As with most initial approaches to a specific
scenario, this may very well be the case, which other approach do you
suggest?
I check there is a change coming to mod_proxy_hcheck,
https://svn.apache.org/viewvc?view=revision&revision=1887119, not sure
if it applies to my case as it does not seem to mention response
timeout, rather than checks piling up.
Thanks in advance
--
Daniel Ferradal
HTTPD Project
#httpd help at Freenode
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy_hcheck response timeout?
Posted by Daniel Ferradal <df...@apache.org>.
Ahh, thanks now I understand. I appreciate your feedback.
El lun, 26 abr 2021 a las 17:48, Suvendu Sekhar Mondal
(<su...@gmail.com>) escribió:
>
> Hello Daniel,
>
> On Mon, Apr 26, 2021 at 7:02 PM Daniel Ferradal <df...@apache.org> wrote:
> >
> > Thanks for answering
> >
> > Can you please elaborate on this?
> >
> >
> What I did was, I changed the code of the page a bit so that it throws
> an exception(in my case of type Java) after waiting for some time. In
> the real world it could be something like this:
>
> 1. Apache hits that page during health check
> 2. That page has the code which does the "real" health check of the
> target application - it could be like invoking another URL or
> something else which tells you about your app's health
> 3. After invoking that URL, it waits for x amount of time for the
> response to come before giving up. This is the alternate timeout.
> 4. If it gets a response from that URL call within x amount of time -
> then we can say that the worker is healthy, else I am throwing an
> exception which will send back HTTP 500
> 5. As Apache health check is getting 500 back, it fails that worker's
> health check
>
> During my POC, that's the only way I was able to alternatively
> simulate a health check timeout failure.
>
> Thanks!
> Suvendu
>
> > > What I found is that if I can generate an exception(say
> > > timeoutexception) after waiting for some time then the health check is
> > > failing properly and marking the worker as down. This could be the
> > > alternative option until the health check module is enhanced.
> >
> >
> > --
> > Daniel Ferradal
> > HTTPD Project
> > #httpd help at Freenode
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
> >
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
--
Daniel Ferradal
HTTPD Project
#httpd help at Freenode
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy_hcheck response timeout?
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Hello Daniel,
On Mon, Apr 26, 2021 at 7:02 PM Daniel Ferradal <df...@apache.org> wrote:
>
> Thanks for answering
>
> Can you please elaborate on this?
>
>
What I did was, I changed the code of the page a bit so that it throws
an exception(in my case of type Java) after waiting for some time. In
the real world it could be something like this:
1. Apache hits that page during health check
2. That page has the code which does the "real" health check of the
target application - it could be like invoking another URL or
something else which tells you about your app's health
3. After invoking that URL, it waits for x amount of time for the
response to come before giving up. This is the alternate timeout.
4. If it gets a response from that URL call within x amount of time -
then we can say that the worker is healthy, else I am throwing an
exception which will send back HTTP 500
5. As Apache health check is getting 500 back, it fails that worker's
health check
During my POC, that's the only way I was able to alternatively
simulate a health check timeout failure.
Thanks!
Suvendu
> > What I found is that if I can generate an exception(say
> > timeoutexception) after waiting for some time then the health check is
> > failing properly and marking the worker as down. This could be the
> > alternative option until the health check module is enhanced.
>
>
> --
> Daniel Ferradal
> HTTPD Project
> #httpd help at Freenode
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy_hcheck response timeout?
Posted by Daniel Ferradal <df...@apache.org>.
Thanks for answering
Can you please elaborate on this?
> What I found is that if I can generate an exception(say
> timeoutexception) after waiting for some time then the health check is
> failing properly and marking the worker as down. This could be the
> alternative option until the health check module is enhanced.
--
Daniel Ferradal
HTTPD Project
#httpd help at Freenode
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy_hcheck response timeout?
Posted by Suvendu Sekhar Mondal <su...@gmail.com>.
Hi Daniel,
On Thu, Apr 22, 2021 at 3:50 PM Daniel Ferradal <df...@apache.org> wrote:
>
> Hello,
>
> I'm trying to examine a case in which we want a balancer member to be
> set on fail if it can't responde in specific time to the hcheck
> request
>
> This example configuration:
>
> <Proxy balancer://appbalancer>
> # example member 1
> BalancerMember http://ip:port/ hcmethod=HEAD hcuri=/ping hcpasses=1
> hcfails=1 hcinterval=5 timeout=5 retry=30
>
> # more realistic example member 2 and others
> BalancerMember http://ip:port/ hcmethod=HEAD hcuri=/ping hcpasses=1
> hcfails=1 hcinterval=5 timeout=60 retry=30
>
> BalancerMember http://ip:port/ hcmethod=HEAD hcuri=/ping hcpasses=1
> hcfails=1 hcinterval=5 timeout=60 retry=30
>
> BalancerMember http://ip:port/ hcmethod=HEAD hcuri=/ping hcpasses=1
> hcfails=1 hcinterval=5 timeout=60 retry=30
>
> BalancerMember http://ip:port/ hcmethod=HEAD hcuri=/ping hcpasses=1
> hcfails=1 hcinterval=5 timeout=60 retry=30
> </Proxy>
>
> ProxyPass /app balancer://appbalancer/app failontimeout=on
>
> Let's assume the backend server 1 takes more than 5 seconds for the
> /ping I would want to be set on fail and the Apache httpd server use
> the other balancermember, if there are more members in the balancer,
> for httpd to choose those other members which respond to the ping in a
> timely fashion, but in my tests, if /ping reply goes above 5 seconds,
> mod_proxy_hcheck is still saying it is ok.
>
> Is my approach correct? As with most initial approaches to a specific
> scenario, this may very well be the case... still bear with me.
>
> Right now if the backend responds to the /ping in for example, 6
> seconds, it is not being marked on fail, so as I understand any normal
> request which lasts more than 5 seconds will, I don´t need hcheck for
> that but this is just an example, there may be other more real
> scenarios like the other members I have defined
>
> for example Balancermember 2 we set timeout 60 seconds, app takes up
> to 30-40 seconds to reply to specific requests but suppose we have
> plenty of balancer members and may want to disable the member when it
> degrades the /ping reply to more than 10 seconds because that means it
> is getting degraded.
>
> Is it possible to set a timeout for hchecks to set a member as failed?
>
> Is my approach correct? As with most initial approaches to a specific
> scenario, this may very well be the case, which other approach do you
> suggest?
>
I quickly did a small test where I purposefully made a JSP page slower
than mod_proxy timeout value by using Thread.sleep(). Then used that
JSP in my hcuri. It looks like mod_proxy_hcheck is not using/affected
by the timeout setting - just like you found. So it looks like the
health check module does not have any timeout setting.
What I found is that if I can generate an exception(say
timeoutexception) after waiting for some time then the health check is
failing properly and marking the worker as down. This could be the
alternative option until the health check module is enhanced.
> I check there is a change coming to mod_proxy_hcheck,
> https://svn.apache.org/viewvc?view=revision&revision=1887119, not sure
> if it applies to my case as it does not seem to mention response
> timeout, rather than checks piling up.
>
>
> Thanks in advance
>
>
>
> --
> Daniel Ferradal
> HTTPD Project
> #httpd help at Freenode
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy_hcheck response timeout?
Posted by Daniel Ferradal <df...@apache.org>.
Sorry for the inconvenience in the test but, now Im looking and now it
seems to be working as expected ** head scratch here **
The client that is testing this also confirms it is working.
I have been checking the logs from 11:20 to 12:30 and I see entries
going to the second worker, but since then.. I just checked.. nothing,
zero to the second worker. (I had previously stop|started apache and
the module is even loaded from some other path to make sure I don't
confuse one for the other...)
El mar, 27 abr 2021 a las 12:55, Daniel Ferradal
(<df...@apache.org>) escribió:
>
> Wow Yann,
>
> Thanks so much for taking the time.
>
> --Correction:
> When I said:
>
> >Is my approach correct? As with most initial approaches to a specific
> > scenario, this may very well be the case, which other approach do you
> > suggest?
>
> I really meant "Is my approach incorrect?" Trying to mean It is
> logical to assume things wrongly at first approach to a subject.
>
> -----
>
> I applied the patch and I'm already testing it.
>
> In a given configuration like:
> BalancerMember http://ip1:7009/ hcmethod=GET hcuri=/ping hcpasses=1
> hcfails=1 hcinterval=5 timeout=5 retry=30
> BalancerMember http://ip2:7009/ hcmethod=GET hcuri=/ping hcpasses=1
> hcfails=1 hcinterval=5 timeout=5 retry=30
>
> --
>
> The backend ip2:7009 always takes 6 seconds to reply to /ping uri for
> testing purposes.
>
> $ time curl -vk "http://ip2:7009/sites/ping"
> ...
> < HTTP/1.1 200 OK
> ...
> * Connection #0 to host ip2 left intact
> * Closing connection #0
> {"response":{"name":"testAlive.owcs","time":6.006}}
> real 0m6.057s
> user 0m0.007s
> sys 0m0.002s
>
>
> With the patch applied I seem to see mod_health_hcheck saying that it
> is disabling the worker ip2 "sometimes" but not constantly, in a lapse
> of about 1 hour I've seen the message just 8 times:
>
> [Tue Apr 27 11:12:38.396687 2021] [proxy_hcheck:info] [pid 28713:tid
> 140079122761472] AH03303: Health check DISABLING http://ip2:port/
>
> Since the backend ip2 is configured to respond in 6 seconds all the
> times to that /ping URI (not the rest of normal uri). So
> mod_proxy_hcheck should be disabling this worker constantly. Am I
> correct?
>
> If you need more information or some specific logging do not hesitate to ask.
>
> Cheers
>
>
>
>
>
> El lun, 26 abr 2021 a las 22:51, Yann Ylavic (<yl...@gmail.com>) escribió:
> >
> > Hi Daniel,
> >
> > On Thu, Apr 22, 2021 at 12:21 PM Daniel Ferradal <df...@apache.org> wrote:
> > >
> > > Is my approach correct? As with most initial approaches to a specific
> > > scenario, this may very well be the case, which other approach do you
> > > suggest?
> >
> > I think that it's missing in the code but can't think of a workaround.
> > Does the attached patch working for you?
> >
> > Regards;
> > Yann.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> > For additional commands, e-mail: users-help@httpd.apache.org
>
>
>
> --
> Daniel Ferradal
> HTTPD Project
> #httpd help at Freenode
--
Daniel Ferradal
HTTPD Project
#httpd help at Freenode
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy_hcheck response timeout?
Posted by Daniel Ferradal <df...@apache.org>.
Wow Yann,
Thanks so much for taking the time.
--Correction:
When I said:
>Is my approach correct? As with most initial approaches to a specific
> scenario, this may very well be the case, which other approach do you
> suggest?
I really meant "Is my approach incorrect?" Trying to mean It is
logical to assume things wrongly at first approach to a subject.
-----
I applied the patch and I'm already testing it.
In a given configuration like:
BalancerMember http://ip1:7009/ hcmethod=GET hcuri=/ping hcpasses=1
hcfails=1 hcinterval=5 timeout=5 retry=30
BalancerMember http://ip2:7009/ hcmethod=GET hcuri=/ping hcpasses=1
hcfails=1 hcinterval=5 timeout=5 retry=30
--
The backend ip2:7009 always takes 6 seconds to reply to /ping uri for
testing purposes.
$ time curl -vk "http://ip2:7009/sites/ping"
...
< HTTP/1.1 200 OK
...
* Connection #0 to host ip2 left intact
* Closing connection #0
{"response":{"name":"testAlive.owcs","time":6.006}}
real 0m6.057s
user 0m0.007s
sys 0m0.002s
With the patch applied I seem to see mod_health_hcheck saying that it
is disabling the worker ip2 "sometimes" but not constantly, in a lapse
of about 1 hour I've seen the message just 8 times:
[Tue Apr 27 11:12:38.396687 2021] [proxy_hcheck:info] [pid 28713:tid
140079122761472] AH03303: Health check DISABLING http://ip2:port/
Since the backend ip2 is configured to respond in 6 seconds all the
times to that /ping URI (not the rest of normal uri). So
mod_proxy_hcheck should be disabling this worker constantly. Am I
correct?
If you need more information or some specific logging do not hesitate to ask.
Cheers
El lun, 26 abr 2021 a las 22:51, Yann Ylavic (<yl...@gmail.com>) escribió:
>
> Hi Daniel,
>
> On Thu, Apr 22, 2021 at 12:21 PM Daniel Ferradal <df...@apache.org> wrote:
> >
> > Is my approach correct? As with most initial approaches to a specific
> > scenario, this may very well be the case, which other approach do you
> > suggest?
>
> I think that it's missing in the code but can't think of a workaround.
> Does the attached patch working for you?
>
> Regards;
> Yann.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
--
Daniel Ferradal
HTTPD Project
#httpd help at Freenode
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] mod_proxy_hcheck response timeout?
Posted by Yann Ylavic <yl...@gmail.com>.
Hi Daniel,
On Thu, Apr 22, 2021 at 12:21 PM Daniel Ferradal <df...@apache.org> wrote:
>
> Is my approach correct? As with most initial approaches to a specific
> scenario, this may very well be the case, which other approach do you
> suggest?
I think that it's missing in the code but can't think of a workaround.
Does the attached patch working for you?
Regards;
Yann.