You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Serge Krawczenko <sk...@gmail.com> on 2022/06/08 13:19:54 UTC
[users@httpd] reverse proxy issue
Hello,
My issue is very simple but complicated by sudden behavior.
We're unable to resolve it for years so decided to request community's help.
We have application server behind httpd as a reverse proxy,
with basic ldap auth, so only ldap authenticated users are able to get into
the application.
( Require valid-user
Allow from all
)
Everything works well and configuration is very basic.
However,
Sometimes like once in a month or two it fails and httpd stops acting like
a reverse proxy.
I.e. https://url/my-app fails with 404 trying to find my-app locally.
(the requested URL /my-app was not found on this server)
It is 'fixed' by restarting httpd and is very annoying for users.
Debug turned on for proxy and ldap modules and there's absolutely nothing
suspicious.
When the situation occurs, there's just no attempt to refer to the
'proxypass' it is
just trying to get the local path immediately.
It cannot be reproduced as well and there's no specific periodicity for
this failure.
Anything i'm missing? Any more debug to turn on?
appreciate any ideas.
Re: [users@httpd] reverse proxy issue
Posted by Frank Gingras <th...@apache.org>.
In this case, running apachectl -S or httpd -S helps to spot overlaps.
Secondly, define all vhosts as *:PORT (only use IP:PORT when you know what
you are doing, and NEVER use <hostname>:PORT).
If you're still stuck, share the sanitized version of httpd -S
Also, is there a reason why you're still using prefork and not event
instead?
On Thu, 9 Jun 2022 at 05:05, Serge Krawczenko <sk...@gmail.com> wrote:
> Thank you, i think i'm getting the idea
>
> Indeed, there are a number of VirtualHost entries and somehow i didn't
> check if they all fail with described symptoms or just specific one.
> Therefore my assumption is - when it happens - no matter for what reason -
> this VirtualHost somehow gets removed from
> the routing/search order (or whatever else, i'm not familiar with
> architecture) and requests are routed to different irrelevant VirtualHost.
>
> I went deeper and found in logs that assumption seems to be correct.
> AH02043: SSL virtual host for servername <my vhost> found
> AH01964: Connection to child 4 established (server DEFAULT:443)
>
> and this is WRONG one, actually the mock from 00default.conf
>
> MPM module configuration i think is default, i can see
>
> LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
>
> in /etc/httpd/conf.modules.d/00-mpm.conf
>
>
> So the goal now is to determine the cause why the proper VirtualHost gets
> skipped.
> And it would be nice to find a permanent fix for it.
>
> Regarding the ProxyPass statement, it points to application FQDN,
> is there any possibility that Application temporary outage (e.g. 404) will
> cause the VirtualHost elimination?
>
>
> On Wed, Jun 8, 2022 at 8:07 PM Yann Ylavic <yl...@gmail.com> wrote:
>
>> On Wed, Jun 8, 2022 at 3:21 PM Serge Krawczenko <sk...@gmail.com>
>> wrote:
>> >
>> > We have application server behind httpd as a reverse proxy,
>> > with basic ldap auth, so only ldap authenticated users are able to get
>> into the application.
>> > ( Require valid-user
>> > Allow from all
>> > )
>>
>> It's usually not a good idea to mix Require (httpd >= 2.4) and Allow
>> (httpd < 2.4) directives together, though I don't see how it could
>> lead to the symptoms you are seeing.
>>
>> >
>> > Everything works well and configuration is very basic.
>> > However,
>> > Sometimes like once in a month or two it fails and httpd stops acting
>> like a reverse proxy.
>> > I.e. https://url/my-app fails with 404 trying to find my-app locally.
>> > (the requested URL /my-app was not found on this server)
>>
>> If there are multiple VirtualHosts on the same IP:port in your server,
>> possibly some requests reach one with no "ProxyPass /my-app ..."
>> defined?
>> It may happen if an unknown Host/SNI is handled by the default vhost
>> for instance, but it wouldn't explain why all the following requests
>> go there (unless the app/something at some point starts redirecting to
>> a different URL).
>>
>> >
>> > It is 'fixed' by restarting httpd and is very annoying for users.
>> >
>> > Debug turned on for proxy and ldap modules and there's absolutely
>> nothing suspicious.
>> > When the situation occurs, there's just no attempt to refer to the
>> 'proxypass' it is
>> > just trying to get the local path immediately.
>> >
>> > It cannot be reproduced as well and there's no specific periodicity for
>> this failure.
>> > Anything i'm missing? Any more debug to turn on?
>>
>> Which httpd MPM and modules are used? It could be a non thread-safe
>> module running on a threaded MPM, corrupting some global state when
>> the load increases.
>>
>>
>> Regards;
>> Yann.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
Re: [users@httpd] reverse proxy issue
Posted by Serge Krawczenko <sk...@gmail.com>.
Thank you, i think i'm getting the idea
Indeed, there are a number of VirtualHost entries and somehow i didn't
check if they all fail with described symptoms or just specific one.
Therefore my assumption is - when it happens - no matter for what reason -
this VirtualHost somehow gets removed from
the routing/search order (or whatever else, i'm not familiar with
architecture) and requests are routed to different irrelevant VirtualHost.
I went deeper and found in logs that assumption seems to be correct.
AH02043: SSL virtual host for servername <my vhost> found
AH01964: Connection to child 4 established (server DEFAULT:443)
and this is WRONG one, actually the mock from 00default.conf
MPM module configuration i think is default, i can see
LoadModule mpm_prefork_module modules/mod_mpm_prefork.so
in /etc/httpd/conf.modules.d/00-mpm.conf
So the goal now is to determine the cause why the proper VirtualHost gets
skipped.
And it would be nice to find a permanent fix for it.
Regarding the ProxyPass statement, it points to application FQDN,
is there any possibility that Application temporary outage (e.g. 404) will
cause the VirtualHost elimination?
On Wed, Jun 8, 2022 at 8:07 PM Yann Ylavic <yl...@gmail.com> wrote:
> On Wed, Jun 8, 2022 at 3:21 PM Serge Krawczenko <sk...@gmail.com>
> wrote:
> >
> > We have application server behind httpd as a reverse proxy,
> > with basic ldap auth, so only ldap authenticated users are able to get
> into the application.
> > ( Require valid-user
> > Allow from all
> > )
>
> It's usually not a good idea to mix Require (httpd >= 2.4) and Allow
> (httpd < 2.4) directives together, though I don't see how it could
> lead to the symptoms you are seeing.
>
> >
> > Everything works well and configuration is very basic.
> > However,
> > Sometimes like once in a month or two it fails and httpd stops acting
> like a reverse proxy.
> > I.e. https://url/my-app fails with 404 trying to find my-app locally.
> > (the requested URL /my-app was not found on this server)
>
> If there are multiple VirtualHosts on the same IP:port in your server,
> possibly some requests reach one with no "ProxyPass /my-app ..."
> defined?
> It may happen if an unknown Host/SNI is handled by the default vhost
> for instance, but it wouldn't explain why all the following requests
> go there (unless the app/something at some point starts redirecting to
> a different URL).
>
> >
> > It is 'fixed' by restarting httpd and is very annoying for users.
> >
> > Debug turned on for proxy and ldap modules and there's absolutely
> nothing suspicious.
> > When the situation occurs, there's just no attempt to refer to the
> 'proxypass' it is
> > just trying to get the local path immediately.
> >
> > It cannot be reproduced as well and there's no specific periodicity for
> this failure.
> > Anything i'm missing? Any more debug to turn on?
>
> Which httpd MPM and modules are used? It could be a non thread-safe
> module running on a threaded MPM, corrupting some global state when
> the load increases.
>
>
> Regards;
> Yann.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] reverse proxy issue
Posted by Deepak Goel <de...@gmail.com>.
How does Apache access the application server? Is the ip address of
application server changing (and it coincides with the 404 error)?
Deepak
"The greatness of a nation can be judged by the way its animals are treated
- Mahatma Gandhi"
+91 73500 12833
deicool@gmail.com
Facebook: https://www.facebook.com/deicool
LinkedIn: www.linkedin.com/in/deicool
"Plant a Tree, Go Green"
Make In India : http://www.makeinindia.com/home
On Wed, Jun 8, 2022 at 10:36 PM Yann Ylavic <yl...@gmail.com> wrote:
> On Wed, Jun 8, 2022 at 3:21 PM Serge Krawczenko <sk...@gmail.com>
> wrote:
> >
> > We have application server behind httpd as a reverse proxy,
> > with basic ldap auth, so only ldap authenticated users are able to get
> into the application.
> > ( Require valid-user
> > Allow from all
> > )
>
> It's usually not a good idea to mix Require (httpd >= 2.4) and Allow
> (httpd < 2.4) directives together, though I don't see how it could
> lead to the symptoms you are seeing.
>
> >
> > Everything works well and configuration is very basic.
> > However,
> > Sometimes like once in a month or two it fails and httpd stops acting
> like a reverse proxy.
> > I.e. https://url/my-app fails with 404 trying to find my-app locally.
> > (the requested URL /my-app was not found on this server)
>
> If there are multiple VirtualHosts on the same IP:port in your server,
> possibly some requests reach one with no "ProxyPass /my-app ..."
> defined?
> It may happen if an unknown Host/SNI is handled by the default vhost
> for instance, but it wouldn't explain why all the following requests
> go there (unless the app/something at some point starts redirecting to
> a different URL).
>
> >
> > It is 'fixed' by restarting httpd and is very annoying for users.
> >
> > Debug turned on for proxy and ldap modules and there's absolutely
> nothing suspicious.
> > When the situation occurs, there's just no attempt to refer to the
> 'proxypass' it is
> > just trying to get the local path immediately.
> >
> > It cannot be reproduced as well and there's no specific periodicity for
> this failure.
> > Anything i'm missing? Any more debug to turn on?
>
> Which httpd MPM and modules are used? It could be a non thread-safe
> module running on a threaded MPM, corrupting some global state when
> the load increases.
>
>
> Regards;
> Yann.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
Re: [users@httpd] reverse proxy issue
Posted by Yann Ylavic <yl...@gmail.com>.
On Wed, Jun 8, 2022 at 3:21 PM Serge Krawczenko <sk...@gmail.com> wrote:
>
> We have application server behind httpd as a reverse proxy,
> with basic ldap auth, so only ldap authenticated users are able to get into the application.
> ( Require valid-user
> Allow from all
> )
It's usually not a good idea to mix Require (httpd >= 2.4) and Allow
(httpd < 2.4) directives together, though I don't see how it could
lead to the symptoms you are seeing.
>
> Everything works well and configuration is very basic.
> However,
> Sometimes like once in a month or two it fails and httpd stops acting like a reverse proxy.
> I.e. https://url/my-app fails with 404 trying to find my-app locally.
> (the requested URL /my-app was not found on this server)
If there are multiple VirtualHosts on the same IP:port in your server,
possibly some requests reach one with no "ProxyPass /my-app ..."
defined?
It may happen if an unknown Host/SNI is handled by the default vhost
for instance, but it wouldn't explain why all the following requests
go there (unless the app/something at some point starts redirecting to
a different URL).
>
> It is 'fixed' by restarting httpd and is very annoying for users.
>
> Debug turned on for proxy and ldap modules and there's absolutely nothing suspicious.
> When the situation occurs, there's just no attempt to refer to the 'proxypass' it is
> just trying to get the local path immediately.
>
> It cannot be reproduced as well and there's no specific periodicity for this failure.
> Anything i'm missing? Any more debug to turn on?
Which httpd MPM and modules are used? It could be a non thread-safe
module running on a threaded MPM, corrupting some global state when
the load increases.
Regards;
Yann.
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org