You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Robert Gabriel <ep...@gmail.com> on 2013/06/24 15:03:39 UTC
[users@httpd] Mod_proxy Slow After a Week
Hello all,
We have:
Apache 2.2.3
CentOS 5.5 x86_64
Splunk 5.0.2
I only know the basics but Apache has been serving us very well with the
below config
and only after about a week did pages refresh very slowly, up to a minute
to reload
sometimes.
I tailed both httpd and splunkd logs and saw a consistent delay of 30s
between proxy
and origin server (both on same host) along with plenty of 304s, followed
by what
appeared to be some timeout and then slowly the 200s started coming back in.
A restart of httpd cleared up the issue.
I'm lost, please help.
Could this be a caching problem?
Thank you.
<VirtualHost *:443>
SSLEngine on
SSLCertificateFile /etc/httpd/conf/server.crt
SSLCertificateKeyFile /etc/httpd/conf/server.key
SSLProxyEngine On
SSLCACertificateFile /etc/httpd/conf/gsoc.pem
SSLProtocol all -SSLv2
SSLVerifyClient require
SSLVerifyDepth 1
SSLOptions +StrictRequire
RequestHeader set X-Remote-User %{REMOTE_USER}s
ServerName dashboards.gsoc.co.za:443
ServerAdmin admin@gsoc.co.za
DocumentRoot /srv/http/gdf/
CustomLog /var/log/httpd/gdf/access combined
ErrorLog /var/log/httpd/gdf/error
LogLevel debug
ProxyRequests Off
ProxyPreserveHost Off
ProxyPass /gdf https://172.20.67.2:8000/gdf
ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
<Directory />
SSLRequireSSL
AllowOverride none
AuthName "GDF"
AuthType Basic
AuthDigestProvider file
AuthUserFile /etc/httpd/conf/passwd
Require ssl-verify-client
Require valid-user
Require ssl
Satisfy All
</Directory>
<Location /gdf>
SSLRequireSSL
AuthName "GDF"
AuthType Basic
AuthDigestProvider file
AuthUserFile /etc/httpd/conf/passwd
Require ssl-verify-client
Require valid-user
Require ssl
Satisfy All
</Location>
</VirtualHost>
Re: [users@httpd] Mod_proxy Slow After a Week
Posted by Robert Gabriel <ep...@gmail.com>.
On 24 June 2013 17:49, David Guerra <im...@gmail.com> wrote:
> You are receiving 304's?
> That doesn't make sense. Maybe 504's which many times will point to a
> load issue on the backend server.
>
>
> On Mon, Jun 24, 2013 at 9:03 AM, Robert Gabriel <ep...@gmail.com>wrote:
>
>> Hello all,
>>
>> We have:
>>
>> Apache 2.2.3
>> CentOS 5.5 x86_64
>> Splunk 5.0.2
>>
>> I only know the basics but Apache has been serving us very well with the
>> below config
>>
>> and only after about a week did pages refresh very slowly, up to a minute
>> to reload
>>
>> sometimes.
>>
>> I tailed both httpd and splunkd logs and saw a consistent delay of 30s
>> between proxy
>>
>> and origin server (both on same host) along with plenty of 304s, followed
>> by what
>>
>> appeared to be some timeout and then slowly the 200s started coming back
>> in.
>>
>> A restart of httpd cleared up the issue.
>>
>> I'm lost, please help.
>>
>> Could this be a caching problem?
>>
>> Thank you.
>>
>> <VirtualHost *:443>
>> SSLEngine on
>> SSLCertificateFile /etc/httpd/conf/server.crt
>> SSLCertificateKeyFile /etc/httpd/conf/server.key
>> SSLProxyEngine On
>> SSLCACertificateFile /etc/httpd/conf/gsoc.pem
>> SSLProtocol all -SSLv2
>> SSLVerifyClient require
>> SSLVerifyDepth 1
>> SSLOptions +StrictRequire
>>
>> RequestHeader set X-Remote-User %{REMOTE_USER}s
>>
>> ServerName dashboards.gsoc.co.za:443
>> ServerAdmin admin@gsoc.co.za
>> DocumentRoot /srv/http/gdf/
>> CustomLog /var/log/httpd/gdf/access combined
>> ErrorLog /var/log/httpd/gdf/error
>> LogLevel debug
>>
>> ProxyRequests Off
>> ProxyPreserveHost Off
>> ProxyPass /gdf https://172.20.67.2:8000/gdf
>> ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
>>
>> <Directory />
>> SSLRequireSSL
>> AllowOverride none
>> AuthName "GDF"
>> AuthType Basic
>> AuthDigestProvider file
>> AuthUserFile /etc/httpd/conf/passwd
>> Require ssl-verify-client
>> Require valid-user
>> Require ssl
>> Satisfy All
>> </Directory>
>>
>> <Location /gdf>
>> SSLRequireSSL
>> AuthName "GDF"
>> AuthType Basic
>> AuthDigestProvider file
>> AuthUserFile /etc/httpd/conf/passwd
>> Require ssl-verify-client
>> Require valid-user
>> Require ssl
>> Satisfy All
>> </Location>
>> </VirtualHost>
>>
>>
>
>
> --
> Take Care,
>
> David Guerra
> http://frustratedtech.com/
>
304s for sure, no doubt.
Re: [users@httpd] Mod_proxy Slow After a Week
Posted by David Guerra <im...@gmail.com>.
You are receiving 304's?
That doesn't make sense. Maybe 504's which many times will point to a load
issue on the backend server.
On Mon, Jun 24, 2013 at 9:03 AM, Robert Gabriel <ep...@gmail.com> wrote:
> Hello all,
>
> We have:
>
> Apache 2.2.3
> CentOS 5.5 x86_64
> Splunk 5.0.2
>
> I only know the basics but Apache has been serving us very well with the
> below config
>
> and only after about a week did pages refresh very slowly, up to a minute
> to reload
>
> sometimes.
>
> I tailed both httpd and splunkd logs and saw a consistent delay of 30s
> between proxy
>
> and origin server (both on same host) along with plenty of 304s, followed
> by what
>
> appeared to be some timeout and then slowly the 200s started coming back
> in.
>
> A restart of httpd cleared up the issue.
>
> I'm lost, please help.
>
> Could this be a caching problem?
>
> Thank you.
>
> <VirtualHost *:443>
> SSLEngine on
> SSLCertificateFile /etc/httpd/conf/server.crt
> SSLCertificateKeyFile /etc/httpd/conf/server.key
> SSLProxyEngine On
> SSLCACertificateFile /etc/httpd/conf/gsoc.pem
> SSLProtocol all -SSLv2
> SSLVerifyClient require
> SSLVerifyDepth 1
> SSLOptions +StrictRequire
>
> RequestHeader set X-Remote-User %{REMOTE_USER}s
>
> ServerName dashboards.gsoc.co.za:443
> ServerAdmin admin@gsoc.co.za
> DocumentRoot /srv/http/gdf/
> CustomLog /var/log/httpd/gdf/access combined
> ErrorLog /var/log/httpd/gdf/error
> LogLevel debug
>
> ProxyRequests Off
> ProxyPreserveHost Off
> ProxyPass /gdf https://172.20.67.2:8000/gdf
> ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
>
> <Directory />
> SSLRequireSSL
> AllowOverride none
> AuthName "GDF"
> AuthType Basic
> AuthDigestProvider file
> AuthUserFile /etc/httpd/conf/passwd
> Require ssl-verify-client
> Require valid-user
> Require ssl
> Satisfy All
> </Directory>
>
> <Location /gdf>
> SSLRequireSSL
> AuthName "GDF"
> AuthType Basic
> AuthDigestProvider file
> AuthUserFile /etc/httpd/conf/passwd
> Require ssl-verify-client
> Require valid-user
> Require ssl
> Satisfy All
> </Location>
> </VirtualHost>
>
>
--
Take Care,
David Guerra
http://frustratedtech.com/
Re: [users@httpd] Mod_proxy Slow After a Week
Posted by Robert Gabriel <ep...@gmail.com>.
On 26 June 2013 12:40, Tom Evans <te...@googlemail.com> wrote:
> On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel <ep...@gmail.com>
> wrote:
> > Hello all,
> >
> > We have:
> >
> > Apache 2.2.3
> > CentOS 5.5 x86_64
> > Splunk 5.0.2
>
> Sorry, I didn't read your original post. Please upgrade Apache to
> 2.2.24 and re-test.
>
Thank you, will do.
I'll keep an eye and advise.
Re: [users@httpd] Mod_proxy Slow After a Week
Posted by Tom Evans <te...@googlemail.com>.
On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel <ep...@gmail.com> wrote:
> Hello all,
>
> We have:
>
> Apache 2.2.3
> CentOS 5.5 x86_64
> Splunk 5.0.2
Sorry, I didn't read your original post. Please upgrade Apache to
2.2.24 and re-test.
Cheers
Tom
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Mod_proxy Slow After a Week
Posted by Robert Gabriel <ep...@gmail.com>.
On 24 June 2013 18:27, Tom Evans <te...@googlemail.com> wrote:
On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel <ep...@gmail.com> wrote:
> > Hello all,
> >
> > We have:
> >
> > Apache 2.2.3
> > CentOS 5.5 x86_64
> > Splunk 5.0.2
> >
> > I only know the basics but Apache has been serving us very well with the
> > below config
> >
> > and only after about a week did pages refresh very slowly, up to a
> minute to
> > reload
> >
> > sometimes.
> >
> > I tailed both httpd and splunkd logs and saw a consistent delay of 30s
> > between proxy
> >
> > and origin server (both on same host) along with plenty of 304s,
> followed by
> > what
> >
> > appeared to be some timeout and then slowly the 200s started coming back
> in.
> >
> > A restart of httpd cleared up the issue.
> >
> > I'm lost, please help.
> >
> > Could this be a caching problem?
> >
> > Thank you.
> >
> > <VirtualHost *:443>
> > SSLEngine on
> > SSLCertificateFile /etc/httpd/conf/server.crt
> > SSLCertificateKeyFile /etc/httpd/conf/server.key
> > SSLProxyEngine On
> > SSLCACertificateFile /etc/httpd/conf/gsoc.pem
> > SSLProtocol all -SSLv2
> > SSLVerifyClient require
> > SSLVerifyDepth 1
> > SSLOptions +StrictRequire
> >
> > RequestHeader set X-Remote-User %{REMOTE_USER}s
> >
> > ServerName dashboards.gsoc.co.za:443
> > ServerAdmin admin@gsoc.co.za
> > DocumentRoot /srv/http/gdf/
> > CustomLog /var/log/httpd/gdf/access combined
> > ErrorLog /var/log/httpd/gdf/error
> > LogLevel debug
> >
> > ProxyRequests Off
> > ProxyPreserveHost Off
> > ProxyPass /gdf https://172.20.67.2:8000/gdf
> > ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
> >
> > <Directory />
> > SSLRequireSSL
> > AllowOverride none
> > AuthName "GDF"
> > AuthType Basic
> > AuthDigestProvider file
> > AuthUserFile /etc/httpd/conf/passwd
> > Require ssl-verify-client
> > Require valid-user
> > Require ssl
> > Satisfy All
> > </Directory>
> >
> > <Location /gdf>
> > SSLRequireSSL
> > AuthName "GDF"
> > AuthType Basic
> > AuthDigestProvider file
> > AuthUserFile /etc/httpd/conf/passwd
> > Require ssl-verify-client
> > Require valid-user
> > Require ssl
> > Satisfy All
> > </Location>
> > </VirtualHost>
>
> 30 seconds is the length of the default timeout in apache.
> Unfortunately, that timeout is used in all sorts of cases, so it does
> not tell us what is timing out.
>
> As a rank guess, I would be going for DNS timeout myself. Do you have
> HostnameLookups set to "On" or "Double", or using host names in ACLs
>Sorry, I didn't read your original post. Please upgrade Apache to
>2.2.24 and re-test.
We upgraded Apache and installed CentOS-6.4.
Problem gone.
Re: [users@httpd] Mod_proxy Slow After a Week
Posted by Robert Gabriel <ep...@gmail.com>.
On 26 June 2013 09:59, Thomas Eckert <th...@gmail.com> wrote:
> This is interesting. I'm seeing similar events at one of my reverse proxy.
> After fixing a DNS infrastructure problem I started seeing lots of 304s
> with processing times well above 5 seconds. For header-only requests that's
> a lot of time. If there is any news here please let us know. At the very
> least it would help me.
Will do.
So far so good.
Thank you.
Re: [users@httpd] Mod_proxy Slow After a Week
Posted by Thomas Eckert <th...@gmail.com>.
This is interesting. I'm seeing similar events at one of my reverse proxy.
After fixing a DNS infrastructure problem I started seeing lots of 304s
with processing times well above 5 seconds. For header-only requests that's
a lot of time. If there is any news here please let us know. At the very
least it would help me.
On Mon, Jun 24, 2013 at 6:43 PM, Robert Gabriel <ep...@gmail.com> wrote:
>
>
>
> On 24 June 2013 18:27, Tom Evans <te...@googlemail.com> wrote:
>
>> On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel <ep...@gmail.com>
>> wrote:
>> > Hello all,
>> >
>> > We have:
>> >
>> > Apache 2.2.3
>> > CentOS 5.5 x86_64
>> > Splunk 5.0.2
>> >
>> > I only know the basics but Apache has been serving us very well with the
>> > below config
>> >
>> > and only after about a week did pages refresh very slowly, up to a
>> minute to
>> > reload
>> >
>> > sometimes.
>> >
>> > I tailed both httpd and splunkd logs and saw a consistent delay of 30s
>> > between proxy
>> >
>> > and origin server (both on same host) along with plenty of 304s,
>> followed by
>> > what
>> >
>> > appeared to be some timeout and then slowly the 200s started coming
>> back in.
>> >
>> > A restart of httpd cleared up the issue.
>> >
>> > I'm lost, please help.
>> >
>> > Could this be a caching problem?
>> >
>> > Thank you.
>> >
>> > <VirtualHost *:443>
>> > SSLEngine on
>> > SSLCertificateFile /etc/httpd/conf/server.crt
>> > SSLCertificateKeyFile /etc/httpd/conf/server.key
>> > SSLProxyEngine On
>> > SSLCACertificateFile /etc/httpd/conf/gsoc.pem
>> > SSLProtocol all -SSLv2
>> > SSLVerifyClient require
>> > SSLVerifyDepth 1
>> > SSLOptions +StrictRequire
>> >
>> > RequestHeader set X-Remote-User %{REMOTE_USER}s
>> >
>> > ServerName dashboards.gsoc.co.za:443
>> > ServerAdmin admin@gsoc.co.za
>> > DocumentRoot /srv/http/gdf/
>> > CustomLog /var/log/httpd/gdf/access combined
>> > ErrorLog /var/log/httpd/gdf/error
>> > LogLevel debug
>> >
>> > ProxyRequests Off
>> > ProxyPreserveHost Off
>> > ProxyPass /gdf https://172.20.67.2:8000/gdf
>> > ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
>> >
>> > <Directory />
>> > SSLRequireSSL
>> > AllowOverride none
>> > AuthName "GDF"
>> > AuthType Basic
>> > AuthDigestProvider file
>> > AuthUserFile /etc/httpd/conf/passwd
>> > Require ssl-verify-client
>> > Require valid-user
>> > Require ssl
>> > Satisfy All
>> > </Directory>
>> >
>> > <Location /gdf>
>> > SSLRequireSSL
>> > AuthName "GDF"
>> > AuthType Basic
>> > AuthDigestProvider file
>> > AuthUserFile /etc/httpd/conf/passwd
>> > Require ssl-verify-client
>> > Require valid-user
>> > Require ssl
>> > Satisfy All
>> > </Location>
>> > </VirtualHost>
>>
>> 30 seconds is the length of the default timeout in apache.
>> Unfortunately, that timeout is used in all sorts of cases, so it does
>> not tell us what is timing out.
>>
>> As a rank guess, I would be going for DNS timeout myself. Do you have
>> HostnameLookups set to "On" or "Double", or using host names in ACLs?
>>
>> Cheers
>>
>> Tom
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
> We are not doing any hostname ACLs.
>
> Forgive my limited knowledge, I did RTFM before replying to make sure I
> understood
> the intended config and no, we are not doing any.
>
> "HostnameLookups Off"
>
> How come a restart "fixes" the problem?
>
> Thank you.
>
Re: [users@httpd] Mod_proxy Slow After a Week
Posted by Robert Gabriel <ep...@gmail.com>.
On 24 June 2013 18:27, Tom Evans <te...@googlemail.com> wrote:
> On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel <ep...@gmail.com>
> wrote:
> > Hello all,
> >
> > We have:
> >
> > Apache 2.2.3
> > CentOS 5.5 x86_64
> > Splunk 5.0.2
> >
> > I only know the basics but Apache has been serving us very well with the
> > below config
> >
> > and only after about a week did pages refresh very slowly, up to a
> minute to
> > reload
> >
> > sometimes.
> >
> > I tailed both httpd and splunkd logs and saw a consistent delay of 30s
> > between proxy
> >
> > and origin server (both on same host) along with plenty of 304s,
> followed by
> > what
> >
> > appeared to be some timeout and then slowly the 200s started coming back
> in.
> >
> > A restart of httpd cleared up the issue.
> >
> > I'm lost, please help.
> >
> > Could this be a caching problem?
> >
> > Thank you.
> >
> > <VirtualHost *:443>
> > SSLEngine on
> > SSLCertificateFile /etc/httpd/conf/server.crt
> > SSLCertificateKeyFile /etc/httpd/conf/server.key
> > SSLProxyEngine On
> > SSLCACertificateFile /etc/httpd/conf/gsoc.pem
> > SSLProtocol all -SSLv2
> > SSLVerifyClient require
> > SSLVerifyDepth 1
> > SSLOptions +StrictRequire
> >
> > RequestHeader set X-Remote-User %{REMOTE_USER}s
> >
> > ServerName dashboards.gsoc.co.za:443
> > ServerAdmin admin@gsoc.co.za
> > DocumentRoot /srv/http/gdf/
> > CustomLog /var/log/httpd/gdf/access combined
> > ErrorLog /var/log/httpd/gdf/error
> > LogLevel debug
> >
> > ProxyRequests Off
> > ProxyPreserveHost Off
> > ProxyPass /gdf https://172.20.67.2:8000/gdf
> > ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
> >
> > <Directory />
> > SSLRequireSSL
> > AllowOverride none
> > AuthName "GDF"
> > AuthType Basic
> > AuthDigestProvider file
> > AuthUserFile /etc/httpd/conf/passwd
> > Require ssl-verify-client
> > Require valid-user
> > Require ssl
> > Satisfy All
> > </Directory>
> >
> > <Location /gdf>
> > SSLRequireSSL
> > AuthName "GDF"
> > AuthType Basic
> > AuthDigestProvider file
> > AuthUserFile /etc/httpd/conf/passwd
> > Require ssl-verify-client
> > Require valid-user
> > Require ssl
> > Satisfy All
> > </Location>
> > </VirtualHost>
>
> 30 seconds is the length of the default timeout in apache.
> Unfortunately, that timeout is used in all sorts of cases, so it does
> not tell us what is timing out.
>
> As a rank guess, I would be going for DNS timeout myself. Do you have
> HostnameLookups set to "On" or "Double", or using host names in ACLs?
>
> Cheers
>
> Tom
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
We are not doing any hostname ACLs.
Forgive my limited knowledge, I did RTFM before replying to make sure I
understood
the intended config and no, we are not doing any.
"HostnameLookups Off"
How come a restart "fixes" the problem?
Thank you.
Re: [users@httpd] Mod_proxy Slow After a Week
Posted by Tom Evans <te...@googlemail.com>.
On Mon, Jun 24, 2013 at 2:03 PM, Robert Gabriel <ep...@gmail.com> wrote:
> Hello all,
>
> We have:
>
> Apache 2.2.3
> CentOS 5.5 x86_64
> Splunk 5.0.2
>
> I only know the basics but Apache has been serving us very well with the
> below config
>
> and only after about a week did pages refresh very slowly, up to a minute to
> reload
>
> sometimes.
>
> I tailed both httpd and splunkd logs and saw a consistent delay of 30s
> between proxy
>
> and origin server (both on same host) along with plenty of 304s, followed by
> what
>
> appeared to be some timeout and then slowly the 200s started coming back in.
>
> A restart of httpd cleared up the issue.
>
> I'm lost, please help.
>
> Could this be a caching problem?
>
> Thank you.
>
> <VirtualHost *:443>
> SSLEngine on
> SSLCertificateFile /etc/httpd/conf/server.crt
> SSLCertificateKeyFile /etc/httpd/conf/server.key
> SSLProxyEngine On
> SSLCACertificateFile /etc/httpd/conf/gsoc.pem
> SSLProtocol all -SSLv2
> SSLVerifyClient require
> SSLVerifyDepth 1
> SSLOptions +StrictRequire
>
> RequestHeader set X-Remote-User %{REMOTE_USER}s
>
> ServerName dashboards.gsoc.co.za:443
> ServerAdmin admin@gsoc.co.za
> DocumentRoot /srv/http/gdf/
> CustomLog /var/log/httpd/gdf/access combined
> ErrorLog /var/log/httpd/gdf/error
> LogLevel debug
>
> ProxyRequests Off
> ProxyPreserveHost Off
> ProxyPass /gdf https://172.20.67.2:8000/gdf
> ProxyPassReverse /gdf https://172.20.67.2:8000/gdf
>
> <Directory />
> SSLRequireSSL
> AllowOverride none
> AuthName "GDF"
> AuthType Basic
> AuthDigestProvider file
> AuthUserFile /etc/httpd/conf/passwd
> Require ssl-verify-client
> Require valid-user
> Require ssl
> Satisfy All
> </Directory>
>
> <Location /gdf>
> SSLRequireSSL
> AuthName "GDF"
> AuthType Basic
> AuthDigestProvider file
> AuthUserFile /etc/httpd/conf/passwd
> Require ssl-verify-client
> Require valid-user
> Require ssl
> Satisfy All
> </Location>
> </VirtualHost>
30 seconds is the length of the default timeout in apache.
Unfortunately, that timeout is used in all sorts of cases, so it does
not tell us what is timing out.
As a rank guess, I would be going for DNS timeout myself. Do you have
HostnameLookups set to "On" or "Double", or using host names in ACLs?
Cheers
Tom
---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org