You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Ursa Polaris <po...@gmail.com> on 2011/02/23 19:18:24 UTC
[users@httpd] Apache Randomly Dropping POST Data
So I have had my Apache 2.2 server randomly dropping POST data (all
or nothing, it doesn't just drop bits or pieces) to Perl-CGI scripts
for well over a year now on production servers. So far we've put up
with it because it only seems to happen about 1% of the time, however
as our sites are growing, it's being noticed and complained about more
frequently by our users. I'd like to figure out what's causing it and
how we can fix it, but I'm afraid I don't know enough about deep
Apache 2 config to "do the debugging myself". It has been previously
suggested that the mod_rewrite engine may be the culprit, so we did
away with it entirely on our servers for a few weeks to test that
theory, and there was no change in the presence of this issue. Sadly
it's not an issue that's reliably reproducible, with a test script we
run (which is completely analogous with our other scripts, code-wise)
it can take 40-50 thousand form submissions before the issue crops up,
if at all. Based on that fact, it was suggested that it may be that
some threads are going wonky and starting to munge POST data, so we
dropped our requests per thread from 1,000 to 25 and if anything it
seems to have made things worse (though chances are it's the same rate
and the growth on the sites makes it feel like more).
Server info:
Apache: Apache/2.2.8 (Ubuntu)
Perl: v5.8.8 built for i486-linux-gnu-thread-multi
CGI: 3.15 (CPAN)
How we get the CGI params in our scripts:
use CGI;
my $q = new CGI;
my @params = $q->param();
my %p;
foreach my $key (@params) { $p{$key} = $q->param($key); }
my @cookies = $q->cookie();
my %cookie;
foreach my $key (@cookies) { $cookie{$key} = $q->cookie($key); }
Any help or advice would be greatly appreciated!
- Ursa
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
Posted by Jeff Trawick <tr...@gmail.com>.
On Wed, Feb 23, 2011 at 6:53 PM, Ursa Polaris <po...@gmail.com> wrote:
> This is the first I've heard of mod_cgid, so no we're not using it.
> Apparently it IS in our mods-available folder and ready to use though.
> I'm using Prefork mode which is the default for Ubuntu's stable Apache
> package. Is that a threaded MPM that should make use of mod_cgid?
Since you're using prefork, mod_cgi is fine. Forget about mod_cgid.
> Also, do you think it could be beneficial for us to build our own
> Apache from source rather than relying on what Ubuntu's placed in it's
> stable package library? I notice that it's often several minor
> versions behind, not sure if that could be an issue here though since
> this issue has been going on for a long time now. =/
I doubt that 2.2.latest would have a fix to solve this, but it is one
of those things which isn't too hard to do and could at the very least
rule out one possibility.
>
> On Wed, Feb 23, 2011 at 2:43 PM, Jeff Trawick <tr...@gmail.com> wrote:
>> On Wed, Feb 23, 2011 at 5:23 PM, Ursa Polaris <po...@gmail.com> wrote:
>>
>>> Hopefully that helps give more details and insight as well. I'm
>>> totally at a loss with this issue, but it's really bothering myself
>>> and my users something fierce. =/ I'm hoping not to be forced to move
>>> to nginx, but the lack of support from Apache folks is discouraging
>>> (not to mention the AWFUL documentation for mod_perl2)...
>>
>> Don't take this the wrong way, but within ~4 hours you've had
>> responses from two different Apache devs representing many years of
>> experience supporting customers on this technology. That's not so
>> shabby. Note that this is potentially as time consuming for others as
>> it has been to you already unless you happen to catch the attention of
>> someone who has solved this type of issue before and there's a match
>> with their solution and your observations.
>>
>> Changing some/all of the technology will presumably help, if you're
>> willing to risk other problems that have to be solved. Maybe that is
>> cheap to experiment with. It may be possible not to move totally to
>> nginx to eliminate some parts of the technology (see below).
>>
>> So real CGIs are the ones with the problem. If you're using a
>> threaded MPM, are you using mod_cgid? (you should be)
>>
>> mod_dumpio can log the request body/POST data at some intermediate
>> point between the network and the CGI. strace of CGI children (if
>> practical to do that) would give another data point. You could
>> temporarily re-implement the most crucial part of the CGI in C just to
>> get Perl/CPAN out of the picture -- doesn't have to perform the real
>> function; just has to confirm that the POST data got to the CGI
>> process. (I doubt Perl/CPAN is the problem, but then every individual
>> part looks unlikely as well.)
>>
>> Good luck!
>>
>> ---------------------------------------------------------------------
>> The official User-To-User support forum of the Apache HTTP Server Project.
>> See <URL:http://httpd.apache.org/userslist.html> for more info.
>> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>> " from the digest: users-digest-unsubscribe@httpd.apache.org
>> For additional commands, e-mail: users-help@httpd.apache.org
>>
>>
>
--
Born in Roswell... married an alien...
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
Posted by Ursa Polaris <po...@gmail.com>.
This is the first I've heard of mod_cgid, so no we're not using it.
Apparently it IS in our mods-available folder and ready to use though.
I'm using Prefork mode which is the default for Ubuntu's stable Apache
package. Is that a threaded MPM that should make use of mod_cgid?
Also, do you think it could be beneficial for us to build our own
Apache from source rather than relying on what Ubuntu's placed in it's
stable package library? I notice that it's often several minor
versions behind, not sure if that could be an issue here though since
this issue has been going on for a long time now. =/
On Wed, Feb 23, 2011 at 2:43 PM, Jeff Trawick <tr...@gmail.com> wrote:
> On Wed, Feb 23, 2011 at 5:23 PM, Ursa Polaris <po...@gmail.com> wrote:
>
>> Hopefully that helps give more details and insight as well. I'm
>> totally at a loss with this issue, but it's really bothering myself
>> and my users something fierce. =/ I'm hoping not to be forced to move
>> to nginx, but the lack of support from Apache folks is discouraging
>> (not to mention the AWFUL documentation for mod_perl2)...
>
> Don't take this the wrong way, but within ~4 hours you've had
> responses from two different Apache devs representing many years of
> experience supporting customers on this technology. That's not so
> shabby. Note that this is potentially as time consuming for others as
> it has been to you already unless you happen to catch the attention of
> someone who has solved this type of issue before and there's a match
> with their solution and your observations.
>
> Changing some/all of the technology will presumably help, if you're
> willing to risk other problems that have to be solved. Maybe that is
> cheap to experiment with. It may be possible not to move totally to
> nginx to eliminate some parts of the technology (see below).
>
> So real CGIs are the ones with the problem. If you're using a
> threaded MPM, are you using mod_cgid? (you should be)
>
> mod_dumpio can log the request body/POST data at some intermediate
> point between the network and the CGI. strace of CGI children (if
> practical to do that) would give another data point. You could
> temporarily re-implement the most crucial part of the CGI in C just to
> get Perl/CPAN out of the picture -- doesn't have to perform the real
> function; just has to confirm that the POST data got to the CGI
> process. (I doubt Perl/CPAN is the problem, but then every individual
> part looks unlikely as well.)
>
> Good luck!
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
Posted by Jeff Trawick <tr...@gmail.com>.
On Wed, Feb 23, 2011 at 5:23 PM, Ursa Polaris <po...@gmail.com> wrote:
> Hopefully that helps give more details and insight as well. I'm
> totally at a loss with this issue, but it's really bothering myself
> and my users something fierce. =/ I'm hoping not to be forced to move
> to nginx, but the lack of support from Apache folks is discouraging
> (not to mention the AWFUL documentation for mod_perl2)...
Don't take this the wrong way, but within ~4 hours you've had
responses from two different Apache devs representing many years of
experience supporting customers on this technology. That's not so
shabby. Note that this is potentially as time consuming for others as
it has been to you already unless you happen to catch the attention of
someone who has solved this type of issue before and there's a match
with their solution and your observations.
Changing some/all of the technology will presumably help, if you're
willing to risk other problems that have to be solved. Maybe that is
cheap to experiment with. It may be possible not to move totally to
nginx to eliminate some parts of the technology (see below).
So real CGIs are the ones with the problem. If you're using a
threaded MPM, are you using mod_cgid? (you should be)
mod_dumpio can log the request body/POST data at some intermediate
point between the network and the CGI. strace of CGI children (if
practical to do that) would give another data point. You could
temporarily re-implement the most crucial part of the CGI in C just to
get Perl/CPAN out of the picture -- doesn't have to perform the real
function; just has to confirm that the POST data got to the CGI
process. (I doubt Perl/CPAN is the problem, but then every individual
part looks unlikely as well.)
Good luck!
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
Posted by Ursa Polaris <po...@gmail.com>.
We have tested using a remote browser with a user sitting at a
browser to repro the issue (hours of hitting the enter key, holy cow),
and we have also used scripts that make LWP POST requests on the same
netblock, as well as running the LWP POST from localhost - all
versions saw about a 1% occurrence of POST data being dropped at the
Perl/CGI runtime (except for the user testing with browser, which took
about 40-50 thousand reloads to get a repro, no joke). It's obvious
that Apache is consuming the POST data at some point prior to getting
to the Perl/CGI level, but I'm a Perl-head and not an Apache-head, so
I don't know how to debug Apache to find a trace of where it's being
consumed and removed prior to getting to Perl/CGI.
Our virtualhost configuration is very minimal and simple for a
basic web apps server with some static/SSI content. There's a cgi-bin
and a handful of mod_perl2 handlers, but the scripts having the issues
are the ones in the cgi-bin path and just use plain Perl with CGI
GET/POST parameter parsing. Here's a list of the modules we have
enabled on the server, all of them are default config (from Ubuntu's
stable packages), nothing custom:
/etc/apache2/mods-enabled/alias.load
/etc/apache2/mods-enabled/auth_basic.load
/etc/apache2/mods-enabled/authn_file.load
/etc/apache2/mods-enabled/authz_default.load
/etc/apache2/mods-enabled/authz_groupfile.load
/etc/apache2/mods-enabled/authz_host.load
/etc/apache2/mods-enabled/authz_user.load
/etc/apache2/mods-enabled/autoindex.load
/etc/apache2/mods-enabled/cgi.load
/etc/apache2/mods-enabled/dir.load
/etc/apache2/mods-enabled/env.load
/etc/apache2/mods-enabled/include.load
/etc/apache2/mods-enabled/mime.load
/etc/apache2/mods-enabled/mod-wsgi.load
/etc/apache2/mods-enabled/negotiation.load
/etc/apache2/mods-enabled/perl.load
/etc/apache2/mods-enabled/php5.load
/etc/apache2/mods-enabled/rewrite.load
/etc/apache2/mods-enabled/setenvif.load
/etc/apache2/mods-enabled/status.load
Hopefully that helps give more details and insight as well. I'm
totally at a loss with this issue, but it's really bothering myself
and my users something fierce. =/ I'm hoping not to be forced to move
to nginx, but the lack of support from Apache folks is discouraging
(not to mention the AWFUL documentation for mod_perl2)...
On Wed, Feb 23, 2011 at 1:46 PM, Jeff Trawick <tr...@gmail.com> wrote:
> On Wed, Feb 23, 2011 at 4:01 PM, Ursa Polaris <po...@gmail.com> wrote:
>> I guess I forgot to mention that we have verified using WireShark
>> that Chrome, Firefox and IE are all correctly sending the POST data
>> over the network in these cases. It's not a browser issue.
>
> that's great info
> (note that you have to be a little bit careful when looking at the
> packet trace; you may see a TCP connection with the entire request
> header and POST body sent yet it receives no response; then the
> browser opens a new TCP connection, sends the request header without
> body, and after Timeout seconds gets an error response)
>
> unless you have any single sign-on modules loaded (always a good
> target to blame) or other third-party modules which can process the
> request body, no other ideas here; possibly the timing of the body
> being sent vs. error being triggered would narrow down the
> possibilities
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
Posted by Jeff Trawick <tr...@gmail.com>.
On Wed, Feb 23, 2011 at 4:01 PM, Ursa Polaris <po...@gmail.com> wrote:
> I guess I forgot to mention that we have verified using WireShark
> that Chrome, Firefox and IE are all correctly sending the POST data
> over the network in these cases. It's not a browser issue.
that's great info
(note that you have to be a little bit careful when looking at the
packet trace; you may see a TCP connection with the entire request
header and POST body sent yet it receives no response; then the
browser opens a new TCP connection, sends the request header without
body, and after Timeout seconds gets an error response)
unless you have any single sign-on modules loaded (always a good
target to blame) or other third-party modules which can process the
request body, no other ideas here; possibly the timing of the body
being sent vs. error being triggered would narrow down the
possibilities
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
Posted by Ursa Polaris <po...@gmail.com>.
I guess I forgot to mention that we have verified using WireShark
that Chrome, Firefox and IE are all correctly sending the POST data
over the network in these cases. It's not a browser issue.
On Wed, Feb 23, 2011 at 11:30 AM, Jeff Trawick <tr...@gmail.com> wrote:
> On Wed, Feb 23, 2011 at 1:51 PM, Eric Covener <co...@gmail.com> wrote:
>> On Wed, Feb 23, 2011 at 1:18 PM, Ursa Polaris <po...@gmail.com> wrote:
>>> So I have had my Apache 2.2 server randomly dropping POST data (all
>>> or nothing, it doesn't just drop bits or pieces)
>>
>> Doesn't IE sometimes forget to send the POST data when it re-connects
>> after an error?
>
> yes
>
> a few things to look for to try to matc to IE-missing-post-body:
>
> 1. client is IE (from user-agent)
> 2. request fails after Timeout seconds
> 3. the first request on the connection (log via %k)
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
> " from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
Posted by Jeff Trawick <tr...@gmail.com>.
On Wed, Feb 23, 2011 at 1:51 PM, Eric Covener <co...@gmail.com> wrote:
> On Wed, Feb 23, 2011 at 1:18 PM, Ursa Polaris <po...@gmail.com> wrote:
>> So I have had my Apache 2.2 server randomly dropping POST data (all
>> or nothing, it doesn't just drop bits or pieces)
>
> Doesn't IE sometimes forget to send the POST data when it re-connects
> after an error?
yes
a few things to look for to try to matc to IE-missing-post-body:
1. client is IE (from user-agent)
2. request fails after Timeout seconds
3. the first request on the connection (log via %k)
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org
Re: [users@httpd] Apache Randomly Dropping POST Data
Posted by Eric Covener <co...@gmail.com>.
On Wed, Feb 23, 2011 at 1:18 PM, Ursa Polaris <po...@gmail.com> wrote:
> So I have had my Apache 2.2 server randomly dropping POST data (all
> or nothing, it doesn't just drop bits or pieces)
Doesn't IE sometimes forget to send the POST data when it re-connects
after an error?
---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
" from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org