You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@httpd.apache.org by Timothy Partee <ti...@gmail.com> on 2010/08/06 08:50:34 UTC

[users@httpd] Apache/Perl CGI intermittent missing POST data

   I've spent a few weeks trying to research through Google to find
others having this issue, but with no success. A few Apache-savvy
folks have recommended I do something with handlers that was way over
my head to figure out what's going on here, so please be gentle with
me - I may know how to configure a few virtual hosts and an SSL
server, but I'm not an Apache guru by any stretch.

   Basically what's happening is I have a small horde of Perl (not
mod_perl, just plain Perl) scripts which use the CGI.pm module to
parse out incoming CGI parameters and cookies, and about 2% of the
time when I get the POST data params from CGI they are a completely
empty collection. I can consistently reproduce this with a simple POST
form page to a simple Perl CGI script which dumps a set of static form
data (as in the form fields are hidden and therefore no data entry is
needed so I can just hit the submit button over and over again for
rapid testing) to the browser. Everything runs fine, no errors on the
server being reported, no error in the Perl script reported, just a
blank params collection about 2% of the time, seemingly completely at
random. Under high load and under low load there seems to be no
difference.

   This started happening about a year ago when I updated my Ubuntu
packages to the latest stable packages, and I'm currently running
Apache version 2.2.8 Ubuntu Stable with Perl version 5.8.8 and CGI.pm
version 3.15 - I'm not sure what other information or version numbers
might be useful to folks. I would have reported this as an Apache bug,
but they told me to come here and ask for advice from the user group
on how to tap into handlers and do an Apache-side debug about this
before proceeding to file it as a bug.

   Also, I'm sure a ton of you folks will recommend converting
everything to mod_perl2 and that's a very noble intention, but I
simply don't have the time to convert some 200+ scripts and deal with
the headache of making sure they're all working smoothly on a new
framework like that - not right now anyway. Maybe when I win the
lottery or my boss realizes that refactoring is a Good Thing (tm)
worth devoting man-hours on!

   Any advice or direction in figuring this out would be GREATLY
appreciated. Thanks!

- Tim Partee

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Apache/Perl CGI intermittent missing POST data

Posted by Joost de Heer <jo...@sanguis.xs4all.nl>.
On 08/09/2010 10:26 AM, Timothy Partee wrote:
>     None of our rewrite rules should ever be operating on POST form
> processor scripts, however it was suggested to us that mod_rewrite is
> the likely culprit as it may be over-reactive and sometimes
> accidentally nuking our POST data when it shouldn't. It was suggested
> to disable mod_rewrite entirely for the CGI directories, not just add
> a condition to exclude them since that according to this theory will
> not save the data from getting nuked. Can mods be disabled altogether
> on a per-directory basis? Thanks,

Yes, this is possible. Use either one of these:

<Location /cgi-bin/>
	RewriteEngine off
</Location>

<Directory /path/to/cgi-bin>
	RewriteEngine off
</Directory>

Joost

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Apache/Perl CGI intermittent missing POST data

Posted by Timothy Partee <ti...@gmail.com>.
   None of our rewrite rules should ever be operating on POST form
processor scripts, however it was suggested to us that mod_rewrite is
the likely culprit as it may be over-reactive and sometimes
accidentally nuking our POST data when it shouldn't. It was suggested
to disable mod_rewrite entirely for the CGI directories, not just add
a condition to exclude them since that according to this theory will
not save the data from getting nuked. Can mods be disabled altogether
on a per-directory basis? Thanks,

- Tim Partee

On Mon, Aug 9, 2010 at 1:07 AM, Joost de Heer <jo...@sanguis.xs4all.nl> wrote:
>> It was suggested that we exclude the CGI folders from the mod_rewrite
>> rules and processing, however we could find no info in Apache
>> documentation on how to exclude mod_rewrite rules/operation from any
>> specific directories. Any pointers in the right direction would be
>> appreciated, thanks!
>
> RewriteCond %{PATH_INFO} !^/cgi-bin/.*
> RewriteRule [your rule]
>
> The problem is probably related to redirection of POST's: that is forbidden
> by the RFC's, and it is automatically changed into a GET, while losing the
> POST variables.
>
> Joost
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>  "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Apache/Perl CGI intermittent missing POST data

Posted by Joost de Heer <jo...@sanguis.xs4all.nl>.
> It was suggested that we exclude the CGI folders from the mod_rewrite
> rules and processing, however we could find no info in Apache
> documentation on how to exclude mod_rewrite rules/operation from any
> specific directories. Any pointers in the right direction would be
> appreciated, thanks!

RewriteCond %{PATH_INFO} !^/cgi-bin/.*
RewriteRule [your rule]

The problem is probably related to redirection of POST's: that is forbidden by 
the RFC's, and it is automatically changed into a GET, while losing the POST 
variables.

Joost

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Apache/Perl CGI intermittent missing POST data

Posted by Timothy Partee <ti...@gmail.com>.
   Okay, so for the last few days now I've been trying to repro the
intermittent POST data issues on my Apache servers through a test POST
form script and thus far I've had a zero repro rate out of over 40,000
POST submissions from the test script - however I've added error
logging for every time a POST form processor receives an empty set of
CGI params and it's gone off a half dozen to couple dozen times per
day out of a few thousand POST processings per day. Seems negligible,
but we commonly receive user complaints and bug reports abut this
issue nonetheless, and we're getting very tired of it, to say the
least.
   Since we cannot repro the issue in a controlled sense, and
therefore can't get mod_dumpio in on the investigation, we need some
new direction. One 'professional' on this list contacted us privately
and mentioned that mod_rewrite is the likely culprit and we are indeed
using mod_rewrite for some redirects from troublesome referrer sources
and people who attempt to host images from our site in off-site pages.
It was suggested that we exclude the CGI folders from the mod_rewrite
rules and processing, however we could find no info in Apache
documentation on how to exclude mod_rewrite rules/operation from any
specific directories. Any pointers in the right direction would be
appreciated, thanks!

- Tim Partee


On Fri, Aug 6, 2010 at 3:55 AM, Jeff Trawick <tr...@gmail.com> wrote:
> On Fri, Aug 6, 2010 at 2:50 AM, Timothy Partee <ti...@gmail.com> wrote:
>>   Basically what's happening is I have a small horde of Perl (not
>> mod_perl, just plain Perl) scripts which use the CGI.pm module to
>> parse out incoming CGI parameters and cookies, and about 2% of the
>> time when I get the POST data params from CGI they are a completely
>> empty collection. I can consistently reproduce this with a simple POST
>> form page to a simple Perl CGI script which dumps a set of static form
>> data (as in the form fields are hidden and therefore no data entry is
>> needed so I can just hit the submit button over and over again for
>> rapid testing) to the browser. Everything runs fine, no errors on the
>> server being reported, no error in the Perl script reported, just a
>> blank params collection about 2% of the time, seemingly completely at
>> random. Under high load and under low load there seems to be no
>> difference.
>
> First, confirm exactly what the browser is sending when you encounter
> the problem symptom:
>
> a) Content-Length non-zero + POST data
> b) Content-Length zero
> c) Content-Length non-zero but no post data (we'd expect the web
> server to wait for Timeout seconds before giving up on the POST data)
>
> You can easily log the incoming Content-Length in the access log.  You
> can log the incoming headers/POST data with mod_dumpio.  You can step
> outside of the web server process itself and use a packet trace tool
> such as Wireshark.  You can use something like the ieHttpHeaders
> plug-in for IE, or equivalent for Firefox, to see the perspective of
> the browser process.
>
> ---------------------------------------------------------------------
> The official User-To-User support forum of the Apache HTTP Server Project.
> See <URL:http://httpd.apache.org/userslist.html> for more info.
> To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
>   "   from the digest: users-digest-unsubscribe@httpd.apache.org
> For additional commands, e-mail: users-help@httpd.apache.org
>
>

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Apache/Perl CGI intermittent missing POST data

Posted by Ursa Polaris <po...@gmail.com>.
On Fri, Aug 6, 2010 at 3:55 AM, Jeff Trawick <tr...@gmail.com> wrote:
> On Fri, Aug 6, 2010 at 2:50 AM, Timothy Partee <ti...@gmail.com> wrote:
>>   Basically what's happening is I have a small horde of Perl (not
>> mod_perl, just plain Perl) scripts which use the CGI.pm module to
>> parse out incoming CGI parameters and cookies, and about 2% of the
>> time when I get the POST data params from CGI they are a completely
>> empty collection. I can consistently reproduce this with a simple POST
>> form page to a simple Perl CGI script which dumps a set of static form
>> data (as in the form fields are hidden and therefore no data entry is
>> needed so I can just hit the submit button over and over again for
>> rapid testing) to the browser. Everything runs fine, no errors on the
>> server being reported, no error in the Perl script reported, just a
>> blank params collection about 2% of the time, seemingly completely at
>> random. Under high load and under low load there seems to be no
>> difference.
>
> First, confirm exactly what the browser is sending when you encounter
> the problem symptom:
>
> a) Content-Length non-zero + POST data
> b) Content-Length zero
> c) Content-Length non-zero but no post data (we'd expect the web
> server to wait for Timeout seconds before giving up on the POST data)
>
> You can easily log the incoming Content-Length in the access log.  You
> can log the incoming headers/POST data with mod_dumpio.  You can step
> outside of the web server process itself and use a packet trace tool
> such as Wireshark.  You can use something like the ieHttpHeaders
> plug-in for IE, or equivalent for Firefox, to see the perspective of
> the browser process.

   Ah yes, I forgot to mention I had already used Firebug and
Wireshark to confirm that the browsers are correctly sending the POST
data to the server 100% of the time. I'll try to look into mod_dumpio
today and see if I can figure out how to use it to help trace what's
going on. Thanks,

- Tim Partee

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org


Re: [users@httpd] Apache/Perl CGI intermittent missing POST data

Posted by Jeff Trawick <tr...@gmail.com>.
On Fri, Aug 6, 2010 at 2:50 AM, Timothy Partee <ti...@gmail.com> wrote:
>   Basically what's happening is I have a small horde of Perl (not
> mod_perl, just plain Perl) scripts which use the CGI.pm module to
> parse out incoming CGI parameters and cookies, and about 2% of the
> time when I get the POST data params from CGI they are a completely
> empty collection. I can consistently reproduce this with a simple POST
> form page to a simple Perl CGI script which dumps a set of static form
> data (as in the form fields are hidden and therefore no data entry is
> needed so I can just hit the submit button over and over again for
> rapid testing) to the browser. Everything runs fine, no errors on the
> server being reported, no error in the Perl script reported, just a
> blank params collection about 2% of the time, seemingly completely at
> random. Under high load and under low load there seems to be no
> difference.

First, confirm exactly what the browser is sending when you encounter
the problem symptom:

a) Content-Length non-zero + POST data
b) Content-Length zero
c) Content-Length non-zero but no post data (we'd expect the web
server to wait for Timeout seconds before giving up on the POST data)

You can easily log the incoming Content-Length in the access log.  You
can log the incoming headers/POST data with mod_dumpio.  You can step
outside of the web server process itself and use a packet trace tool
such as Wireshark.  You can use something like the ieHttpHeaders
plug-in for IE, or equivalent for Firefox, to see the perspective of
the browser process.

---------------------------------------------------------------------
The official User-To-User support forum of the Apache HTTP Server Project.
See <URL:http://httpd.apache.org/userslist.html> for more info.
To unsubscribe, e-mail: users-unsubscribe@httpd.apache.org
   "   from the digest: users-digest-unsubscribe@httpd.apache.org
For additional commands, e-mail: users-help@httpd.apache.org