You are viewing a plain text version of this content. The canonical link for it is here.
Posted to apreq-dev@httpd.apache.org by "Paul G. Weiss" <pa...@weiss.name> on 2004/02/24 23:47:48 UTC
Problem with posting lots of data
I have a weird problem that shows up when posting lots of data through
libapreq2. I've tried this both with libapreq2-2.02-dev and the current
CVS version.
To illustrate consider the handler below. You type an integer n in a text
box and hit "Build" and it builds a form that has hidden input fields:
<input type=hidden name="name.1" value="value.1">
<input type=hidden name="name.2" value="value.2">
...
<input type=hidden name="name.n" value="value.n">
then you hit the "Check" button and the script tells you how many
parameters had names of the form /^name\.\d+$/.
If all is well the number will be same as the number you built.
But for n>442 the numbers don't match.
That isn't all. It works fine through http (as long as n < FIELD_LIMIT),
but fails through https. So you would think that the problem is with
mod_ssl/openssl. However, if you use CGI instead of Apache::Request (see
below - comment out the Apache::Request line and uncomment the CGI line),
it works fine, even through https! This tells me that it isn't
mod_ssl/openssl alone, but in the interaction between it and libapreq2,
perhaps some difference in how the data is buffered?
Can anyone confirm this?
I'm using
Apache 2.0.48
mod_perl/1.99_12
Perl/v5.8.1
mod_ssl/2.0.48
OpenSSL/0.9.7a
(the openssl is the Fedora RPM).
-Paul
The handler--
package Test::Worker;
use Apache::Const qw(OK DONE REDIRECT DECLINED HTTP_NOT_MODIFIED);
use CGI;
sub handler
{
my $r = shift;
my $req = Apache::Request->new($r, POST_MAX => 10_000_000, FIELD_LIMIT
=> 20_000);
# my $req = new CGI;
my @inputs = $req->param;
my $name_count = scalar(grep(/^name\.\d+$/, @inputs));
$r->content_type('text/html');
$r->print(<<THIS);
<HTML>
<BODY>
<form method="post">
THIS
my $n = $req->param('n');
if ($n > 0) {
$r->print(<<THIS);
Building for $n <input type=submit name=check value="Check">
THIS
while ($n > 0) {
$r->print(qq{<input type=hidden name="name.$n"
value="value.$n">\n});
$n -= 1;
}
} else {
$r->print("found $name_count hidden fields<br>\n");
$r->print(<<THIS);
<input type=text name=n value="$n"> <input type=submit name=build
value="Build">
THIS
}
$r->print("<BODY>\n<HTML>\n");
return OK;
}
1;
[
Re: Problem with posting lots of data
Posted by "Paul G. Weiss" <pa...@weiss.name>.
Here are the results of the experiments using multipart/form-data and use
Apache::Request together with CGI.
First, using multipart/form-data. Apache::Request script gets it right for
n <= 77. Using CGI, it is correct for n <= 635. I don't know what the
problem is for n > 635 for CGI, but this is the apreq-dev list.
Incidentally there is a little weirdness here with Apache::Request even
w/o using ssl. On a fresh server restart, it gets it wrong for high n,
say n=400. However, after a few requests, it gets it right for high n.
Now combining Apache::Request and CGI. First note that there are two way
to do this, and they have different results:
The first way is to do:
my $req = Apache::Request->new($r, POST_MAX => 10_000_000, FIELD_LIMIT
=> 20_000);
my $cgi = CGI->new;
my $cgi_name_count = scalar(grep(/^name\.\d+$/, $cgi->param));
my $name_count = scalar(grep(/^name\.\d+$/, $req->param));
i.e. do the CGI read before the Apache::Request one. Here, both
Apache::Request and CGI report correct results for n<=1000. After that it
starts screwing up. Once it screws up a few times, then it doesn't even
work for n=1000. (I ran with -X so as to have a single server).
The second is to do:
my $req = Apache::Request->new($r, POST_MAX => 10_000_000, FIELD_LIMIT
=> 20_000);
my $name_count = scalar(grep(/^name\.\d+$/, $req->param));
my $cgi = CGI->new;
my $cgi_name_count = scalar(grep(/^name\.\d+$/, $cgi->param));
i.e. do the Apache::Request read before the CGI one. Here, CGI is no
help. It still fails for n>442. When it fails, sometimes CGI and
Apache::Request agree on the count, sometimes they don't.
On Tue, 24 Feb 2004 19:57:23 -0500, Paul G. Weiss
<pg...@tremontstreetshul.org> wrote:
> Joe Schaefer writes:
>> "Paul G. Weiss" <pa...@weiss.name> writes:
>>> But for n>442 the numbers don't match.
>> That is somewhat expected, since the parsers will currently give up
>> when the number of fields exceeds 200 (the actual
>> number depends on the amount of data still inside the input brigade at
>> that particular moment.)
>>> That isn't all. It works fine through http
>> By "it works fine", you mean the script below works for n < 200?
>
> By "it works fine" I mean that it works even for n <= 10_000 (which is
> what I set FIELD_LIMIT to) for http, and fails for n >= 443 for https.
> For CGI, it works for arbitrary n.
>>> (as long as n < FIELD_LIMIT),
>> FIELD_LIMIT is going away, along with the max_fields and read_ahead
>> attributes of apreq_config_t.
>
> So 200 fields will be a hard limit then? But we will still be allowed
> to have many values per field, right?
>>> but fails through https. So you would think that the problem is with
>>> mod_ssl/openssl. However, if you use CGI instead of Apache::Request
>>> (see below - comment out the Apache::Request line and uncomment the
>>> CGI line), it works fine, even through https!
>> I'm curious- what happens if both of them are uncommented?
>> (In mp2, Apache::Request and CGI.pm can both be used simultaneously
>> within the same request).
>
> I'll try tonight, but wouldn't it depend upon the order? Won't CGI
> consume the posted data, making it unavailable for Apache::Request. And
> similarly the other way around?
>> What about using "multipart/form-data" for the encoding, instead
>> of the default ("application/x-www-form-urlencoded")? Does that
>> change anything?
>
> Again, I'll try tonight when I have access to the machine.
>>> This tells me that it isn't mod_ssl/openssl alone, but in the
>>> interaction between it and libapreq2, perhaps some difference in how
>>> the data is buffered? Can anyone confirm this? I'm using
>>> Apache 2.0.48
>>> mod_perl/1.99_12
>>> Perl/v5.8.1
>>> mod_ssl/2.0.48
>>> OpenSSL/0.9.7a
>> Not easily- at the moment my current build box looks nothing like
>> yours (no SSL in particular). Anyone else able to take a crack? --
>> Joe Schaefer
>
Re: Problem with posting lots of data
Posted by Joe Schaefer <jo...@sunstarsys.com>.
"Paul G. Weiss" <pa...@weiss.name> writes:
> But for n>442 the numbers don't match.
That is somewhat expected, since the parsers will currently
give up when the number of fields exceeds 200 (the actual
number depends on the amount of data still inside the input
brigade at that particular moment.)
> That isn't all. It works fine through http
By "it works fine", you mean the script below works for n < 200?
> (as long as n < FIELD_LIMIT),
FIELD_LIMIT is going away, along with the max_fields and read_ahead
attributes of apreq_config_t.
> but fails through https. So you would think that the problem is with
> mod_ssl/openssl. However, if you use CGI instead of Apache::Request
> (see below - comment out the Apache::Request line and uncomment the
> CGI line), it works fine, even through https!
I'm curious- what happens if both of them are uncommented?
(In mp2, Apache::Request and CGI.pm can both be used simultaneously
within the same request).
What about using "multipart/form-data" for the encoding, instead
of the default ("application/x-www-form-urlencoded")? Does that
change anything?
> This tells me that it isn't mod_ssl/openssl alone, but in the
> interaction between it and libapreq2, perhaps some difference in how
> the data is buffered?
>
> Can anyone confirm this?
>
> I'm using
> Apache 2.0.48
> mod_perl/1.99_12
> Perl/v5.8.1
> mod_ssl/2.0.48
> OpenSSL/0.9.7a
Not easily- at the moment my current build box looks nothing like
yours (no SSL in particular). Anyone else able to take a crack?
--
Joe Schaefer
RE: Problem with posting lots of data
Posted by "Eric J. Hansen" <eh...@worldmachine.com>.
I've tried this on my dev machine, and found exactly the same problem. Printing
the decoded values reveals that after parameter 442, the list "wraps",
and the param list starts again at the first element. i.e., value.441,
value.442, value.1, value.2, ...
FYI, here's my platform:
Apache 2.0.48
mod_perl/1.99_11
Perl/v5.8.2
mod_ssl/2.0.48
OpenSSL/0.9.7c
libapreq2-2.02-dev
Just like Paul, I'm only seeing this problem under mod_ssl. Under non-SSL I was
able to pass many 1,000's of parameters with nary a hitch.
-Eric
> -----Original Message-----
> From: Paul G. Weiss [mailto:paul@weiss.name]
> Sent: Tuesday, February 24, 2004 5:48 PM
> To: apreq-dev@httpd.apache.org
> Subject: Problem with posting lots of data
>
>
> I have a weird problem that shows up when posting lots of
> data through
> libapreq2. I've tried this both with libapreq2-2.02-dev and
[...]
> But for n>442 the numbers don't match.
>
> That isn't all. It works fine through http (as long as n <
> FIELD_LIMIT),
> but fails through https. So you would think that the problem
> is with
> mod_ssl/openssl. However, if you use CGI instead of
> Apache::Request (see
> below - comment out the Apache::Request line and uncomment
> the CGI line),
> it works fine, even through https! This tells me that it isn't
> mod_ssl/openssl alone, but in the interaction between it and
> libapreq2,
> perhaps some difference in how the data is buffered?
>
> Can anyone confirm this?
> I'm using
> Apache 2.0.48
> mod_perl/1.99_12
> Perl/v5.8.1
> mod_ssl/2.0.48
> OpenSSL/0.9.7a
>