Timeout with $r->read?

[Ewan Edwards <e...@skwayr.com>]

Hi list,

I'm using Apache 2.0.59, mod_perl 2.0.3.

I'm preparing to use Akamai's Edge service to collect data, and the
service provides data aggregated up to a certain size or timeframe in a
slightly odd format: it's a POST request where the body contains one
line per request of an Akamai-hosted resource in
application/x-www-form-urlencoded form. Since the individual requests
are concatenated with newlines, it is not technically
application/x-www-form-urlencoded; libapreq2 and CGI.pm cannot sensibly
parse this data.

So, I've create a response handler that uses $r->read to read the POST
body, separate out the lines and insert the data into a database. For
small POST bodies this works well (800 r/s or so on my test hardware).

For larger post bodies, roughly anything bigger than about 32k, Apache
appears to stall. I've cut the handler down to virtually nothing and the
problem persists; here's the simplified version:

package AkamaiPosts;
use strict;
use warnings;

use Apache2::Const -compile => qw(OK DECLINED);

sub handler {
    my $r = shift;

    return Apache2::Const::DECLINED unless $r->method eq 'POST';

    my ($buf, $data);
    while ($r->read($buf, 65535)) {
        $data .= $buf;
    }
    warn "INFO: received post with ". length($data) ." bytes";
    $r->content_type('text/html');
    $r->print("Success!");
    return Apache2::Const::OK;
}

1;

Reading the data appears to work just fine; regardless of the size of
the POST body, the INFO line immediately appears in the error log and
shows exactly the right size. Also, the access_log records the POST
request with a 200 response (8 bytes served due to 'Success!').

While benchmarking with ab, when the POST body is sufficiently large and
the stall occurs, only the first requests (for any level of concurrency;
I've tried 1 through 10) generate new log entries, subsequent requests
timeout, no new INFO lines are recorded but the access_log contains 500
errors for every concurrent request in flight at the time.

When I run Apache in single process mode and strace it, Apache appears
to get stuck calling poll().

Suggestions would be greatly appreciated.

-- 
 // 2   ______________________________________________________________ 
//     /                                                              \
\\/ /  |  Winter's frozen spam,                                       |
 \\/   |  Delicate jelly of meat,                                     |
       |  Router eat it all.                                          |
       \_________________________________  ___________________________/
        Ewan Edwards {e^2}, e@skwayr.com |/

Re: Timeout with $r->read?

[Ewan Edwards <e...@skwayr.com>]

On Wed, Mar 21, 2007 at 06:02:27PM -0800, Ewan Edwards wrote:
> 
> So, I've create a response handler that uses $r->read to read the POST
> body, separate out the lines and insert the data into a database. For
> small POST bodies this works well (800 r/s or so on my test hardware).
> 
> For larger post bodies, roughly anything bigger than about 32k, Apache
> appears to stall. I've cut the handler down to virtually nothing and the
> problem persists; here's the simplified version:

I know it's rather bad form to reply to one's self, however, I wanted to
follow-up on this posting by saying that it has turned out to not be an
Apache/mod_perl problem at all. ab seems to be rather broken when
sending POST requests of any substantial size.

I had much greater success benchmarking with httperf, although it has a
compile time limit of 10,000 bytes per session event line. I rebuilt
httperf with a much greater line length limit, and our desired benchmark
behaviour completed successfully.

I apologize for wasting bandwidth.

-- 
 // 2   ______________________________________________________________ 
//     /                                                              \
\\/ /  | Will you loan me $20.00 and only give me ten of it? That     |
 \\/   | way, you will owe me ten, and I'll owe you ten, and          |
       | we'll be even!                                               |
       \_________________________________  ___________________________/
        Ewan Edwards {e^2}, e@skwayr.com |/