You are viewing a plain text version of this content. The canonical link for it is here.
Posted to modperl@perl.apache.org by Andy Grundman <an...@hybridized.org> on 2004/02/27 03:56:04 UTC
[mp2] Help with sendfile and offset value
I am writing a secure download ResponseHandler that sends a file to the
user. I have simple sends working fine with sendfile($filename) but I
also need to handle 206 Partial Content as the files I'm dealing with
are big and may need to be resumed.
I am picking up the HTTP Range header to find the starting byte value
and trying to pass that value as the offset param to sendfile. It
appears to work (see below) for something like the first half of the
file, and then stops working at all, returning immediately from sendfile().
Here's some code.
my $filename = "/path/to/file";
my $size = (stat($filename))[7];
$r->headers_out->set('Content-Length' => $size);
# check for Range header to see if we should start at an offset
# A Range header looks like:
# bytes=15361655-
my $offset = 0;
my $range = $r->headers_in->get('Range');
if ($range) {
$offset = $range;
$offset =~ s/bytes=//;
$offset =~ s/-$//;
$offset = 0 if ($offset =~ /\D/); # set back to 0 if no numbers
}
$r->sendfile($filename, $offset);
I can find very little documention about the offset variable, only this:
$ret = $r->sendfile($filename, $offset, $len);
* arg1: $r (Apache::RequestRec)
* arg2: $filename (Apache::RequestRec)
* arg3: $offset (string)
* arg4: $len (integer)
* ret: $ret (integer)
What gets me is, why is offset listed as a string and not an integer?
Is it not an offset in bytes? My first thought is maybe this is an
offset in hex because it did work for the first part of the file but
stopped later in the fille as if it was trying to seek beyond the end of
the file.
I know the offset value is being transposed to something at least,
because I did a test download using several 206 requests, and the
resluting file had gaps in it where it was resumed instead of being
seamless.
Does anyone know anything about sendfile or offset in mp2? I am running
Apache 2.0.48, mod_perl 1.99_10, with Perl 5.8.2 on Gentoo.
Thanks,
-Andy
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [mp2] Help with sendfile and offset value
Posted by Stas Bekman <st...@stason.org>.
Andy Grundman wrote:
> Geoffrey Young wrote:
>
>>> So, even if I am writing my own content out using sendfile, Apache will
>>> start it at the correct spot when it gets a Range request?
>>
>>
>>
>> yes.
>
>
> Awesome! Thank you so much...this is great. I am testing it by
> streaming an mp3 file and I am able to seek using Winamp to any point in
> the file and it handles it flawlessly! I do see it at work, my code
> gets past sendfile() almost immediately and then Apache takes over
> sending the data from there. And it uses no memory while doing
> so...very nice.
>
>> actually, you don't need to calculate the length of the response anymore
>> either. apache 2.0 has a content-length filter that uses the same
>> mechanism
>> as above (calculating the length of the filter stream after everyone has
>> manipulated the content).
>
>
> If I take out the length calcuclation, no Content-Length header gets
> generated. This makes sense though; what if you had 500MB of data... it
> would take many seconds to buffer that from disk while the client
> waited. Better to simply stat the file and send the proper length header.
Now, if you could summarize this as a pod section (with a simple example) to
be added to 2.0 user docs, that will benefit all users who will have to deal
with the same questions and they won't have to ask the same question here.
The best fit is probably after:
http://perl.apache.org/docs/2.0/user/handlers/http.html#Handling_HEAD_Requests
and title it: Handling Byte Range Requests
Thank you.
__________________________________________________________________
Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
http://stason.org/ mod_perl Guide ---> http://perl.apache.org
mailto:stas@stason.org http://use.perl.org http://apacheweek.com
http://modperlbook.org http://apache.org http://ticketmaster.com
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [mp2] Help with sendfile and offset value
Posted by Geoffrey Young <ge...@modperlcookbook.org>.
> Awesome! Thank you so much...this is great.
sure thing :)
>> actually, you don't need to calculate the length of the response anymore
>> either. apache 2.0 has a content-length filter that uses the same
>> mechanism
>> as above (calculating the length of the filter stream after everyone has
>> manipulated the content).
>
>
> If I take out the length calcuclation, no Content-Length header gets
> generated. This makes sense though; what if you had 500MB of data... it
> would take many seconds to buffer that from disk while the client
> waited. Better to simply stat the file and send the proper length header.
apache makes the decision as to whether you actually need a Content-Length
header. if you're using HTTP/1.1, for instance it will use a chunked
transfer encoding instead, so you would see a Transfer-Encoding header.
anyway, glad you got this sorted out. apache 2.0 is cool, isn't it ;)
--Geoff
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [mp2] Help with sendfile and offset value
Posted by Andy Grundman <an...@hybridized.org>.
Geoffrey Young wrote:
>>So, even if I am writing my own content out using sendfile, Apache will
>>start it at the correct spot when it gets a Range request?
>
>
> yes.
Awesome! Thank you so much...this is great. I am testing it by
streaming an mp3 file and I am able to seek using Winamp to any point in
the file and it handles it flawlessly! I do see it at work, my code
gets past sendfile() almost immediately and then Apache takes over
sending the data from there. And it uses no memory while doing
so...very nice.
> actually, you don't need to calculate the length of the response anymore
> either. apache 2.0 has a content-length filter that uses the same mechanism
> as above (calculating the length of the filter stream after everyone has
> manipulated the content).
If I take out the length calcuclation, no Content-Length header gets
generated. This makes sense though; what if you had 500MB of data... it
would take many seconds to buffer that from disk while the client
waited. Better to simply stat the file and send the proper length header.
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [mp2] Help with sendfile and offset value
Posted by Geoffrey Young <ge...@modperlcookbook.org>.
> So, even if I am writing my own content out using sendfile, Apache will
> start it at the correct spot when it gets a Range request?
yes.
> How does it
> do that...run my output up until the correct byte then start sending it
> out to the client?
no. your output is entered into the output filter stream. you generate it
all every time, but apache intercepts the stream and manipulates it before
it gets to the client. remember, byteserving is a bandwidth saver, not a
server-processing saver :) and if you ever doubted that, trace how many
requests msie makes for the same resource _before_ you actually see it (it
used to be 3, and don't forget the processing doesn't stop just because the
client went away)
mod_deflate works the same way, as does mod_include - they all take the
content from the content handler (a cgi script, your handler, etc) and
post-process it on it's way back down the wire.
> I assumed I had to handle it myself.
>
>>
>>> $r->headers_out->set('Content-Length' => $size);
>>
>>
>>
>> don't do that - use $r->set_content_length()
>
>
> Cool. I actually thought the $len variable of sendfile would handle
> this for me, but it didn't appear to do anything.
actually, you don't need to calculate the length of the response anymore
either. apache 2.0 has a content-length filter that uses the same mechanism
as above (calculating the length of the filter stream after everyone has
manipulated the content).
HTH
--Geoff
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [mp2] Help with sendfile and offset value
Posted by Andy Grundman <an...@hybridized.org>.
Geoffrey Young wrote:
>
> Andy Grundman wrote:
>
>>I am writing a secure download ResponseHandler that sends a file to the
>>user. I have simple sends working fine with sendfile($filename) but I
>>also need to handle 206 Partial Content as the files I'm dealing with
>>are big and may need to be resumed.
>
>
> byteserving in apache 2.0 is handled by the byterange output filter, similar
> to the way it handles Content-Length, chunked encoding, etc, regardless of
> who is generating the content. I'm very surprised you need to deal with
> byteserving on your own - did you issue a Range request and Apache didn't
> respond with a partial response?
So, even if I am writing my own content out using sendfile, Apache will
start it at the correct spot when it gets a Range request? How does it
do that...run my output up until the correct byte then start sending it
out to the client? I assumed I had to handle it myself.
>
>>$r->headers_out->set('Content-Length' => $size);
>
>
> don't do that - use $r->set_content_length()
Cool. I actually thought the $len variable of sendfile would handle
this for me, but it didn't appear to do anything.
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html
Re: [mp2] Help with sendfile and offset value
Posted by Geoffrey Young <ge...@modperlcookbook.org>.
Andy Grundman wrote:
> I am writing a secure download ResponseHandler that sends a file to the
> user. I have simple sends working fine with sendfile($filename) but I
> also need to handle 206 Partial Content as the files I'm dealing with
> are big and may need to be resumed.
byteserving in apache 2.0 is handled by the byterange output filter, similar
to the way it handles Content-Length, chunked encoding, etc, regardless of
who is generating the content. I'm very surprised you need to deal with
byteserving on your own - did you issue a Range request and Apache didn't
respond with a partial response?
> $r->headers_out->set('Content-Length' => $size);
don't do that - use $r->set_content_length()
> * arg3: $offset (string)
that's an apr_off_t, not a string. apr_off_t is essentially an int.
> * arg4: $len (integer)
> * ret: $ret (integer)
>
> What gets me is, why is offset listed as a string and not an integer? Is
> it not an offset in bytes? My first thought is maybe this is an offset
> in hex because it did work for the first part of the file but stopped
> later in the fille as if it was trying to seek beyond the end of the file.
>
> I know the offset value is being transposed to something at least,
> because I did a test download using several 206 requests, and the
> resluting file had gaps in it where it was resumed instead of being
> seamless.
>
> Does anyone know anything about sendfile or offset in mp2? I am running
> Apache 2.0.48, mod_perl 1.99_10, with Perl 5.8.2 on Gentoo.
again, I'm surprised you need to deal with this yourself - the byterange
filter should really take care of everything for you. heck, the old
ap_set_byterange and ap_each_byterange API from 1.3 that allowed you to
handle this yourself isn't even public anymore in 2.0 - ap_each_byterange is
gone entirely and ap_set_byterange is now private to http_protocol.c (and
used only in the byterange filter):
static int ap_set_byterange(request_rec *r)
HTH
--Geoff
--
Report problems: http://perl.apache.org/bugs/
Mail list info: http://perl.apache.org/maillist/modperl.html
List etiquette: http://perl.apache.org/maillist/email-etiquette.html