Sending large file as attachment to clients

[Subra A Narayanan <as...@gmail.com>]

Hello folks,

I have an apache module which receives request for data from clients, reads
binary data from a datastore, loads the data in to memory and then sends out
the response with the data in the request body. Here is the code snippet:

 	ap_set_content_type(request, "application/octet-stream");

        //Open db
        //Read BLOB data in to objBuffer
        //bufferLen = length(objBuffer)

        ap_rwrite(objBuffer, bufferLen, request);
	ap_set_content_length(request, bufferLen);

	*return* HTTP_OK;


The above code works as desired. But as you can see, this approach reads in
the entire file in to memory and then write it out to the request body. This
method works fine for small amounts of data but for larger data blocks (say
a 1GB file), the server will run out of memory pretty soon.

I was wondering if there is a way to read and transmit chunks of data to the
client. The module will read, lets say 10k of data at a time, transmit it,
read the next chunk, transmit it.....until the end is reached.

Any help would be greatly appreciated!!

Thanks,
Subra

Re: Sending large file as attachment to clients

[Joe Lewis <jo...@joe-lewis.com>]

Subra A Narayanan wrote:
> Thanks for ur reply Samuel!
>
> Maybe you can read part of the blob with a function like MySQL's SUBSTR.
>   
>
> I can easily read small chunks of data from my datastore. I guess the
> problem that I am trying to solve here - how to send chunks of data as a
> HTTP response without having to read in and store a large file/object in
> memory.
>
>
> Subra
>   

Samuel was aiming the right direction.  If you can read in pieces of the 
data to send, you should be able to loop over that reading a chunk of 
data, then sending it, then reading a chunk the same size (same block, 
for example), and continuing until the data has been sent.  Try that to 
see if you obtain a different result.

Joe

Re: Sending large file as attachment to clients

[Subra A Narayanan <as...@gmail.com>]

Thanks for ur reply Samuel!

Maybe you can read part of the blob with a function like MySQL's SUBSTR.
>

I can easily read small chunks of data from my datastore. I guess the
problem that I am trying to solve here - how to send chunks of data as a
HTTP response without having to read in and store a large file/object in
memory.


Subra

Re: Sending large file as attachment to clients

[Samuel Gómez <e6...@est.fib.upc.edu>]

Hello Subra.

Maybe you can read part of the blob with a function like MySQL's SUBSTR.

I've never tested that. When there are large files to store, I prefer to 
store it as a normal file in a directory, and record its path on the 
database (e.g. in a varchar field).

Samuel



Subra A Narayanan wrote:
> Hello folks,
>
> I have an apache module which receives request for data from clients, reads
> binary data from a datastore, loads the data in to memory and then sends out
> the response with the data in the request body. Here is the code snippet:
>
>  	ap_set_content_type(request, "application/octet-stream");
>
>         //Open db
>         //Read BLOB data in to objBuffer
>         //bufferLen = length(objBuffer)
>
>         ap_rwrite(objBuffer, bufferLen, request);
> 	ap_set_content_length(request, bufferLen);
>
> 	*return* HTTP_OK;
>
>
> The above code works as desired. But as you can see, this approach reads in
> the entire file in to memory and then write it out to the request body. This
> method works fine for small amounts of data but for larger data blocks (say
> a 1GB file), the server will run out of memory pretty soon.
>
> I was wondering if there is a way to read and transmit chunks of data to the
> client. The module will read, lets say 10k of data at a time, transmit it,
> read the next chunk, transmit it.....until the end is reached.
>
> Any help would be greatly appreciated!!
>
> Thanks,
> Subra
>
>   

Re: Sending large file as attachment to clients

[Joe Lewis <jo...@joe-lewis.com>]

Subra A Narayanan wrote:
> Thanks for the replies guys. I will try ur suggestions once I get to work
> today. Just a few more questions:
>
>   
>> Check out the various filter/bucket brigade examples. This may do just
>> what you want. I've used it to 'gate' the final sending for things ike
>> throttling and keeping buggy telco bearer boxes happy.
>>     
>
>
> Dirk: Do you remember any filter/bucket off the top of ur head that I can
> look at?
>   

I would suggest perusing the source code for Nick Kew's mod_csv - a 
module that takes the contents of a .csv file and generates an HTML 
table.  It utilizes bucket brigade creation.

http://apache.webthing.com/mod_csv/

You should find a lot of useful bits of code in there.  I'd still 
suggest the simple loop to see if that works better.


>
>   
>> Samuel was aiming the right direction.  If you can read in pieces of the
>> data to send, you should be able to loop over that reading a chunk of
>> data, then sending it, then reading a chunk the same size (same block,
>> for example), and continuing until the data has been sent.  Try that to
>> see if you obtain a different result.
>>
>>     
>
> Joe: Do you mean this:
>
> while(!EOF)
> {
>    Read chunk of data;
>    ap_rwrite(objBuffer, bufferLen, request);
> }
>
> Can I use the ap_rwrite function for sending chunks of data too? Or should I
> be using some other function in the API for this?
>
> I will test out these suggestions and report my findings asap.
>
> Thanks!
>
>   
That is exactly what I meant.  The ap_rwrite is a function I recently 
had to use for a file synchronization plugin for Apache, and it worked 
quite well (just my opinion, though).  Test both to see which ways work 
faster, and which ways are just easier to understand.

Joe

Re: Sending large file as attachment to clients

[Subra A Narayanan <as...@gmail.com>]

Thanks for the replies guys. I will try ur suggestions once I get to work
today. Just a few more questions:

>
>
> Check out the various filter/bucket brigade examples. This may do just
> what you want. I've used it to 'gate' the final sending for things ike
> throttling and keeping buggy telco bearer boxes happy.


Dirk: Do you remember any filter/bucket off the top of ur head that I can
look at?


>
> Samuel was aiming the right direction.  If you can read in pieces of the
> data to send, you should be able to loop over that reading a chunk of
> data, then sending it, then reading a chunk the same size (same block,
> for example), and continuing until the data has been sent.  Try that to
> see if you obtain a different result.
>

Joe: Do you mean this:

while(!EOF)
{
   Read chunk of data;
   ap_rwrite(objBuffer, bufferLen, request);
}

Can I use the ap_rwrite function for sending chunks of data too? Or should I
be using some other function in the API for this?

I will test out these suggestions and report my findings asap.

Thanks!

Re: Sending large file as attachment to clients

[Dirk-Willem van Gulik <di...@webweaving.org>]

Subra A Narayanan wrote:

> I was wondering if there is a way to read and transmit chunks of data to the
> client. The module will read, lets say 10k of data at a time, transmit it,
> read the next chunk, transmit it.....until the end is reached.

Check out the various filter/bucket brigade examples. This may do just 
what you want. I've used it to 'gate' the final sending for things ike 
throttling and keeping buggy telco bearer boxes happy.

Dw