how to make sub-requests?

[Parin Shah <pa...@gmail.com>]

Hi All,

I am currently working on mod-cache-requester. This module stores
uri's of all the pages those are served from cache. and it re-request
all popular pages those are soon-to-expire from cache so that such
pages are not removed from the cache.

To implement that, mod-cache-requester should be able to make a sub
request for the pages those are about to expire.

this could be achieved in following ways, but each has its own
benefits/limitations.

- using make_sub_request, ap_sub_req_method_uri methods. these
functions create a new request_req for a sub request and they take
current request_req as one of the argument. but mod-cache-requester
may not have current request available.

we can store the original request_req which was used when the page was
served from cache, and then use it as a parameter to the above method.
Is this approach is fine?

- second approach would be to create seperate socket connection and
make request on this connection. Still I personally believe this is
not a very elegant solution.

- other than these methods, please let me know other possible methods
you are aware of.

waiting for some help,
Thanks.
Parin.

Re: how to make sub-requests?

[Graham Leggett <mi...@sharp.fm>]

Parin Shah said:

> - I played with it to make it work without conn_rec, but it doesnt
> soudn very clean to me . the main reason is request_rec itself needs
> conn_rec and also other conn_rec variables like lookup defaults. so
> even if we  could make it work (which I am not 100% sure that it would
> work) it wouldn't be an elegant solution.

Point taken.

> what I meant was that  I might find similar functionality in mod_proxy
> which does same thing. I am not planning to reuse it. but would use it
> as a reference to create requests for mod-c-requester.

mod_proxy does the whole nine yards - it creates a real connection, then
creates requests off that connection using the same core methods you
tried. You're unlikely to find any shortcuts or ways of creating lean
requests here. :(

>> Rather tack yourself onto mod_cache, so that after a request is
>> complete, it can fire off a pass of the cache freshening code while a
>> connection_req is still available to create fake requests from.
> This is a very good alternative solution. I would proceed in this
> direction if I cannot find any other way.

I am starting to think this is probably the only practical way of doing
this, unless we can come up with a dummy request_req creator that leaves
the non essential (for this case) fields as NULL.

What may be useful is to teach httpd how to handle the case where the
connection is missing, and have it fail gracefully/intelligently in this
case, however this is a big job that is likely to slow you down from
getting the first releas out the door.

Regards,
Graham
--

Re: how to make sub-requests?

[Parin Shah <pa...@gmail.com>]

> What I meant was that you modify the ap_read_request() to not crash when
> NULL is passed to it.
> As far as I am aware, the request_req only needs certain fields copied
> out of connection_req, not all of which are required.
> 

- I played with it to make it work without conn_rec, but it doesnt
soudn very clean to me . the main reason is request_rec itself needs
conn_rec and also other conn_rec variables like lookup defaults. so
even if we  could make it work (which I am not 100% sure that it would
work) it wouldn't be an elegant solution.

> I wouln't use proxy, as proxy has nothing to do with cache (you may
> build proxy without cache if you like, or cache without proxy).

what I meant was that  I might find similar functionality in mod_proxy
which does same thing. I am not planning to reuse it. but would use it
as a reference to create requests for mod-c-requester.

> Rather tack yourself onto mod_cache, so that after a request is
> complete, it can fire off a pass of the cache freshening code while a
> connection_req is still available to create fake requests from.
This is a very good alternative solution. I would proceed in this
direction if I cannot find any other way.

-Parin.

Re: how to make sub-requests?

[Graham Leggett <mi...@sharp.fm>]

Parin Shah wrote:

>>Would there be value in teaching ap_read_request() how to not break if
>>you called ap_read_request(NULL)?
>>
> 
> Well, I couldnt find a way to do this. and with NULL value it crashes.

What I meant was that you modify the ap_read_request() to not crash when 
NULL is passed to it.

>>Alternatively ap_read_request() could be broken up into
>>ap_read_request() and ap_create_request(), which would do just the
>>request_req creation part.

> request_rec creation part needs conn_rec, and creating conn_req is
> again not straight forward  so I dont think ap_read_request is gonna
> help us in this case.

As far as I am aware, the request_req only needs certain fields copied 
out of connection_req, not all of which are required.

> I am now going through mod-proxy as chipig and iholsman suggested. It
> seems that allows us to create fake requests. any further help in that
> (or any other) direction would be really great.

I wouln't use proxy, as proxy has nothing to do with cache (you may 
build proxy without cache if you like, or cache without proxy).

Rather tack yourself onto mod_cache, so that after a request is 
complete, it can fire off a pass of the cache freshening code while a 
connection_req is still available to create fake requests from.

Regards,
Graham
--

Re: how to make sub-requests?

[Parin Shah <pa...@gmail.com>]

> Would there be value in teaching ap_read_request() how to not break if
> you called ap_read_request(NULL)?
> 
Well, I couldnt find a way to do this. and with NULL value it crashes.

> Alternatively ap_read_request() could be broken up into
> ap_read_request() and ap_create_request(), which would do just the
> request_req creation part.
> 
request_rec creation part needs conn_rec, and creating conn_req is
again not straight forward  so I dont think ap_read_request is gonna
help us in this case.

> This seems to be the first time a request_req has ever been needed where
> a connection was not available.

I am now going through mod-proxy as chipig and iholsman suggested. It
seems that allows us to create fake requests. any further help in that
(or any other) direction would be really great.

Thanks,
Parin.
> 
> Regards,
> Graham
> --
>

Re: how to make sub-requests?

[Graham Leggett <mi...@sharp.fm>]

Parin Shah wrote:

> There is a way to create a request rec using function:
> ap_read_request(conn_rec *).

Would there be value in teaching ap_read_request() how to not break if 
you called ap_read_request(NULL)?

Alternatively ap_read_request() could be broken up into 
ap_read_request() and ap_create_request(), which would do just the 
request_req creation part.

This seems to be the first time a request_req has ever been needed where 
a connection was not available.

Regards,
Graham
--

Re: how to make sub-requests?

[Parin Shah <pa...@gmail.com>]

Thanks for your replies.

There is a way to create a request rec using function:
ap_read_request(conn_rec *).

now conn_rec could be created using ap_run_create_connection function
which takes pool, socket, bucket and some other arguments. I waasnt
sure how to get socket information  as we cannot use the connection of
any current request_rec for that.

It would be really helpful if you can suggest me some pointers for this.

Thanks,
Parin.

On 8/8/05, William A. Rowe, Jr. <wr...@rowe-clan.net> wrote:
> At 05:15 AM 8/8/2005, Graham Leggett wrote:
> >Parin Shah said:
> >
> >> we can store the original request_req which was used when the page was
> >> served from cache, and then use it as a parameter to the above method.
> >> Is this approach is fine?
> >
> >This isn't very clean, a request_req is just a structure, they should be
> >relatively simple to make. Look for the code within the core that brings
> >the request_req into existence for the first time, you should be able to
> >see how it gets created.
> 
> Actually I'd suggest you use the core to create that request_rec,
> rather than chasing upgrades for all the things you need to add
> custom when the structure changes.  The request_rec can and does
> grow on mmn minor bumps, so that's an extra reason to avoid ever
> allocating one yourself.  You will have very minimal ABI compat
> if you allocate your own, and should probably emit a warning on
> startup if you see the mmn minor has moved.
> 
> Bill
> 
> 
>

Re: how to make sub-requests?

["William A. Rowe, Jr." <wr...@rowe-clan.net>]

At 05:15 AM 8/8/2005, Graham Leggett wrote:
>Parin Shah said:
>
>> we can store the original request_req which was used when the page was
>> served from cache, and then use it as a parameter to the above method.
>> Is this approach is fine?
>
>This isn't very clean, a request_req is just a structure, they should be
>relatively simple to make. Look for the code within the core that brings
>the request_req into existence for the first time, you should be able to
>see how it gets created.

Actually I'd suggest you use the core to create that request_rec,
rather than chasing upgrades for all the things you need to add
custom when the structure changes.  The request_rec can and does
grow on mmn minor bumps, so that's an extra reason to avoid ever
allocating one yourself.  You will have very minimal ABI compat
if you allocate your own, and should probably emit a warning on
startup if you see the mmn minor has moved.

Bill

Re: how to make sub-requests?

[Graham Leggett <mi...@sharp.fm>]

Parin Shah said:

> - using make_sub_request, ap_sub_req_method_uri methods. these
> functions create a new request_req for a sub request and they take
> current request_req as one of the argument. but mod-cache-requester
> may not have current request available.

I suspect the best idea here is to create a bare bones request_req on
demand, and then use this same request_req for all the subrequests to be
fetched in that batch.

> we can store the original request_req which was used when the page was
> served from cache, and then use it as a parameter to the above method.
> Is this approach is fine?

This isn't very clean, a request_req is just a structure, they should be
relatively simple to make. Look for the code within the core that brings
the request_req into existence for the first time, you should be able to
see how it gets created.

Regards,
Graham
--