Doing left-over request processing at startup....

[Mike Meyer <mw...@mired.org>]

I've got a startup design issue with a proxy module for a proprietary
protocol.

Basically, the protocol *requires* that a failed request be reversed -
even if the client doesn't get told the reversal succeeds. If apache
crashes holding an unacknowledged reversal request, then the module
needs to complete the reversal when it starts again. I'd like to do
this ASAP after a restart - which means at some point during
initialization rather than waiting for the first request from a
client.

Is there some sane place during system initialization to hang this? Or
should I just punt, and wait for the first client request to come in?

       Thanks,
       <mike
-- 
Mike Meyer <mw...@mired.org>		http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

Re: Doing left-over request processing at startup....

[Brian McQueen <mc...@gmail.com>]

I do think that the httpd may be un-needed except as a proxy. You know
about Apache's Active MQ and other such queues?   They have a protocol
(not http) which is designed for reliable messaging. Maybe its jms or
stomp?   There might even be a module designed to proxy to active mq.



On 12/29/10, Mike Meyer <mw...@mired.org> wrote:
> On Wed, 29 Dec 2010 08:27:33 -0500
> Jerry Stuckle <je...@smartechhomes.com> wrote:
>
>>
>>
>> On 12/29/2010 7:07 AM, Mike Meyer wrote:
>> > On Wed, 29 Dec 2010 11:00:09 +0200
>> > Issac Goldstand<ma...@beamartyr.net>  wrote:
>> >
>> >> On 29/12/2010 10:52, Mike Meyer wrote:
>> >>> The current code actually works fine - except for this timing
>> >>> issue. If I kill apache in the middle of handling a reversal, and
>> >>> restart it, the first connection will cause the unfinished reversal to
>> >>> be finished. I'm trying to find some way to make this happen at
>> >>> startup, instead of on the first connection.
>> >> Maybe try something simpler?  Write a script which fires at apache
>> >> startup and just makes a dummy request to trigger the queue?  I know
>> >> that's a lot less fun and less elegant ;), but will it get the job
>> >> done?
>> >
>> > Yeah, I'd been thinking about something like that. Start it a bit
>> > after apache, probe the port, connect, count to one, and go away....
>>
>> If you don't need to give feedback to the web user, just place the
>> request in a queue, i.e. in a database.  Then start a cron job every 5
>> minutes (or whenever) to process any work in the queue.  Much simpler.
>
> We do need to give feedback to the web user. In fact, failure to do
> that means the request has failed, and has to be reversed. That we
> need to complete these actions even if apache is rebooted while we're
> trying is the root cause of the problem.
>
>        Thanks,
>        <mike
> --
> Mike Meyer <mw...@mired.org>		http://www.mired.org/consulting.html
> Independent Network/Unix/Perforce consultant, email for more information.
>
> O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
>

-- 
Sent from my mobile device

Make a Small Loan, Make a Big Difference - Check out Kiva.org to Learn How!

Re: Doing left-over request processing at startup....

[Mike Meyer <mw...@mired.org>]

On Wed, 29 Dec 2010 08:27:33 -0500
Jerry Stuckle <je...@smartechhomes.com> wrote:

> 
> 
> On 12/29/2010 7:07 AM, Mike Meyer wrote:
> > On Wed, 29 Dec 2010 11:00:09 +0200
> > Issac Goldstand<ma...@beamartyr.net>  wrote:
> >
> >> On 29/12/2010 10:52, Mike Meyer wrote:
> >>> The current code actually works fine - except for this timing
> >>> issue. If I kill apache in the middle of handling a reversal, and
> >>> restart it, the first connection will cause the unfinished reversal to
> >>> be finished. I'm trying to find some way to make this happen at
> >>> startup, instead of on the first connection.
> >> Maybe try something simpler?  Write a script which fires at apache
> >> startup and just makes a dummy request to trigger the queue?  I know
> >> that's a lot less fun and less elegant ;), but will it get the job done?
> >
> > Yeah, I'd been thinking about something like that. Start it a bit
> > after apache, probe the port, connect, count to one, and go away....
> 
> If you don't need to give feedback to the web user, just place the 
> request in a queue, i.e. in a database.  Then start a cron job every 5 
> minutes (or whenever) to process any work in the queue.  Much simpler.

We do need to give feedback to the web user. In fact, failure to do
that means the request has failed, and has to be reversed. That we
need to complete these actions even if apache is rebooted while we're
trying is the root cause of the problem.

       Thanks,
       <mike
-- 
Mike Meyer <mw...@mired.org>		http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

Re: Doing left-over request processing at startup....

[Jerry Stuckle <je...@smartechhomes.com>]

On 12/29/2010 7:07 AM, Mike Meyer wrote:
> On Wed, 29 Dec 2010 11:00:09 +0200
> Issac Goldstand<ma...@beamartyr.net>  wrote:
>
>> On 29/12/2010 10:52, Mike Meyer wrote:
>>> The current code actually works fine - except for this timing
>>> issue. If I kill apache in the middle of handling a reversal, and
>>> restart it, the first connection will cause the unfinished reversal to
>>> be finished. I'm trying to find some way to make this happen at
>>> startup, instead of on the first connection.
>> Maybe try something simpler?  Write a script which fires at apache
>> startup and just makes a dummy request to trigger the queue?  I know
>> that's a lot less fun and less elegant ;), but will it get the job done?
>
> Yeah, I'd been thinking about something like that. Start it a bit
> after apache, probe the port, connect, count to one, and go away....
>
>        Thanks,
>        <mike

If you don't need to give feedback to the web user, just place the 
request in a queue, i.e. in a database.  Then start a cron job every 5 
minutes (or whenever) to process any work in the queue.  Much simpler.

Re: Doing left-over request processing at startup....

[Mike Meyer <mw...@mired.org>]

On Wed, 29 Dec 2010 11:00:09 +0200
Issac Goldstand <ma...@beamartyr.net> wrote:

> On 29/12/2010 10:52, Mike Meyer wrote:
> > The current code actually works fine - except for this timing
> > issue. If I kill apache in the middle of handling a reversal, and
> > restart it, the first connection will cause the unfinished reversal to
> > be finished. I'm trying to find some way to make this happen at
> > startup, instead of on the first connection.
> Maybe try something simpler?  Write a script which fires at apache
> startup and just makes a dummy request to trigger the queue?  I know
> that's a lot less fun and less elegant ;), but will it get the job done?

Yeah, I'd been thinking about something like that. Start it a bit
after apache, probe the port, connect, count to one, and go away....

      Thanks,
      <mike
-- 
Mike Meyer <mw...@mired.org>		http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

Re: Doing left-over request processing at startup....

[Issac Goldstand <ma...@beamartyr.net>]

On 29/12/2010 10:52, Mike Meyer wrote:
>
> The current code actually works fine - except for this timing
> issue. If I kill apache in the middle of handling a reversal, and
> restart it, the first connection will cause the unfinished reversal to
> be finished. I'm trying to find some way to make this happen at
> startup, instead of on the first connection.
>
>    <mike

Maybe try something simpler?  Write a script which fires at apache
startup and just makes a dummy request to trigger the queue?  I know
that's a lot less fun and less elegant ;), but will it get the job done?

  Issac

Re: Doing left-over request processing at startup....

[Mike Meyer <mw...@mired.org>]

On Tue, 28 Dec 2010 12:37:39 -0800
Brian McQueen <mc...@gmail.com> wrote:

> What I was imaging is all requests going through that same queue and
> some daemon works the queue all the time.  The httpd is one interface
> to the queue.  So there is no such hook required. Just start up the
> queue drainer and start the httpd. Spikes won't take down the site.
> Its a good design.

How do you get the queue to handle the normal case (I'm dealing with
failure cases here), where the client sends a request to the proxy,
which forwards it to the back end, gets the response, and then sends
that back to the client and gets an acknowledgment? If the queue can
talk to the client, then the apache proxy isn't doing anything useful,
is it? If the queue can't talk to the client, then handing it a
request means the request has to fail, so it should only get involved
in failed requests.

Which once again brings us back to the original question: some apache
process died in the middle of handling a request, so it needs to be
handed to the queue to reverse, and this gets discovered at apache
startup time. I can certainly arrange the system to start the queue
handler before apache boots, but I still need a place in the startup
processing after checking the state file to hand things off to the
queue.

Come to think of it, I'm not sure this problem can be avoided if
there's an apache module doing the work of insuring success or
reversal at all: if the module dies before a request is finished (one
way or the other) and apache is shut down before it's dealt with, then
it's going to get found at startup time and we're back to that
question. But the point of the module is to do the success/reversal
handling.

The current code actually works fine - except for this timing
issue. If I kill apache in the middle of handling a reversal, and
restart it, the first connection will cause the unfinished reversal to
be finished. I'm trying to find some way to make this happen at
startup, instead of on the first connection.

   <mike
-- 
Mike Meyer <mw...@mired.org>		http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

Re: Doing left-over request processing at startup....

[Brian McQueen <mc...@gmail.com>]

What I was imaging is all requests going through that same queue and
some daemon works the queue all the time.  The httpd is one interface
to the queue.  So there is no such hook required. Just start up the
queue drainer and start the httpd. Spikes won't take down the site.
Its a good design.

On 12/28/10, Mike Meyer <mw...@mired.org> wrote:
> On Tue, 28 Dec 2010 09:07:38 -0800
> Brian McQueen <mc...@gmail.com> wrote:
>
>> Put it in a queue and have a separate process work the queue. Don't
>> try so hard to cram it into the web server.
>
> Given that the queue runner only needs to run after apache shuts down
> leaving unfinished business in the queue, and processing that has to
> be coordinated with the module startup processing, that changes the
> design but leaves the same fundamental question:
>
> Is there a good place in the apache startup sequence to start the
> queue processing?
>
> 	<mike
>
>
>
> --
> Mike Meyer <mw...@mired.org>		http://www.mired.org/consulting.html
> Independent Network/Unix/Perforce consultant, email for more information.
>
> O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
>

-- 
Sent from my mobile device

Make a Small Loan, Make a Big Difference - Check out Kiva.org to Learn How!

Re: Doing left-over request processing at startup....

[Mike Meyer <mw...@mired.org>]

On Tue, 28 Dec 2010 09:07:38 -0800
Brian McQueen <mc...@gmail.com> wrote:

> Put it in a queue and have a separate process work the queue. Don't
> try so hard to cram it into the web server.

Given that the queue runner only needs to run after apache shuts down
leaving unfinished business in the queue, and processing that has to
be coordinated with the module startup processing, that changes the
design but leaves the same fundamental question:

Is there a good place in the apache startup sequence to start the
queue processing?

	<mike



-- 
Mike Meyer <mw...@mired.org>		http://www.mired.org/consulting.html
Independent Network/Unix/Perforce consultant, email for more information.

O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

Re: Doing left-over request processing at startup....

[Brian McQueen <mc...@gmail.com>]

Put it in a queue and have a separate process work the queue. Don't
try so hard to cram it into the web server.



On 12/28/10, Mike Meyer <mw...@mired.org> wrote:
> I've got a startup design issue with a proxy module for a proprietary
> protocol.
>
> Basically, the protocol *requires* that a failed request be reversed -
> even if the client doesn't get told the reversal succeeds. If apache
> crashes holding an unacknowledged reversal request, then the module
> needs to complete the reversal when it starts again. I'd like to do
> this ASAP after a restart - which means at some point during
> initialization rather than waiting for the first request from a
> client.
>
> Is there some sane place during system initialization to hang this? Or
> should I just punt, and wait for the first client request to come in?
>
>        Thanks,
>        <mike
> --
> Mike Meyer <mw...@mired.org>		http://www.mired.org/consulting.html
> Independent Network/Unix/Perforce consultant, email for more information.
>
> O< ascii ribbon campaign - stop html mail - www.asciiribbon.org
>

-- 
Sent from my mobile device

Make a Small Loan, Make a Big Difference - Check out Kiva.org to Learn How!