Basic queue concept

[Joel Poloney <jp...@gmail.com>]

List,

I'm new to message queues, but I generally understand the concept. I've been
playing with ActiveMQ this afternoon using the PHP Stomp client to make
requests. I noticed that in the Stomp client you don't have access to any
given queue size, whereas in the JMS API I have access to these numbers. So,
assuming I stick with the Stomp API, I can think of 2 basic ways that I
could implement a consumer:

1. I have a consumer in a while(1) { //consume } fashion. That would
basically run forever. As I understand it, this is the way most web servers
work (at the core, core level). In this model, I would have to make sure
that the consumer was always running (perhaps do 3-way redundancy and have 3
consumers doing this). My concern with this model is that if I implement
this in PHP, I'm afraid of memory issues and so on. I don't necessarily
trust a PHP script to run in a while(1) fashion forever.

2. I could somehow get the queue size and batch process these. Basically,
get the queue size (say it was 200 entries), consume 200 messages and then
the PHP script would end. Then you would have this script in a cronjob that
runs every x amount of time.

Unfortunately, I have to implement this in PHP. So, I'm not really sure
which way would be the proper way to go about doing this. If any one has any
thoughts or comments, it would be greatly appreciated.

Thanks,

-- Joel

Re: Basic queue concept

[Joel Poloney <jp...@gmail.com>]

On Thu, Apr 3, 2008 at 12:19 AM, Arjen van der Meijden <
acmmailing@tweakers.net> wrote:

> My consumer just consumes a message, processes it and then goes on waiting
> for another one. So if that message happens to be a 'kill'-message, it dies
> (if the message was sent after the consumer started).
>
> When I shutdown those consumers using their init.d-scripts, the
> init.d-scripts sends both a normal sigint and one of those kill-messages to
> make sure the script gets interrupted anytime soon. For the million-messages
> consumer, that isn't really necessary, but for the few-hundred consumer it
> is quite usefull to wake it up, so it can die right away.
>

:) Awesome. I totally understand now.


> One of the reasons to start using these consumers was indeed to facilitate
> batch processing, although I haven't implemented such a feature yet. But I
> wouldn't worry too much about memory leaks, as long as you clean up the data
> afterwards and maybe keep away from using too fancy resources and too
> complex object trees.
> I would however add some more clean-kill stuff, so the current internal
> list is processed cleanly, prior to dieing.
>

I don't anticipate doing any XML or DOM parsing here, but I'll keep that in
mind for the future. Thanks again for all your help!

Cheers,

-- Joel

Re: Basic queue concept

[Arjen van der Meijden <ac...@tweakers.net>]

On 3-4-2008 8:50, Joel Poloney wrote:
> Is your "kill"-message on every call? Or is that only when you manually
> interrupt it? I'm wondering if you changed the Stomp-implementation such
> that if you've consumed all of the messages (and it's waiting for a new
> message to come through), it automatically dies. Or is it part of your
> "clean kill" that would make sure it finishes what it was doing, and then
> properly shuts down.

My consumer just consumes a message, processes it and then goes on 
waiting for another one. So if that message happens to be a 
'kill'-message, it dies (if the message was sent after the consumer 
started).

When I shutdown those consumers using their init.d-scripts, the 
init.d-scripts sends both a normal sigint and one of those kill-messages 
to make sure the script gets interrupted anytime soon. For the 
million-messages consumer, that isn't really necessary, but for the 
few-hundred consumer it is quite usefull to wake it up, so it can die 
right away.

> In my particular application, I would most likely have an internal queue in
> the consumer that would basically consume messages until it hit some number
> (say 100) and then batch process all 100 of them at once. Then it would go
> back to trying to consume messages again and filling up the internal queue.
> It's better for me to batch process 100 messages at once rather than 1 at a
> time. That's why I was asking about the memory leak issues.

One of the reasons to start using these consumers was indeed to 
facilitate batch processing, although I haven't implemented such a 
feature yet. But I wouldn't worry too much about memory leaks, as long 
as you clean up the data afterwards and maybe keep away from using too 
fancy resources and too complex object trees.
I would however add some more clean-kill stuff, so the current internal 
list is processed cleanly, prior to dieing.

Best regards,

Arjen

Re: Basic queue concept

[Joel Poloney <jp...@gmail.com>]

On Wed, Apr 2, 2008 at 11:39 PM, Arjen van der Meijden <
acmmailing@tweakers.net> wrote:

> I would go with 1. But you may want to add some control-code to ensure you
> can easily and cleanly kill your scripts. In my code, I added a few signal
> handlers, so the consumer first finishes its most recent job and then dies.
> But than the fread-calls in the Stomp-implementation won't get interrupted.
> So I've a 'kill'-message for that situation, so my consumer dies as soon as
> its either blocked in fread waiting for a new message or right after its
> current job.


Is your "kill"-message on every call? Or is that only when you manually
interrupt it? I'm wondering if you changed the Stomp-implementation such
that if you've consumed all of the messages (and it's waiting for a new
message to come through), it automatically dies. Or is it part of your
"clean kill" that would make sure it finishes what it was doing, and then
properly shuts down.

In my particular application, I would most likely have an internal queue in
the consumer that would basically consume messages until it hit some number
(say 100) and then batch process all 100 of them at once. Then it would go
back to trying to consume messages again and filling up the internal queue.
It's better for me to batch process 100 messages at once rather than 1 at a
time. That's why I was asking about the memory leak issues.

Thanks,

-- Joel

Re: Basic queue concept

[Arjen van der Meijden <ac...@tweakers.net>]

On 3-4-2008 7:47, Joel Poloney wrote:
> 1. I have a consumer in a while(1) { //consume } fashion. That would
> basically run forever. As I understand it, this is the way most web servers
> work (at the core, core level). In this model, I would have to make sure
> that the consumer was always running (perhaps do 3-way redundancy and have 3
> consumers doing this). My concern with this model is that if I implement
> this in PHP, I'm afraid of memory issues and so on. I don't necessarily
> trust a PHP script to run in a while(1) fashion forever.

I have this set-up, with four different consumers with varying load (two 
do more than two million entries per day, one only a few hundred and one 
a few ten thousand). I have seen no memory leak whatsoever from them.
The only major memory leakage from php I've seen recently was with 
SimpleXML, so if you intend to do XML-processing with those scripts, use 
DOM or expat.

Ironically, ActiveMQ leaks memory in my setup and therefore has a lower 
uptime than my php-consumers :X

> 2. I could somehow get the queue size and batch process these. Basically,
> get the queue size (say it was 200 entries), consume 200 messages and then
> the PHP script would end. Then you would have this script in a cronjob that
> runs every x amount of time.
> 
> Unfortunately, I have to implement this in PHP. So, I'm not really sure
> which way would be the proper way to go about doing this. If any one has any
> thoughts or comments, it would be greatly appreciated.

I would go with 1. But you may want to add some control-code to ensure 
you can easily and cleanly kill your scripts. In my code, I added a few 
signal handlers, so the consumer first finishes its most recent job and 
then dies. But than the fread-calls in the Stomp-implementation won't 
get interrupted. So I've a 'kill'-message for that situation, so my 
consumer dies as soon as its either blocked in fread waiting for a new 
message or right after its current job.

Best regards,

Arjen