Stomp/Perl usage?

[Peter Leonard <pe...@peteleonard.com>]

Hi Everyone,

I've been researching ActiveMQ as a potential solution for a system I'm 
working on, and I was hoping for a little feedback from the AMQ community.

Is anyone out there using AMQ to support applications written in Perl? 
I've found the Stomp/Perl client, but it's described as being of 
Alpha-quality, and it doesn't seem like there's much in terms of 
community or support specifically for it.

Assuming that the Perl API for Stomp is in need of some TLC (not an 
insurmountable issue on my end), how is the general Stomp interface at 
this point?

(Some quick background - the system requirements would be approximately 
500 messages/second over an HA environment, supporting message 
persistance for potentially several hours, if possible.  Early reading 
with AMQ suggests that setting up a cluster would be the way to go).

	Thank you,

	Peter

Re: Stomp/Perl usage?

[James Strachan <ja...@gmail.com>]

I've updated the clustering page a little to reflect the current state
of the code and make things a little clearer...

http://activemq.org/site/clustering.html

On 7/14/06, James Strachan <ja...@gmail.com> wrote:
> On 7/13/06, Peter Leonard <pe...@peteleonard.com> wrote:
> > James Strachan wrote:
> > > On 7/13/06, Peter Leonard <pe...@peteleonard.com> wrote:
> > >
> > >>
> > >> Hi Everyone,
> > >>
> > >> I've been researching ActiveMQ as a potential solution for a system I'm
> > >> working on, and I was hoping for a little feedback from the AMQ
> > >> community.
> > >>
> > >> Is anyone out there using AMQ to support applications written in Perl?
> > >> I've found the Stomp/Perl client, but it's described as being of
> > >> Alpha-quality, and it doesn't seem like there's much in terms of
> > >> community or support specifically for it.
> > >>
> > >> Assuming that the Perl API for Stomp is in need of some TLC (not an
> > >> insurmountable issue on my end), how is the general Stomp interface at
> > >> this point?
> > >
> > >
> > > Stomp support is good; we've lots of clients so far and the protocol
> > > has been well and trully hammered...
> > >
> > > http://activemq.org/site/cross-language-clients.html
> > >
> > > Though there are sometimes some gremlins with certain clients; but the
> > > clients are generally simple enough to be easy to fix if you hit any
> > > issues. (e.g. the perl client is quite small and pretty simple code to
> > > fix if there's anything you need changing etc).
> > >
> >
> > Sounds good - is there anyone out there using the Perl client that could
> > shed any light on the situation WRT Perl?  I've perused the client
> > source, and it's extremely spartan - I'm sure it works, but it could
> > definitely use some polishing.  :)
>
> :)
>
> Given the relative simplicity of the clients in perl, python, ruby and
> other scripting languages (and stomp itself being quite simple too)
> its probably best to just fire questions here about the stomp commands
> that are exchanged and we can all dive in and help. Particularly as
> folks like me are pretty clueless when it comes to Perl :)
>
> Incidentally one of the most advanced scripting clients is probably
> the ruby one; if you can read Ruby code its well worth taking a look
> at etc...
>
> http://stomp.codehaus.org/Ruby
>
>
> > To confirm - all the Stomp clients are implementing what's documented here:
> >
> > http://stomp.codehaus.org/Protocol
> >
> > Is there anything beyond that doc?  For example, failover (see further
> > commentary below).
>
> So the above is the only real doc - though there's some more detail on
> how to use all the extra features of ActiveMQ and Stomp here
>
> http://incubator.apache.org/activemq/stomp.html
>
> Failover is just a matter of detecting a socket failed and
> reconnecting again to one of maybe a list of sockets. On a
> reconnection you really should re-subscribe to all the things you were
> subscribed to before and resend any in-flight sends/acks.
>
>
> > I can see the desire to implement both failover & perhaps auto-discovery
> > in the Perl client (in addition, setting it up as an easily-installable
> > Perl module, like any other CPAN module).  Are there other features in
> > the other Stomp clients that would be desireable in the Perl client?
>
> I think those are the big ones. Discovery is the trickiest one I think.
>
>
> One other thing we added to the ruby client was being able to replay
> transactions on the client to preserve order. (If you look at the code
> its probably quite self explanatory as its a really small amount of
> code).
>
> Basically if you are using a transaction on the client, keep the
> messages around until the commit. If the transaction is rolled back,
> just redeliver the same messages again. This avoids closing the
> consumer and the messages all being redispatched again and causing
> order to be broken etc (plus its much faster too).
>
> BTW we should maybe move this over to the dev list :)
>
>
> > >> (Some quick background - the system requirements would be approximately
> > >> 500 messages/second over an HA environment, supporting message
> > >> persistance for potentially several hours, if possible.  Early reading
> > >> with AMQ suggests that setting up a cluster would be the way to go).
> > >
> > >
> > > Yeah - the perl would connect to a Master which could failover to a
> > > Slave...
> > >
> > > http://activemq.org/site/masterslave.html
> > >
> > > the only thing that needs adding is the ability to failver in the Perl
> > > client which I don't think is there yet.
> > >
> >
> > Funny you should bring that up.  :)
> >
> > I'm a little confused with the AMQ docs with respect to MasterSlave
> > versus clustering versus other topologies.
> >
> > Some of the docs talk about MasterSlave as the way to implement HA
> > setups, but then there are other docs talking about clustering brokers
> > to create a messaging fabric, which suggests a network of equals.
> >
> > To add to the confusion, the website differentiates between "Broker
> > Clusters" and "Networks of Brokers", talking about them as two different
> > things, but yet the paragraphs describing the two terms seem somewhat alike:
> >
> > http://incubator.apache.org/activemq/clustering.html
> >
> > Some insight would be appreciated - perhaps the answer to this question
> > would be best -
>
> Sorry about that; its hard keeping all the docs up to date and
> sometimes things use different terminology to confuse folks. I hope we
> can tidy up the clustering docs to make things absolutely clear
> (clustering can be quite confusing).
>
> Basically Master/Slave does the replicaton thing; where messages are
> replicated to the slave so if you loose the entire hardware/data
> centre running the master you can carry on without loosing any
> messages.
>
> You can also create a 'fabric' of brokers using store and forward
> (i.e. networks of brokers). This allows clients to connect to any
> broker and failover to another broker (just as clients do with
> master/slave when the master dies).
>
>
> > If you had to set up AMQ to support 500 messages/second, in HA fashion,
> > with message persistance  (so clients could retrieve old messages) and
> > you had a network of 10-15 machines that would run services interacting
> > with AMQ, how would you set it up?
>
> A pair of brokers in a Master/Slave pair and everyone else a client of
> the broker(s). 500/second is fairly low volume. On modern blades
> (particularly with things like opterons) its only in the 10,000/second
> range that I'd start thinking about networks of brokers. YMMV
>
> http://incubator.apache.org/activemq/performance.html
>
>
> > I know that some of this is more of a philosophical battle, and I'm not
> > trying to start a holy war of any kind.  I am personally a little
> > hesitant about the Master/Slave relationship, if only because I know it
> > from the RDBMS world, and I know the hell that can arise when a Master
> > DB blows up.  If a messaging fabric makes that problem go away, I'm a
> > happy camper.
>
> The only real reason for master/slave is to replicate messages to be
> able to deal with hardware failures. If you are not using persistent
> messaging or trust your hardware/disks or have a SAN (or reliable
> shared drive of some kind) then you don't need to use it - you can
> just have a number of brokers. Networks are only required when you
> want to store and forward between brokers. You could even just have
> separate brokers - and just get your consumers to connect to every
> broker (there's a feature request to support that out of the
> box...)...
>
> http://issues.apache.org/activemq/browse/AMQ-816
>
> the only problem with having networks of brokers or stand alone
> brokers is if a broker goes down; any messages on the disk of the box
> stay there until the broker is restarted. Whereas with Master/Slave
> the messages are all replicated so you can loose an entire box/data
> centre and still keep going.
> --
>
> James
> -------
> http://radio.weblogs.com/0112098/
>


-- 

James
-------
http://radio.weblogs.com/0112098/

Re: Stomp/Perl usage?

[James Strachan <ja...@gmail.com>]

On 7/13/06, Peter Leonard <pe...@peteleonard.com> wrote:
> James Strachan wrote:
> > On 7/13/06, Peter Leonard <pe...@peteleonard.com> wrote:
> >
> >>
> >> Hi Everyone,
> >>
> >> I've been researching ActiveMQ as a potential solution for a system I'm
> >> working on, and I was hoping for a little feedback from the AMQ
> >> community.
> >>
> >> Is anyone out there using AMQ to support applications written in Perl?
> >> I've found the Stomp/Perl client, but it's described as being of
> >> Alpha-quality, and it doesn't seem like there's much in terms of
> >> community or support specifically for it.
> >>
> >> Assuming that the Perl API for Stomp is in need of some TLC (not an
> >> insurmountable issue on my end), how is the general Stomp interface at
> >> this point?
> >
> >
> > Stomp support is good; we've lots of clients so far and the protocol
> > has been well and trully hammered...
> >
> > http://activemq.org/site/cross-language-clients.html
> >
> > Though there are sometimes some gremlins with certain clients; but the
> > clients are generally simple enough to be easy to fix if you hit any
> > issues. (e.g. the perl client is quite small and pretty simple code to
> > fix if there's anything you need changing etc).
> >
>
> Sounds good - is there anyone out there using the Perl client that could
> shed any light on the situation WRT Perl?  I've perused the client
> source, and it's extremely spartan - I'm sure it works, but it could
> definitely use some polishing.  :)

:)

Given the relative simplicity of the clients in perl, python, ruby and
other scripting languages (and stomp itself being quite simple too)
its probably best to just fire questions here about the stomp commands
that are exchanged and we can all dive in and help. Particularly as
folks like me are pretty clueless when it comes to Perl :)

Incidentally one of the most advanced scripting clients is probably
the ruby one; if you can read Ruby code its well worth taking a look
at etc...

http://stomp.codehaus.org/Ruby


> To confirm - all the Stomp clients are implementing what's documented here:
>
> http://stomp.codehaus.org/Protocol
>
> Is there anything beyond that doc?  For example, failover (see further
> commentary below).

So the above is the only real doc - though there's some more detail on
how to use all the extra features of ActiveMQ and Stomp here

http://incubator.apache.org/activemq/stomp.html

Failover is just a matter of detecting a socket failed and
reconnecting again to one of maybe a list of sockets. On a
reconnection you really should re-subscribe to all the things you were
subscribed to before and resend any in-flight sends/acks.


> I can see the desire to implement both failover & perhaps auto-discovery
> in the Perl client (in addition, setting it up as an easily-installable
> Perl module, like any other CPAN module).  Are there other features in
> the other Stomp clients that would be desireable in the Perl client?

I think those are the big ones. Discovery is the trickiest one I think.


One other thing we added to the ruby client was being able to replay
transactions on the client to preserve order. (If you look at the code
its probably quite self explanatory as its a really small amount of
code).

Basically if you are using a transaction on the client, keep the
messages around until the commit. If the transaction is rolled back,
just redeliver the same messages again. This avoids closing the
consumer and the messages all being redispatched again and causing
order to be broken etc (plus its much faster too).

BTW we should maybe move this over to the dev list :)


> >> (Some quick background - the system requirements would be approximately
> >> 500 messages/second over an HA environment, supporting message
> >> persistance for potentially several hours, if possible.  Early reading
> >> with AMQ suggests that setting up a cluster would be the way to go).
> >
> >
> > Yeah - the perl would connect to a Master which could failover to a
> > Slave...
> >
> > http://activemq.org/site/masterslave.html
> >
> > the only thing that needs adding is the ability to failver in the Perl
> > client which I don't think is there yet.
> >
>
> Funny you should bring that up.  :)
>
> I'm a little confused with the AMQ docs with respect to MasterSlave
> versus clustering versus other topologies.
>
> Some of the docs talk about MasterSlave as the way to implement HA
> setups, but then there are other docs talking about clustering brokers
> to create a messaging fabric, which suggests a network of equals.
>
> To add to the confusion, the website differentiates between "Broker
> Clusters" and "Networks of Brokers", talking about them as two different
> things, but yet the paragraphs describing the two terms seem somewhat alike:
>
> http://incubator.apache.org/activemq/clustering.html
>
> Some insight would be appreciated - perhaps the answer to this question
> would be best -

Sorry about that; its hard keeping all the docs up to date and
sometimes things use different terminology to confuse folks. I hope we
can tidy up the clustering docs to make things absolutely clear
(clustering can be quite confusing).

Basically Master/Slave does the replicaton thing; where messages are
replicated to the slave so if you loose the entire hardware/data
centre running the master you can carry on without loosing any
messages.

You can also create a 'fabric' of brokers using store and forward
(i.e. networks of brokers). This allows clients to connect to any
broker and failover to another broker (just as clients do with
master/slave when the master dies).


> If you had to set up AMQ to support 500 messages/second, in HA fashion,
> with message persistance  (so clients could retrieve old messages) and
> you had a network of 10-15 machines that would run services interacting
> with AMQ, how would you set it up?

A pair of brokers in a Master/Slave pair and everyone else a client of
the broker(s). 500/second is fairly low volume. On modern blades
(particularly with things like opterons) its only in the 10,000/second
range that I'd start thinking about networks of brokers. YMMV

http://incubator.apache.org/activemq/performance.html


> I know that some of this is more of a philosophical battle, and I'm not
> trying to start a holy war of any kind.  I am personally a little
> hesitant about the Master/Slave relationship, if only because I know it
> from the RDBMS world, and I know the hell that can arise when a Master
> DB blows up.  If a messaging fabric makes that problem go away, I'm a
> happy camper.

The only real reason for master/slave is to replicate messages to be
able to deal with hardware failures. If you are not using persistent
messaging or trust your hardware/disks or have a SAN (or reliable
shared drive of some kind) then you don't need to use it - you can
just have a number of brokers. Networks are only required when you
want to store and forward between brokers. You could even just have
separate brokers - and just get your consumers to connect to every
broker (there's a feature request to support that out of the
box...)...

http://issues.apache.org/activemq/browse/AMQ-816

the only problem with having networks of brokers or stand alone
brokers is if a broker goes down; any messages on the disk of the box
stay there until the broker is restarted. Whereas with Master/Slave
the messages are all replicated so you can loose an entire box/data
centre and still keep going.
-- 

James
-------
http://radio.weblogs.com/0112098/

Re: Stomp/Perl usage?

[Peter Leonard <pe...@peteleonard.com>]

James Strachan wrote:
> On 7/13/06, Peter Leonard <pe...@peteleonard.com> wrote:
> 
>>
>> Hi Everyone,
>>
>> I've been researching ActiveMQ as a potential solution for a system I'm
>> working on, and I was hoping for a little feedback from the AMQ 
>> community.
>>
>> Is anyone out there using AMQ to support applications written in Perl?
>> I've found the Stomp/Perl client, but it's described as being of
>> Alpha-quality, and it doesn't seem like there's much in terms of
>> community or support specifically for it.
>>
>> Assuming that the Perl API for Stomp is in need of some TLC (not an
>> insurmountable issue on my end), how is the general Stomp interface at
>> this point?
> 
> 
> Stomp support is good; we've lots of clients so far and the protocol
> has been well and trully hammered...
> 
> http://activemq.org/site/cross-language-clients.html
> 
> Though there are sometimes some gremlins with certain clients; but the
> clients are generally simple enough to be easy to fix if you hit any
> issues. (e.g. the perl client is quite small and pretty simple code to
> fix if there's anything you need changing etc).
> 

Sounds good - is there anyone out there using the Perl client that could 
shed any light on the situation WRT Perl?  I've perused the client 
source, and it's extremely spartan - I'm sure it works, but it could 
definitely use some polishing.  :)

To confirm - all the Stomp clients are implementing what's documented here:

http://stomp.codehaus.org/Protocol

Is there anything beyond that doc?  For example, failover (see further 
commentary below).

I can see the desire to implement both failover & perhaps auto-discovery 
in the Perl client (in addition, setting it up as an easily-installable 
Perl module, like any other CPAN module).  Are there other features in 
the other Stomp clients that would be desireable in the Perl client?

> 
>> (Some quick background - the system requirements would be approximately
>> 500 messages/second over an HA environment, supporting message
>> persistance for potentially several hours, if possible.  Early reading
>> with AMQ suggests that setting up a cluster would be the way to go).
> 
> 
> Yeah - the perl would connect to a Master which could failover to a 
> Slave...
> 
> http://activemq.org/site/masterslave.html
> 
> the only thing that needs adding is the ability to failver in the Perl
> client which I don't think is there yet.
> 

Funny you should bring that up.  :)

I'm a little confused with the AMQ docs with respect to MasterSlave 
versus clustering versus other topologies.

Some of the docs talk about MasterSlave as the way to implement HA 
setups, but then there are other docs talking about clustering brokers 
to create a messaging fabric, which suggests a network of equals.

To add to the confusion, the website differentiates between "Broker 
Clusters" and "Networks of Brokers", talking about them as two different 
things, but yet the paragraphs describing the two terms seem somewhat alike:

http://incubator.apache.org/activemq/clustering.html

Some insight would be appreciated - perhaps the answer to this question 
would be best -

If you had to set up AMQ to support 500 messages/second, in HA fashion, 
with message persistance  (so clients could retrieve old messages) and 
you had a network of 10-15 machines that would run services interacting 
with AMQ, how would you set it up?

Would you set up 2-3 dedicated boxes for AMQ brokers?  Would you run AMQ 
brokers on some or all of those servers (given that they have excess 
capacity) instead?

I know that some of this is more of a philosophical battle, and I'm not 
trying to start a holy war of any kind.  I am personally a little 
hesitant about the Master/Slave relationship, if only because I know it 
from the RDBMS world, and I know the hell that can arise when a Master 
DB blows up.  If a messaging fabric makes that problem go away, I'm a 
happy camper.

	Thank you for your time,

	Pete

Re: Stomp/Perl usage?

[James Strachan <ja...@gmail.com>]

On 7/13/06, Peter Leonard <pe...@peteleonard.com> wrote:
>
> Hi Everyone,
>
> I've been researching ActiveMQ as a potential solution for a system I'm
> working on, and I was hoping for a little feedback from the AMQ community.
>
> Is anyone out there using AMQ to support applications written in Perl?
> I've found the Stomp/Perl client, but it's described as being of
> Alpha-quality, and it doesn't seem like there's much in terms of
> community or support specifically for it.
>
> Assuming that the Perl API for Stomp is in need of some TLC (not an
> insurmountable issue on my end), how is the general Stomp interface at
> this point?

Stomp support is good; we've lots of clients so far and the protocol
has been well and trully hammered...

http://activemq.org/site/cross-language-clients.html

Though there are sometimes some gremlins with certain clients; but the
clients are generally simple enough to be easy to fix if you hit any
issues. (e.g. the perl client is quite small and pretty simple code to
fix if there's anything you need changing etc).


> (Some quick background - the system requirements would be approximately
> 500 messages/second over an HA environment, supporting message
> persistance for potentially several hours, if possible.  Early reading
> with AMQ suggests that setting up a cluster would be the way to go).

Yeah - the perl would connect to a Master which could failover to a Slave...

http://activemq.org/site/masterslave.html

the only thing that needs adding is the ability to failver in the Perl
client which I don't think is there yet.

-- 

James
-------
http://radio.weblogs.com/0112098/