You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@airavata.apache.org by Danushka Menikkumbura <da...@gmail.com> on 2013/03/27 21:07:59 UTC
AMQP Support for WS-Messenger
Hi,
When I go through [1], I get the impression that what we really need is an
extensible event bus that can provide notification push/pull functionality
so that a gateway developer has the flexibility to come up with his own
protocol for receiving notifications. At a very higher level the messenger
would have two main layers, the routing layer and the transport layer. We
would have the routing rules defined in the routing layer that makes
notifications available to different transports, such as WS-Notification,
WS-Eventing, file, mail, AMQP, JMS, etc or any other custom type developed
as you wish.
Along the line of AMQP, basically the requirements could be twofold.
1. AMQP broker semantics to support pub/sub behavior
2. AMPQ wire-level format
I am not quite sure which makes AMQP more appealing to the scientific
community, specially in environments like XSEDE. Furthermore, I would like
to know a typical scenario, either real or hypothetical that could exploit
the capabilities of AMQP.
[1] - https://issues.apache.org/jira/browse/AIRAVATA-339
Thanks,
Danushka
Re: AMQP Support for WS-Messenger
Posted by Danushka Menikkumbura <da...@gmail.com>.
Because actual scenario is, grid system has AMQP
> messaging system, and we just want to extract out those data, so
> realistically gateway developers might not very interested in AMQP
> complexities, but still interested in data receiving in those messages.
>
Correct!. We need to design the API carefully.
Thanks,
Danushka
Re: AMQP Support for WS-Messenger
Posted by Danushka Menikkumbura <da...@gmail.com>.
> Just wondering, can we use JMS to provide API support ?
>
A native AMQP API is more powerful than JMS. JMS would be option 2.
> IMO, it does matter. I have seen people getting into trouble when they
> using wrong AMQP implementations in their applications. If you
> carefully google it, you will see many articles. There are certain
> parameters which we need to be concern about when selecting a proper
> AMQP implementation. I would also like to know what those parameters
> are.
We are fine with any AMQP implementation as long as it is compliant with
the specification. Naturally, there could be issues in implementation
level. Lets go for RabbitMQ and see how it fits in.
Thanks,
Danushka
Re: AMQP Support for WS-Messenger
Posted by Amila Jayasekara <th...@gmail.com>.
On Thu, Mar 28, 2013 at 7:22 PM, Lahiru Gunathilake <gl...@gmail.com> wrote:
> Hi Danushka,
>
> Yes this seems very useful.More importantly, when someone develop a science
> gateway they would really like to subscribe to different types of messages
> and show the users differnt information (Ex: workflow stuck, workflow node
> failed, workflow node in progress etc). So we have to come up with very
> easy and less complex API for users to deal with this, and I would say we
> can hide AMQP level information when we design the API (like we handle AMQP
> complexity underneath).
Just wondering, can we use JMS to provide API support ?
Because actual scenario is, grid system has AMQP
> messaging system, and we just want to extract out those data, so
> realistically gateway developers might not very interested in AMQP
> complexities, but still interested in data receiving in those messages.
>
> Amila, I think we can test with RabbitMQ because its the most widely used
> implementation, but It really does'nt matter,
IMO, it does matter. I have seen people getting into trouble when they
using wrong AMQP implementations in their applications. If you
carefully google it, you will see many articles. There are certain
parameters which we need to be concern about when selecting a proper
AMQP implementation. I would also like to know what those parameters
are.
Thanks
Amila
anything should work out of
> the box.
>
> Regards
> Lahiru
>
> On Thu, Mar 28, 2013 at 5:31 PM, Danushka Menikkumbura <
> danushka.menikkumbura@gmail.com> wrote:
>
>> Hi Suresh,
>>
>> The AMQP exchanges fit the Airavata notification model quite nicely.
>> Basically we can map elements in the workflow tracking schema to the
>> Header/Topic exchanges (or even to Direct/Fanout exchanges). For an
>> example, we can define a topic (MyTopic) and associate few message types in
>> the schema with it so that whoever subscribes to MyTopic would receive
>> those messages. Furthermore we can let subscribers have durable
>> subscriptions so that the messages will get stored and forwarded in case
>> the subscriber in on an unreliable network. Likewise we can make use of the
>> rich set of building blocks available in AMQP model to have a very robust
>> notification framework.
>>
>> RabbitMQ has a nice article [1] about the AMQP model that explains
>> different types of exchanges, etc nicely. I suggest you read the "Exchanges
>> and Exchange Types" section.
>>
>> [1] - http://www.rabbitmq.com/tutorials/amqp-concepts.html
>>
>> Thanks,
>> Danushka
>>
>>
>> On Thu, Mar 28, 2013 at 1:48 AM, Suresh Marru <sm...@apache.org> wrote:
>>
>> > Hi Danushka,
>> >
>> > Good thinking, I agree with you for the most part. There are few other
>> > concrete use cases I can suggest, but let me get your thoughts on a
>> > particular example:
>> >
>> > Currently WS-Eventing is used to track workflow progress. In reality its
>> > not so much of the eventing spec, but the Airavata Information Model
>> which
>> > is currently based on Workflow Tracking schema [1]. As you can see form
>> the
>> > Web Based XBaya discussion, if a light weight java script based workflow
>> > GUI has to monitor progress and show real-time status (like the current
>> > color coded), would you say AMQP has any additional advantages? Or if you
>> > were to design from scratch, what would you pick which can build upon a
>> > pre-defined information model.
>> >
>> > Suresh
>> > [1] -
>> >
>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>> >
>> > On Mar 27, 2013, at 4:07 PM, Danushka Menikkumbura <
>> > danushka.menikkumbura@gmail.com> wrote:
>> >
>> > > Hi,
>> > >
>> > > When I go through [1], I get the impression that what we really need is
>> > an
>> > > extensible event bus that can provide notification push/pull
>> > functionality
>> > > so that a gateway developer has the flexibility to come up with his own
>> > > protocol for receiving notifications. At a very higher level the
>> > messenger
>> > > would have two main layers, the routing layer and the transport layer.
>> We
>> > > would have the routing rules defined in the routing layer that makes
>> > > notifications available to different transports, such as
>> WS-Notification,
>> > > WS-Eventing, file, mail, AMQP, JMS, etc or any other custom type
>> > developed
>> > > as you wish.
>> > >
>> > > Along the line of AMQP, basically the requirements could be twofold.
>> > >
>> > > 1. AMQP broker semantics to support pub/sub behavior
>> > > 2. AMPQ wire-level format
>> > >
>> > > I am not quite sure which makes AMQP more appealing to the scientific
>> > > community, specially in environments like XSEDE. Furthermore, I would
>> > like
>> > > to know a typical scenario, either real or hypothetical that could
>> > exploit
>> > > the capabilities of AMQP.
>> > >
>> > > [1] - https://issues.apache.org/jira/browse/AIRAVATA-339
>> > >
>> > > Thanks,
>> > > Danushka
>> >
>> >
>>
>
>
>
> --
> System Analyst Programmer
> PTI Lab
> Indiana University
Re: AMQP Support for WS-Messenger
Posted by Lahiru Gunathilake <gl...@gmail.com>.
Hi Danushka,
Yes this seems very useful.More importantly, when someone develop a science
gateway they would really like to subscribe to different types of messages
and show the users differnt information (Ex: workflow stuck, workflow node
failed, workflow node in progress etc). So we have to come up with very
easy and less complex API for users to deal with this, and I would say we
can hide AMQP level information when we design the API (like we handle AMQP
complexity underneath). Because actual scenario is, grid system has AMQP
messaging system, and we just want to extract out those data, so
realistically gateway developers might not very interested in AMQP
complexities, but still interested in data receiving in those messages.
Amila, I think we can test with RabbitMQ because its the most widely used
implementation, but It really does'nt matter, anything should work out of
the box.
Regards
Lahiru
On Thu, Mar 28, 2013 at 5:31 PM, Danushka Menikkumbura <
danushka.menikkumbura@gmail.com> wrote:
> Hi Suresh,
>
> The AMQP exchanges fit the Airavata notification model quite nicely.
> Basically we can map elements in the workflow tracking schema to the
> Header/Topic exchanges (or even to Direct/Fanout exchanges). For an
> example, we can define a topic (MyTopic) and associate few message types in
> the schema with it so that whoever subscribes to MyTopic would receive
> those messages. Furthermore we can let subscribers have durable
> subscriptions so that the messages will get stored and forwarded in case
> the subscriber in on an unreliable network. Likewise we can make use of the
> rich set of building blocks available in AMQP model to have a very robust
> notification framework.
>
> RabbitMQ has a nice article [1] about the AMQP model that explains
> different types of exchanges, etc nicely. I suggest you read the "Exchanges
> and Exchange Types" section.
>
> [1] - http://www.rabbitmq.com/tutorials/amqp-concepts.html
>
> Thanks,
> Danushka
>
>
> On Thu, Mar 28, 2013 at 1:48 AM, Suresh Marru <sm...@apache.org> wrote:
>
> > Hi Danushka,
> >
> > Good thinking, I agree with you for the most part. There are few other
> > concrete use cases I can suggest, but let me get your thoughts on a
> > particular example:
> >
> > Currently WS-Eventing is used to track workflow progress. In reality its
> > not so much of the eventing spec, but the Airavata Information Model
> which
> > is currently based on Workflow Tracking schema [1]. As you can see form
> the
> > Web Based XBaya discussion, if a light weight java script based workflow
> > GUI has to monitor progress and show real-time status (like the current
> > color coded), would you say AMQP has any additional advantages? Or if you
> > were to design from scratch, what would you pick which can build upon a
> > pre-defined information model.
> >
> > Suresh
> > [1] -
> >
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-tracking/src/main/resources/schemas/workflow_tracking_types.xsd
> >
> > On Mar 27, 2013, at 4:07 PM, Danushka Menikkumbura <
> > danushka.menikkumbura@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > When I go through [1], I get the impression that what we really need is
> > an
> > > extensible event bus that can provide notification push/pull
> > functionality
> > > so that a gateway developer has the flexibility to come up with his own
> > > protocol for receiving notifications. At a very higher level the
> > messenger
> > > would have two main layers, the routing layer and the transport layer.
> We
> > > would have the routing rules defined in the routing layer that makes
> > > notifications available to different transports, such as
> WS-Notification,
> > > WS-Eventing, file, mail, AMQP, JMS, etc or any other custom type
> > developed
> > > as you wish.
> > >
> > > Along the line of AMQP, basically the requirements could be twofold.
> > >
> > > 1. AMQP broker semantics to support pub/sub behavior
> > > 2. AMPQ wire-level format
> > >
> > > I am not quite sure which makes AMQP more appealing to the scientific
> > > community, specially in environments like XSEDE. Furthermore, I would
> > like
> > > to know a typical scenario, either real or hypothetical that could
> > exploit
> > > the capabilities of AMQP.
> > >
> > > [1] - https://issues.apache.org/jira/browse/AIRAVATA-339
> > >
> > > Thanks,
> > > Danushka
> >
> >
>
--
System Analyst Programmer
PTI Lab
Indiana University
Re: AMQP Support for WS-Messenger
Posted by Amila Jayasekara <th...@gmail.com>.
On Thu, Mar 28, 2013 at 5:31 PM, Danushka Menikkumbura
<da...@gmail.com> wrote:
> Hi Suresh,
>
> The AMQP exchanges fit the Airavata notification model quite nicely.
> Basically we can map elements in the workflow tracking schema to the
> Header/Topic exchanges (or even to Direct/Fanout exchanges). For an
> example, we can define a topic (MyTopic) and associate few message types in
> the schema with it so that whoever subscribes to MyTopic would receive
> those messages. Furthermore we can let subscribers have durable
> subscriptions so that the messages will get stored and forwarded in case
> the subscriber in on an unreliable network. Likewise we can make use of the
> rich set of building blocks available in AMQP model to have a very robust
> notification framework.
>
> RabbitMQ has a nice article [1] about the AMQP model that explains
> different types of exchanges, etc nicely. I suggest you read the "Exchanges
> and Exchange Types" section.
Hi Danushka,
I am not overly familiar with AMQP. But I am just curious to know,
which AMQP implementation you suggest for Airavata ? (Knowing Airavata
requirements)
Thanks
Amila
>
> [1] - http://www.rabbitmq.com/tutorials/amqp-concepts.html
>
> Thanks,
> Danushka
>
>
> On Thu, Mar 28, 2013 at 1:48 AM, Suresh Marru <sm...@apache.org> wrote:
>
>> Hi Danushka,
>>
>> Good thinking, I agree with you for the most part. There are few other
>> concrete use cases I can suggest, but let me get your thoughts on a
>> particular example:
>>
>> Currently WS-Eventing is used to track workflow progress. In reality its
>> not so much of the eventing spec, but the Airavata Information Model which
>> is currently based on Workflow Tracking schema [1]. As you can see form the
>> Web Based XBaya discussion, if a light weight java script based workflow
>> GUI has to monitor progress and show real-time status (like the current
>> color coded), would you say AMQP has any additional advantages? Or if you
>> were to design from scratch, what would you pick which can build upon a
>> pre-defined information model.
>>
>> Suresh
>> [1] -
>> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>>
>> On Mar 27, 2013, at 4:07 PM, Danushka Menikkumbura <
>> danushka.menikkumbura@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > When I go through [1], I get the impression that what we really need is
>> an
>> > extensible event bus that can provide notification push/pull
>> functionality
>> > so that a gateway developer has the flexibility to come up with his own
>> > protocol for receiving notifications. At a very higher level the
>> messenger
>> > would have two main layers, the routing layer and the transport layer. We
>> > would have the routing rules defined in the routing layer that makes
>> > notifications available to different transports, such as WS-Notification,
>> > WS-Eventing, file, mail, AMQP, JMS, etc or any other custom type
>> developed
>> > as you wish.
>> >
>> > Along the line of AMQP, basically the requirements could be twofold.
>> >
>> > 1. AMQP broker semantics to support pub/sub behavior
>> > 2. AMPQ wire-level format
>> >
>> > I am not quite sure which makes AMQP more appealing to the scientific
>> > community, specially in environments like XSEDE. Furthermore, I would
>> like
>> > to know a typical scenario, either real or hypothetical that could
>> exploit
>> > the capabilities of AMQP.
>> >
>> > [1] - https://issues.apache.org/jira/browse/AIRAVATA-339
>> >
>> > Thanks,
>> > Danushka
>>
>>
Re: AMQP Support for WS-Messenger
Posted by Danushka Menikkumbura <da...@gmail.com>.
Hi Suresh,
The AMQP exchanges fit the Airavata notification model quite nicely.
Basically we can map elements in the workflow tracking schema to the
Header/Topic exchanges (or even to Direct/Fanout exchanges). For an
example, we can define a topic (MyTopic) and associate few message types in
the schema with it so that whoever subscribes to MyTopic would receive
those messages. Furthermore we can let subscribers have durable
subscriptions so that the messages will get stored and forwarded in case
the subscriber in on an unreliable network. Likewise we can make use of the
rich set of building blocks available in AMQP model to have a very robust
notification framework.
RabbitMQ has a nice article [1] about the AMQP model that explains
different types of exchanges, etc nicely. I suggest you read the "Exchanges
and Exchange Types" section.
[1] - http://www.rabbitmq.com/tutorials/amqp-concepts.html
Thanks,
Danushka
On Thu, Mar 28, 2013 at 1:48 AM, Suresh Marru <sm...@apache.org> wrote:
> Hi Danushka,
>
> Good thinking, I agree with you for the most part. There are few other
> concrete use cases I can suggest, but let me get your thoughts on a
> particular example:
>
> Currently WS-Eventing is used to track workflow progress. In reality its
> not so much of the eventing spec, but the Airavata Information Model which
> is currently based on Workflow Tracking schema [1]. As you can see form the
> Web Based XBaya discussion, if a light weight java script based workflow
> GUI has to monitor progress and show real-time status (like the current
> color coded), would you say AMQP has any additional advantages? Or if you
> were to design from scratch, what would you pick which can build upon a
> pre-defined information model.
>
> Suresh
> [1] -
> https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-tracking/src/main/resources/schemas/workflow_tracking_types.xsd
>
> On Mar 27, 2013, at 4:07 PM, Danushka Menikkumbura <
> danushka.menikkumbura@gmail.com> wrote:
>
> > Hi,
> >
> > When I go through [1], I get the impression that what we really need is
> an
> > extensible event bus that can provide notification push/pull
> functionality
> > so that a gateway developer has the flexibility to come up with his own
> > protocol for receiving notifications. At a very higher level the
> messenger
> > would have two main layers, the routing layer and the transport layer. We
> > would have the routing rules defined in the routing layer that makes
> > notifications available to different transports, such as WS-Notification,
> > WS-Eventing, file, mail, AMQP, JMS, etc or any other custom type
> developed
> > as you wish.
> >
> > Along the line of AMQP, basically the requirements could be twofold.
> >
> > 1. AMQP broker semantics to support pub/sub behavior
> > 2. AMPQ wire-level format
> >
> > I am not quite sure which makes AMQP more appealing to the scientific
> > community, specially in environments like XSEDE. Furthermore, I would
> like
> > to know a typical scenario, either real or hypothetical that could
> exploit
> > the capabilities of AMQP.
> >
> > [1] - https://issues.apache.org/jira/browse/AIRAVATA-339
> >
> > Thanks,
> > Danushka
>
>
Re: AMQP Support for WS-Messenger
Posted by Suresh Marru <sm...@apache.org>.
Hi Danushka,
Good thinking, I agree with you for the most part. There are few other concrete use cases I can suggest, but let me get your thoughts on a particular example:
Currently WS-Eventing is used to track workflow progress. In reality its not so much of the eventing spec, but the Airavata Information Model which is currently based on Workflow Tracking schema [1]. As you can see form the Web Based XBaya discussion, if a light weight java script based workflow GUI has to monitor progress and show real-time status (like the current color coded), would you say AMQP has any additional advantages? Or if you were to design from scratch, what would you pick which can build upon a pre-defined information model.
Suresh
[1] - https://svn.apache.org/repos/asf/airavata/trunk/modules/commons/workflow-tracking/src/main/resources/schemas/workflow_tracking_types.xsd
On Mar 27, 2013, at 4:07 PM, Danushka Menikkumbura <da...@gmail.com> wrote:
> Hi,
>
> When I go through [1], I get the impression that what we really need is an
> extensible event bus that can provide notification push/pull functionality
> so that a gateway developer has the flexibility to come up with his own
> protocol for receiving notifications. At a very higher level the messenger
> would have two main layers, the routing layer and the transport layer. We
> would have the routing rules defined in the routing layer that makes
> notifications available to different transports, such as WS-Notification,
> WS-Eventing, file, mail, AMQP, JMS, etc or any other custom type developed
> as you wish.
>
> Along the line of AMQP, basically the requirements could be twofold.
>
> 1. AMQP broker semantics to support pub/sub behavior
> 2. AMPQ wire-level format
>
> I am not quite sure which makes AMQP more appealing to the scientific
> community, specially in environments like XSEDE. Furthermore, I would like
> to know a typical scenario, either real or hypothetical that could exploit
> the capabilities of AMQP.
>
> [1] - https://issues.apache.org/jira/browse/AIRAVATA-339
>
> Thanks,
> Danushka