You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Dean <cl...@tizatron.com> on 2012/05/11 22:57:45 UTC
Event Publish and Subscribe ( was perhaps Re: Cloudstack Questions )
Cross reference to:
http://mail-archives.apache.org/mod_mbox/incubator-cloudstack-dev/201204.mbox/browser
[ from: Marlon Davids ]
< munch >
> 2) How do we monitor VM's that are in Cloudstack when they are in an isolated VLAN does
> anyone have a clever workaround?
> 3) Has anyone developed a script for parsing and alerting on warning events in the
> management Log yet?
I would like to propose cloudstack consider a pub/sub model for event handling to complement API calls like listEvents.
Polling can be problematic and sensitive to scaling.
A simple example would be state change on a physical device. The admin server can simply publish a message on a network socket indicating that the device has changed it's state.
If a subscriber was interested in that device, it could make an api call to the admin server for state change information for that device only. The admin server may choose to validate that physical device against the current state table in the database.
The API would reply that this node changed it's state from Operational to Prep For Maintenance. (or whatever the transition state would be)
The message exchange could be wrapped around vm states, resource additions/removals etc.
Using a library like zeromq, a developer can write any number of consumers in any language they wanted to subscribe to the Event Bus.
Comments?
--
-Dean
RE: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Alex Huang <Al...@citrix.com>.
> Are there any use-cases for which it make sense to have publishers (that use
> CloudStack API to publish) other than CloudStack? If its not overkill we can
> have generic mechanism of publish/subscribe as service in CloudStack. I can
> think of tenants workload VM's be able to leverage this and publish
> something to subscribers.
There will be third party vendors who want to write plugins for CloudStack. I think a requirement for such a system includes
- generic events (in practical terms, it shouldn't be done with enums or anything that tie extension into modifications done in the same place).
- Provide domain separation between event publishers. (In practical terms, should be done with strings. Like java packages names).
- Provide some generic categorization of the events (ALERTs, STATE change, etc)
- mechanism to query the Events available
- mechanism to subscribe to specific events/categories/domains
--Alex
RE: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Abhinandan Prateek <Ab...@citrix.com>.
Summarized requirements as per discussion in the forum at http://wiki.cloudstack.org/display/RelOps/Cloudstack+Events
Following are some items for which community opinion will help:
1. What should be used to classify messages (*topic*) - domain or account ?
2. Is there any differentiation between internal and external consumers ? Like authentication is easier for internal consumers. We can provide a different mechanism for internal consumption.
3. Authentication and authorization in a multi tenancy environment for the message consuming clients ? This becomes more important when pushing events, probably not now. Any other security concerns ?
-abhi
From: George Reese [mailto:george.reese@enstratus.com]<mailto:[mailto:george.reese@enstratus.com]>
Sent: Friday, June 08, 2012 1:19 AM
To: cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>
Cc: Abhinandan Prateek
Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack Questions )
It also makes sense in a private cloud context as well. There's nothing different about public versus private on this matter, except the public cloud is going to have a lot more interested clients with the added burden of multi-tenancy management.
#1 I think publishing to a MQ is a necessary first step, but it's not externally meaningful
#2 I think pushing via notifications needs to be the end game.
At enStratus right now, we are working on a notifications specification that hopefully will suit the needs of cloud platforms without being very difficult to implement.
-George
On Jun 7, 2012, at 2:37 PM, Chiradeep Vittal wrote:
To George's earlier point about SNS, I think it makes a lot of sense in a
public cloud where the end-users of the cloud want notifications (as
opposed to the cloud operator). The feature should include access controls
and scoping, independent of the actual transport.
On 6/7/12 7:02 AM, "Abhinandan Prateek" <Ab...@citrix.com>>
wrote:
Created a placeholder for this feature here in Jira
http://bugs.cloudstack.org/browse/CS-15258
-----Original Message-----
From: Chip Childers [mailto:chip.childers@sungard.com]<mailto:[mailto:chip.childers@sungard.com]>
Sent: Thursday, June 07, 2012 7:15 PM
To: cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>
Cc: Chiradeep Vittal
Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
On Thu, Jun 7, 2012 at 7:18 AM, Abhinandan Prateek
<Ab...@citrix.com>> wrote:
Planning to use apache Qpid implementation of AMQP for publishing
cloudstack events.
+1 for implementing event notification via AMQP. This will be really
helpful in more advanced deployment environments. It would also be
great if
the implementation was considered optional, because I think the standard
deployment won't make use of this feature.
+1 for using the Qpid Java Client. It supports 0-8, 0-9 and 0-10 AMQP
specs, and they keep up with the AMQP spec development pretty well.
That would give a deployment options for which broker they use (which
might
end up being RabbitMQ more often than not).
I think that any CloudStack reference architectures (install guides,
etc...) and testing should use QPID as the broker as well... the Java
broker
seems to be the best option for project level testing (Although we use
the
C++ broker internally).
I'd also be very interested in hearing more about the topic and message
taxonomy that you are thinking of implementing. I'm not able to sign
the ICLA
yet, but I'm more than happy to help work through the design for this
functionality with you (on list of course). This is actually a pretty
important bit
of functionality in my environment.
Thanks!
-chip
--
George Reese - Chief Technology Officer, enStratus
e: george.reese@enstratus.com<ma...@enstratus.com> Skype: nspollution t: @GeorgeReese p: +1.207.956.0217
enStratus: Enterprise Cloud Management - @enStratus - http://www.enstratus.com<http://www.enstratus.com/>
To schedule a meeting with me: http://tungle.me/GeorgeReese
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack Questions )
Posted by George Reese <ge...@enstratus.com>.
It also makes sense in a private cloud context as well. There's nothing different about public versus private on this matter, except the public cloud is going to have a lot more interested clients with the added burden of multi-tenancy management.
#1 I think publishing to a MQ is a necessary first step, but it's not externally meaningful
#2 I think pushing via notifications needs to be the end game.
At enStratus right now, we are working on a notifications specification that hopefully will suit the needs of cloud platforms without being very difficult to implement.
-George
On Jun 7, 2012, at 2:37 PM, Chiradeep Vittal wrote:
> To George's earlier point about SNS, I think it makes a lot of sense in a
> public cloud where the end-users of the cloud want notifications (as
> opposed to the cloud operator). The feature should include access controls
> and scoping, independent of the actual transport.
>
> On 6/7/12 7:02 AM, "Abhinandan Prateek" <Ab...@citrix.com>
> wrote:
>
>> Created a placeholder for this feature here in Jira
>> http://bugs.cloudstack.org/browse/CS-15258
>>
>>> -----Original Message-----
>>> From: Chip Childers [mailto:chip.childers@sungard.com]
>>> Sent: Thursday, June 07, 2012 7:15 PM
>>> To: cloudstack-dev@incubator.apache.org
>>> Cc: Chiradeep Vittal
>>> Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>>> Questions )
>>>
>>> On Thu, Jun 7, 2012 at 7:18 AM, Abhinandan Prateek
>>> <Ab...@citrix.com> wrote:
>>>> Planning to use apache Qpid implementation of AMQP for publishing
>>> cloudstack events.
>>>
>>> +1 for implementing event notification via AMQP. This will be really
>>> helpful in more advanced deployment environments. It would also be
>>> great if
>>> the implementation was considered optional, because I think the standard
>>> deployment won't make use of this feature.
>>>
>>> +1 for using the Qpid Java Client. It supports 0-8, 0-9 and 0-10 AMQP
>>> specs, and they keep up with the AMQP spec development pretty well.
>>> That would give a deployment options for which broker they use (which
>>> might
>>> end up being RabbitMQ more often than not).
>>>
>>> I think that any CloudStack reference architectures (install guides,
>>> etc...) and testing should use QPID as the broker as well... the Java
>>> broker
>>> seems to be the best option for project level testing (Although we use
>>> the
>>> C++ broker internally).
>>>
>>> I'd also be very interested in hearing more about the topic and message
>>> taxonomy that you are thinking of implementing. I'm not able to sign
>>> the ICLA
>>> yet, but I'm more than happy to help work through the design for this
>>> functionality with you (on list of course). This is actually a pretty
>>> important bit
>>> of functionality in my environment.
>>>
>>> Thanks!
>>>
>>> -chip
>
--
George Reese - Chief Technology Officer, enStratus
e: george.reese@enstratus.com Skype: nspollution t: @GeorgeReese p: +1.207.956.0217
enStratus: Enterprise Cloud Management - @enStratus - http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Chiradeep Vittal <Ch...@citrix.com>.
To George's earlier point about SNS, I think it makes a lot of sense in a
public cloud where the end-users of the cloud want notifications (as
opposed to the cloud operator). The feature should include access controls
and scoping, independent of the actual transport.
On 6/7/12 7:02 AM, "Abhinandan Prateek" <Ab...@citrix.com>
wrote:
>Created a placeholder for this feature here in Jira
>http://bugs.cloudstack.org/browse/CS-15258
>
>>-----Original Message-----
>>From: Chip Childers [mailto:chip.childers@sungard.com]
>>Sent: Thursday, June 07, 2012 7:15 PM
>>To: cloudstack-dev@incubator.apache.org
>>Cc: Chiradeep Vittal
>>Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>>Questions )
>>
>>On Thu, Jun 7, 2012 at 7:18 AM, Abhinandan Prateek
>><Ab...@citrix.com> wrote:
>>> Planning to use apache Qpid implementation of AMQP for publishing
>>cloudstack events.
>>
>>+1 for implementing event notification via AMQP. This will be really
>>helpful in more advanced deployment environments. It would also be
>>great if
>>the implementation was considered optional, because I think the standard
>>deployment won't make use of this feature.
>>
>>+1 for using the Qpid Java Client. It supports 0-8, 0-9 and 0-10 AMQP
>>specs, and they keep up with the AMQP spec development pretty well.
>>That would give a deployment options for which broker they use (which
>>might
>>end up being RabbitMQ more often than not).
>>
>>I think that any CloudStack reference architectures (install guides,
>>etc...) and testing should use QPID as the broker as well... the Java
>>broker
>>seems to be the best option for project level testing (Although we use
>>the
>>C++ broker internally).
>>
>>I'd also be very interested in hearing more about the topic and message
>>taxonomy that you are thinking of implementing. I'm not able to sign
>>the ICLA
>>yet, but I'm more than happy to help work through the design for this
>>functionality with you (on list of course). This is actually a pretty
>>important bit
>>of functionality in my environment.
>>
>>Thanks!
>>
>>-chip
RE: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Abhinandan Prateek <Ab...@citrix.com>.
Created a placeholder for this feature here in Jira http://bugs.cloudstack.org/browse/CS-15258
>-----Original Message-----
>From: Chip Childers [mailto:chip.childers@sungard.com]
>Sent: Thursday, June 07, 2012 7:15 PM
>To: cloudstack-dev@incubator.apache.org
>Cc: Chiradeep Vittal
>Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>Questions )
>
>On Thu, Jun 7, 2012 at 7:18 AM, Abhinandan Prateek
><Ab...@citrix.com> wrote:
>> Planning to use apache Qpid implementation of AMQP for publishing
>cloudstack events.
>
>+1 for implementing event notification via AMQP. This will be really
>helpful in more advanced deployment environments. It would also be great if
>the implementation was considered optional, because I think the standard
>deployment won't make use of this feature.
>
>+1 for using the Qpid Java Client. It supports 0-8, 0-9 and 0-10 AMQP
>specs, and they keep up with the AMQP spec development pretty well.
>That would give a deployment options for which broker they use (which might
>end up being RabbitMQ more often than not).
>
>I think that any CloudStack reference architectures (install guides,
>etc...) and testing should use QPID as the broker as well... the Java broker
>seems to be the best option for project level testing (Although we use the
>C++ broker internally).
>
>I'd also be very interested in hearing more about the topic and message
>taxonomy that you are thinking of implementing. I'm not able to sign the ICLA
>yet, but I'm more than happy to help work through the design for this
>functionality with you (on list of course). This is actually a pretty important bit
>of functionality in my environment.
>
>Thanks!
>
>-chip
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Chip Childers <ch...@sungard.com>.
On Thu, Jun 7, 2012 at 7:18 AM, Abhinandan Prateek
<Ab...@citrix.com> wrote:
> Planning to use apache Qpid implementation of AMQP for publishing cloudstack events.
+1 for implementing event notification via AMQP. This will be really
helpful in more advanced deployment environments. It would also be
great if the implementation was considered optional, because I think
the standard deployment won't make use of this feature.
+1 for using the Qpid Java Client. It supports 0-8, 0-9 and 0-10 AMQP
specs, and they keep up with the AMQP spec development pretty well.
That would give a deployment options for which broker they use (which
might end up being RabbitMQ more often than not).
I think that any CloudStack reference architectures (install guides,
etc...) and testing should use QPID as the broker as well... the Java
broker seems to be the best option for project level testing (Although
we use the C++ broker internally).
I'd also be very interested in hearing more about the topic and
message taxonomy that you are thinking of implementing. I'm not able
to sign the ICLA yet, but I'm more than happy to help work through the
design for this functionality with you (on list of course). This is
actually a pretty important bit of functionality in my environment.
Thanks!
-chip
RE: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Abhinandan Prateek <Ab...@citrix.com>.
Planning to use apache Qpid implementation of AMQP for publishing cloudstack events.
Amazon like notifications system ala AWS SNS can be added to this eventually.
-abhi
>-----Original Message-----
>From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
>Sent: Friday, May 18, 2012 10:40 PM
>To: George Reese; cloudstack-dev@incubator.apache.org
>Cc: Nitin Mehta; Kishan Kavala; Dean
>Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>Questions )
>
>While embedding the publisher inside CloudStack could be one way to go, it is
>probably more immediately useful if we push events to a popular queueing
>system such as AMQP?
>Having CS have its own unique way of publishing events does make it more
>work for others to utilize these events.
>Agree on having a design that allows for other systems of notifications.
>
>From: George Reese
><ge...@enstratus.com>>
>Date: Fri, 18 May 2012 05:24:42 -0700
>To: "cloudstack-dev@incubator.apache.org<mailto:cloudstack-
>dev@incubator.apache.org>" <cloudstack-
>dev@incubator.apache.org<ma...@incubator.apache.org>>
>Cc: Nitin Mehta <Ni...@citrix.com>>,
>Kishan Kavala <Ki...@citrix.com>>,
>Dean <cl...@tizatron.com>>, Chiradeep Vittal
><ch...@citrix.com>>
>Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>Questions )
>
>Here's the way event pub sub should work...
>
>People interested in events will subscribe to them via API and CloudStack will
>maintain a record that includes:
>
>* What type of event(s) the subscriber cares about
>* The URI of an endpoint to notify on events
>* An optional API key for signing event publications
>
>If you want it to be really rich, you allow for multiple notifications endpoints,
>including SMS and email.
>
>When the existing event management system in CloudStack identifies a state
>change, it published it to a message queue.
>
>A new CloudStack component is subscribed to the message queue and pulls
>the state change off. It then sends the event to all interested parties. It does
>not keep delivery state. It sends and forgets. The only thing resembling state
>is the counting of failures, eventually killing an endpoint that fails to often.
>
>-George
>
>On May 17, 2012, at 10:53 PM, Abhinandan Prateek wrote:
>
>Cloudstack already have a well-defined set of events, currently being used by
>the Usage module.
>We can extend this to publish the events to registered listeners/subscribers.
>
>Nitin,
> We should evaluate the google guava package for this purpose.
>
>-abhi
>
>-----Original Message-----
>From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
>Sent: Saturday, May 12, 2012 3:48 AM
>To: cloudstack-dev@incubator.apache.org<mailto:cloudstack-
>dev@incubator.apache.org>
>Cc: Dean
>Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>Questions )
>
>+1.
>This could also be used for example to update DNS records on a DNS server
>whenever a VM is started / stopped in a network.
>A subscriber to the event bus can do this, and potentially things like updating a
>CMDB or an LDAP server.
>
>The important caveat is that this will be asynchronous to the stuff happening
>inside the management server: it has limited utility for doing stuff that is
>required to succeed before an API operation can succeed.
>For example, if a VM start requires a NAT rule to be programmed on the
>firewall before the start can be considered successful. You could write a event
>bus subscriber that does this, but the management server would not know if
>it was successful or not.
>
>There's at least a couple of entry points:
>1. com.cloud.utils.fsm.StateListener. Classes that implement this listener get
>notified on finite state event transitions. Examples include VM start / stop 2.
>com.cloud.network.element.NetworkElement. Classes that implement this
>get notified of network state transitions. Examples include adding nics and
>removing nics to/from a network.
>
>I think the first question regarding monitoring VMs on the isolated VLAN had a
>different origin though.
>The intention there was to have a monitoring service (e.g., Nagios) reach into
>the VM and monitor stuff like CPU, IO, or even applications.
>The issue there was around network reachability.
>
>--
>Chiradeep
>
>
>
>On 5/11/12 2:09 PM, "Adrian Cole"
><ad...@jclouds.org>> wrote:
>
>+1
>
>Makes sense to have pubsub. Inside the java codebase, we could consider a
>clean and idiomatic lib like guava which is easy to unit test.
>
>http://codingjunkie.net/guava-eventbus/
>
>Then, expose out-of-JVM hooks for any of the popular services people use.
>
>-A
>On May 11, 2012 1:58 PM, "Dean"
><cl...@tizatron.com>> wrote:
>
>Cross reference to:
>
>
>
>http://mail-archives.apache.org/mod_mbox/incubator-cloudstack-
>dev/201204.
>mbox/browser
>
>[ from: Marlon Davids ]
>< munch >
>2) How do we monitor VM's that are in Cloudstack when they are in an
>isolated VLAN does anyone have a clever workaround?
>3) Has anyone developed a script for parsing and alerting on warning events in
>the management Log yet?
>
>I would like to propose cloudstack consider a pub/sub model for event
>handling to complement API calls like listEvents.
>
>Polling can be problematic and sensitive to scaling.
>
>A simple example would be state change on a physical device. The admin
>server can simply publish a message on a network socket indicating that the
>device has changed it's state.
>
>If a subscriber was interested in that device, it could make an api call to the
>admin server for state change information for that device only.
>The
>admin server may choose to validate that physical device against the current
>state table in the database.
>
>The API would reply that this node changed it's state from Operational to
>Prep For Maintenance. (or whatever the transition state would be)
>
>The message exchange could be wrapped around vm states, resource
>additions/removals etc.
>
>Using a library like zeromq, a developer can write any number of consumers
>in any language they wanted to subscribe to the Event Bus.
>
>Comments?
>
>--
>-Dean
>
>
>--
>George Reese - Chief Technology Officer, enStratus
>e: george.reese@enstratus.com<ma...@enstratus.com>
>Skype: nspollution t: @GeorgeReese p: +1.207.956.0217
>enStratus: Enterprise Cloud Management - @enStratus -
>http://www.enstratus.com<http://www.enstratus.com/>
>To schedule a meeting with me: http://tungle.me/GeorgeReese
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack Questions )
Posted by George Reese <ge...@enstratus.com>.
You mean for something like enStratus to send one of its events to CloudStack to publish via the CloudStack event notifications?
I don't think so. I think it's best to keep this dead simple. Just notify interested party of CloudStack events, and I'd even start simple there with just state changes. Let it evolve from there.
-George
On May 18, 2012, at 8:00 AM, Murali Reddy wrote:
>
> Are there any use-cases for which it make sense to have publishers (that
> use CloudStack API to publish) other than CloudStack? If its not overkill
> we can have generic mechanism of publish/subscribe as service in
> CloudStack. I can think of tenants workload VM's be able to leverage this
> and publish something.
>
> On 18/05/12 5:54 PM, "George Reese" <ge...@enstratus.com> wrote:
>
>> Here's the way event pub sub should workŠ
>>
>> People interested in events will subscribe to them via API and CloudStack
>> will maintain a record that includes:
>>
>> * What type of event(s) the subscriber cares about
>> * The URI of an endpoint to notify on events
>> * An optional API key for signing event publications
>>
>> If you want it to be really rich, you allow for multiple notifications
>> endpoints, including SMS and email.
>>
>> When the existing event management system in CloudStack identifies a
>> state change, it published it to a message queue.
>>
>> A new CloudStack component is subscribed to the message queue and pulls
>> the state change off. It then sends the event to all interested parties.
>> It does not keep delivery state. It sends and forgets. The only thing
>> resembling state is the counting of failures, eventually killing an
>> endpoint that fails to often.
>>
>> -George
>>
>> On May 17, 2012, at 10:53 PM, Abhinandan Prateek wrote:
>>
>>> Cloudstack already have a well-defined set of events, currently being
>>> used by the Usage module.
>>> We can extend this to publish the events to registered
>>> listeners/subscribers.
>>>
>>> Nitin,
>>> We should evaluate the google guava package for this purpose.
>>>
>>> -abhi
>>>
>>>> -----Original Message-----
>>>> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
>>>> Sent: Saturday, May 12, 2012 3:48 AM
>>>> To: cloudstack-dev@incubator.apache.org
>>>> Cc: Dean
>>>> Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>>>> Questions )
>>>>
>>>> +1.
>>>> This could also be used for example to update DNS records on a DNS
>>>> server
>>>> whenever a VM is started / stopped in a network.
>>>> A subscriber to the event bus can do this, and potentially things like
>>>> updating a
>>>> CMDB or an LDAP server.
>>>>
>>>> The important caveat is that this will be asynchronous to the stuff
>>>> happening
>>>> inside the management server: it has limited utility for doing stuff
>>>> that is
>>>> required to succeed before an API operation can succeed.
>>>> For example, if a VM start requires a NAT rule to be programmed on the
>>>> firewall before the start can be considered successful. You could
>>>> write a event
>>>> bus subscriber that does this, but the management server would not
>>>> know if
>>>> it was successful or not.
>>>>
>>>> There's at least a couple of entry points:
>>>> 1. com.cloud.utils.fsm.StateListener. Classes that implement this
>>>> listener get
>>>> notified on finite state event transitions. Examples include VM start
>>>> / stop 2.
>>>> com.cloud.network.element.NetworkElement. Classes that implement this
>>>> get notified of network state transitions. Examples include adding
>>>> nics and
>>>> removing nics to/from a network.
>>>>
>>>> I think the first question regarding monitoring VMs on the isolated
>>>> VLAN had a
>>>> different origin though.
>>>> The intention there was to have a monitoring service (e.g., Nagios)
>>>> reach into
>>>> the VM and monitor stuff like CPU, IO, or even applications.
>>>> The issue there was around network reachability.
>>>>
>>>> --
>>>> Chiradeep
>>>>
>>>>
>>>>
>>>> On 5/11/12 2:09 PM, "Adrian Cole" <ad...@jclouds.org> wrote:
>>>>
>>>>> +1
>>>>>
>>>>> Makes sense to have pubsub. Inside the java codebase, we could
>>>>> consider a clean and idiomatic lib like guava which is easy to unit
>>>>> test.
>>>>>
>>>>> http://codingjunkie.net/guava-eventbus/
>>>>>
>>>>> Then, expose out-of-JVM hooks for any of the popular services people
>>>>> use.
>>>>>
>>>>> -A
>>>>> On May 11, 2012 1:58 PM, "Dean" <cl...@tizatron.com> wrote:
>>>>>
>>>>>> Cross reference to:
>>>>>>
>>>>>>
>>>>>>
>>>>>> http://mail-archives.apache.org/mod_mbox/incubator-cloudstack-
>>>> dev/201204.
>>>>>> mbox/browser
>>>>>>
>>>>>> [ from: Marlon Davids ]
>>>>>> < munch >
>>>>>>> 2) How do we monitor VM's that are in Cloudstack when they are in
>>>>>>> an
>>>>>> isolated VLAN does
>>>>>>> anyone have a clever workaround?
>>>>>>> 3) Has anyone developed a script for parsing and alerting on
>>>>>>> warning
>>>>>> events in the
>>>>>>> management Log yet?
>>>>>>
>>>>>> I would like to propose cloudstack consider a pub/sub model for event
>>>>>> handling to complement API calls like listEvents.
>>>>>>
>>>>>> Polling can be problematic and sensitive to scaling.
>>>>>>
>>>>>> A simple example would be state change on a physical device. The
>>>>>> admin server can simply publish a message on a network socket
>>>>>> indicating that the device has changed it's state.
>>>>>>
>>>>>> If a subscriber was interested in that device, it could make an api
>>>>>> call to the admin server for state change information for that
>>>>>> device only.
>>>>>> The
>>>>>> admin server may choose to validate that physical device against the
>>>>>> current state table in the database.
>>>>>>
>>>>>> The API would reply that this node changed it's state from
>>>>>> Operational to Prep For Maintenance. (or whatever the transition
>>>>>> state would be)
>>>>>>
>>>>>> The message exchange could be wrapped around vm states, resource
>>>>>> additions/removals etc.
>>>>>>
>>>>>> Using a library like zeromq, a developer can write any number of
>>>>>> consumers in any language they wanted to subscribe to the Event Bus.
>>>>>>
>>>>>> Comments?
>>>>>>
>>>>>> --
>>>>>> -Dean
>>>
>>
>> --
>> George Reese - Chief Technology Officer, enStratus
>> e: george.reese@enstratus.com Skype: nspollution t: @GeorgeReese
>> p: +1.207.956.0217
>> enStratus: Enterprise Cloud Management - @enStratus -
>> http://www.enstratus.com
>> To schedule a meeting with me: http://tungle.me/GeorgeReese
>>
>>
>
>
--
George Reese - Chief Technology Officer, enStratus
e: george.reese@enstratus.com Skype: nspollution t: @GeorgeReese p: +1.207.956.0217
enStratus: Enterprise Cloud Management - @enStratus - http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Murali Reddy <Mu...@citrix.com>.
Are there any use-cases for which it make sense to have publishers (that
use CloudStack API to publish) other than CloudStack? If its not overkill
we can have generic mechanism of publish/subscribe as service in
CloudStack. I can think of tenants workload VM's be able to leverage this
and publish something.
On 18/05/12 5:54 PM, "George Reese" <ge...@enstratus.com> wrote:
>Here's the way event pub sub should workŠ
>
>People interested in events will subscribe to them via API and CloudStack
>will maintain a record that includes:
>
>* What type of event(s) the subscriber cares about
>* The URI of an endpoint to notify on events
>* An optional API key for signing event publications
>
>If you want it to be really rich, you allow for multiple notifications
>endpoints, including SMS and email.
>
>When the existing event management system in CloudStack identifies a
>state change, it published it to a message queue.
>
>A new CloudStack component is subscribed to the message queue and pulls
>the state change off. It then sends the event to all interested parties.
>It does not keep delivery state. It sends and forgets. The only thing
>resembling state is the counting of failures, eventually killing an
>endpoint that fails to often.
>
>-George
>
>On May 17, 2012, at 10:53 PM, Abhinandan Prateek wrote:
>
>> Cloudstack already have a well-defined set of events, currently being
>>used by the Usage module.
>> We can extend this to publish the events to registered
>>listeners/subscribers.
>>
>> Nitin,
>> We should evaluate the google guava package for this purpose.
>>
>> -abhi
>>
>>> -----Original Message-----
>>> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
>>> Sent: Saturday, May 12, 2012 3:48 AM
>>> To: cloudstack-dev@incubator.apache.org
>>> Cc: Dean
>>> Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>>> Questions )
>>>
>>> +1.
>>> This could also be used for example to update DNS records on a DNS
>>>server
>>> whenever a VM is started / stopped in a network.
>>> A subscriber to the event bus can do this, and potentially things like
>>>updating a
>>> CMDB or an LDAP server.
>>>
>>> The important caveat is that this will be asynchronous to the stuff
>>>happening
>>> inside the management server: it has limited utility for doing stuff
>>>that is
>>> required to succeed before an API operation can succeed.
>>> For example, if a VM start requires a NAT rule to be programmed on the
>>> firewall before the start can be considered successful. You could
>>>write a event
>>> bus subscriber that does this, but the management server would not
>>>know if
>>> it was successful or not.
>>>
>>> There's at least a couple of entry points:
>>> 1. com.cloud.utils.fsm.StateListener. Classes that implement this
>>>listener get
>>> notified on finite state event transitions. Examples include VM start
>>>/ stop 2.
>>> com.cloud.network.element.NetworkElement. Classes that implement this
>>> get notified of network state transitions. Examples include adding
>>>nics and
>>> removing nics to/from a network.
>>>
>>> I think the first question regarding monitoring VMs on the isolated
>>>VLAN had a
>>> different origin though.
>>> The intention there was to have a monitoring service (e.g., Nagios)
>>>reach into
>>> the VM and monitor stuff like CPU, IO, or even applications.
>>> The issue there was around network reachability.
>>>
>>> --
>>> Chiradeep
>>>
>>>
>>>
>>> On 5/11/12 2:09 PM, "Adrian Cole" <ad...@jclouds.org> wrote:
>>>
>>>> +1
>>>>
>>>> Makes sense to have pubsub. Inside the java codebase, we could
>>>> consider a clean and idiomatic lib like guava which is easy to unit
>>>>test.
>>>>
>>>> http://codingjunkie.net/guava-eventbus/
>>>>
>>>> Then, expose out-of-JVM hooks for any of the popular services people
>>>>use.
>>>>
>>>> -A
>>>> On May 11, 2012 1:58 PM, "Dean" <cl...@tizatron.com> wrote:
>>>>
>>>>> Cross reference to:
>>>>>
>>>>>
>>>>>
>>>>> http://mail-archives.apache.org/mod_mbox/incubator-cloudstack-
>>> dev/201204.
>>>>> mbox/browser
>>>>>
>>>>> [ from: Marlon Davids ]
>>>>> < munch >
>>>>>> 2) How do we monitor VM's that are in Cloudstack when they are in
>>>>>> an
>>>>> isolated VLAN does
>>>>>> anyone have a clever workaround?
>>>>>> 3) Has anyone developed a script for parsing and alerting on
>>>>>> warning
>>>>> events in the
>>>>>> management Log yet?
>>>>>
>>>>> I would like to propose cloudstack consider a pub/sub model for event
>>>>> handling to complement API calls like listEvents.
>>>>>
>>>>> Polling can be problematic and sensitive to scaling.
>>>>>
>>>>> A simple example would be state change on a physical device. The
>>>>> admin server can simply publish a message on a network socket
>>>>> indicating that the device has changed it's state.
>>>>>
>>>>> If a subscriber was interested in that device, it could make an api
>>>>> call to the admin server for state change information for that
>>>>>device only.
>>>>> The
>>>>> admin server may choose to validate that physical device against the
>>>>> current state table in the database.
>>>>>
>>>>> The API would reply that this node changed it's state from
>>>>> Operational to Prep For Maintenance. (or whatever the transition
>>>>> state would be)
>>>>>
>>>>> The message exchange could be wrapped around vm states, resource
>>>>> additions/removals etc.
>>>>>
>>>>> Using a library like zeromq, a developer can write any number of
>>>>> consumers in any language they wanted to subscribe to the Event Bus.
>>>>>
>>>>> Comments?
>>>>>
>>>>> --
>>>>> -Dean
>>
>
>--
>George Reese - Chief Technology Officer, enStratus
>e: george.reese@enstratus.com Skype: nspollution t: @GeorgeReese
>p: +1.207.956.0217
>enStratus: Enterprise Cloud Management - @enStratus -
>http://www.enstratus.com
>To schedule a meeting with me: http://tungle.me/GeorgeReese
>
>
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Murali Reddy <Mu...@citrix.com>.
Are there any use-cases for which it make sense to have publishers (that
use CloudStack API to publish) other than CloudStack? If its not overkill
we can have generic mechanism of publish/subscribe as service in
CloudStack. I can think of tenants workload VM's be able to leverage this
and publish something to subscribers.
On 18/05/12 5:54 PM, "George Reese" <ge...@enstratus.com> wrote:
>Here's the way event pub sub should workŠ
>
>People interested in events will subscribe to them via API and CloudStack
>will maintain a record that includes:
>
>* What type of event(s) the subscriber cares about
>* The URI of an endpoint to notify on events
>* An optional API key for signing event publications
>
>If you want it to be really rich, you allow for multiple notifications
>endpoints, including SMS and email.
>
>When the existing event management system in CloudStack identifies a
>state change, it published it to a message queue.
>
>A new CloudStack component is subscribed to the message queue and pulls
>the state change off. It then sends the event to all interested parties.
>It does not keep delivery state. It sends and forgets. The only thing
>resembling state is the counting of failures, eventually killing an
>endpoint that fails to often.
>
>-George
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack Questions )
Posted by George Reese <ge...@enstratus.com>.
We're not looking for a message queue, we're looking for a notifications system ala AWS SNS. CloudStack needs to be able to *push* events out to endpoints; not send them to an MQ for interested parties to pull.
-George
On May 18, 2012, at 12:09 PM, Chiradeep Vittal wrote:
> While embedding the publisher inside CloudStack could be one way to go, it is probably more immediately useful if we push events to a popular queueing system such as AMQP?
> Having CS have its own unique way of publishing events does make it more work for others to utilize these events.
> Agree on having a design that allows for other systems of notifications.
>
> From: George Reese <ge...@enstratus.com>
> Date: Fri, 18 May 2012 05:24:42 -0700
> To: "cloudstack-dev@incubator.apache.org" <cl...@incubator.apache.org>
> Cc: Nitin Mehta <Ni...@citrix.com>, Kishan Kavala <Ki...@citrix.com>, Dean <cl...@tizatron.com>, Chiradeep Vittal <ch...@citrix.com>
> Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack Questions )
>
> Here's the way event pub sub should work…
>
> People interested in events will subscribe to them via API and CloudStack will maintain a record that includes:
>
> * What type of event(s) the subscriber cares about
> * The URI of an endpoint to notify on events
> * An optional API key for signing event publications
>
> If you want it to be really rich, you allow for multiple notifications endpoints, including SMS and email.
>
> When the existing event management system in CloudStack identifies a state change, it published it to a message queue.
>
> A new CloudStack component is subscribed to the message queue and pulls the state change off. It then sends the event to all interested parties. It does not keep delivery state. It sends and forgets. The only thing resembling state is the counting of failures, eventually killing an endpoint that fails to often.
>
> -George
>
> On May 17, 2012, at 10:53 PM, Abhinandan Prateek wrote:
>
>> Cloudstack already have a well-defined set of events, currently being used by the Usage module.
>> We can extend this to publish the events to registered listeners/subscribers.
>>
>> Nitin,
>> We should evaluate the google guava package for this purpose.
>>
>> -abhi
>>
>>> -----Original Message-----
>>> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
>>> Sent: Saturday, May 12, 2012 3:48 AM
>>> To: cloudstack-dev@incubator.apache.org
>>> Cc: Dean
>>> Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>>> Questions )
>>>
>>> +1.
>>> This could also be used for example to update DNS records on a DNS server
>>> whenever a VM is started / stopped in a network.
>>> A subscriber to the event bus can do this, and potentially things like updating a
>>> CMDB or an LDAP server.
>>>
>>> The important caveat is that this will be asynchronous to the stuff happening
>>> inside the management server: it has limited utility for doing stuff that is
>>> required to succeed before an API operation can succeed.
>>> For example, if a VM start requires a NAT rule to be programmed on the
>>> firewall before the start can be considered successful. You could write a event
>>> bus subscriber that does this, but the management server would not know if
>>> it was successful or not.
>>>
>>> There's at least a couple of entry points:
>>> 1. com.cloud.utils.fsm.StateListener. Classes that implement this listener get
>>> notified on finite state event transitions. Examples include VM start / stop 2.
>>> com.cloud.network.element.NetworkElement. Classes that implement this
>>> get notified of network state transitions. Examples include adding nics and
>>> removing nics to/from a network.
>>>
>>> I think the first question regarding monitoring VMs on the isolated VLAN had a
>>> different origin though.
>>> The intention there was to have a monitoring service (e.g., Nagios) reach into
>>> the VM and monitor stuff like CPU, IO, or even applications.
>>> The issue there was around network reachability.
>>>
>>> --
>>> Chiradeep
>>>
>>>
>>>
>>> On 5/11/12 2:09 PM, "Adrian Cole" <ad...@jclouds.org> wrote:
>>>
>>>> +1
>>>>
>>>> Makes sense to have pubsub. Inside the java codebase, we could
>>>> consider a clean and idiomatic lib like guava which is easy to unit test.
>>>>
>>>> http://codingjunkie.net/guava-eventbus/
>>>>
>>>> Then, expose out-of-JVM hooks for any of the popular services people use.
>>>>
>>>> -A
>>>> On May 11, 2012 1:58 PM, "Dean" <cl...@tizatron.com> wrote:
>>>>
>>>>> Cross reference to:
>>>>>
>>>>>
>>>>>
>>>>> http://mail-archives.apache.org/mod_mbox/incubator-cloudstack-
>>> dev/201204.
>>>>> mbox/browser
>>>>>
>>>>> [ from: Marlon Davids ]
>>>>> < munch >
>>>>>> 2) How do we monitor VM's that are in Cloudstack when they are in
>>>>>> an
>>>>> isolated VLAN does
>>>>>> anyone have a clever workaround?
>>>>>> 3) Has anyone developed a script for parsing and alerting on
>>>>>> warning
>>>>> events in the
>>>>>> management Log yet?
>>>>>
>>>>> I would like to propose cloudstack consider a pub/sub model for event
>>>>> handling to complement API calls like listEvents.
>>>>>
>>>>> Polling can be problematic and sensitive to scaling.
>>>>>
>>>>> A simple example would be state change on a physical device. The
>>>>> admin server can simply publish a message on a network socket
>>>>> indicating that the device has changed it's state.
>>>>>
>>>>> If a subscriber was interested in that device, it could make an api
>>>>> call to the admin server for state change information for that device only.
>>>>> The
>>>>> admin server may choose to validate that physical device against the
>>>>> current state table in the database.
>>>>>
>>>>> The API would reply that this node changed it's state from
>>>>> Operational to Prep For Maintenance. (or whatever the transition
>>>>> state would be)
>>>>>
>>>>> The message exchange could be wrapped around vm states, resource
>>>>> additions/removals etc.
>>>>>
>>>>> Using a library like zeromq, a developer can write any number of
>>>>> consumers in any language they wanted to subscribe to the Event Bus.
>>>>>
>>>>> Comments?
>>>>>
>>>>> --
>>>>> -Dean
>>
>
> --
> George Reese - Chief Technology Officer, enStratus
> e: george.reese@enstratus.com Skype: nspollution t: @GeorgeReese p: +1.207.956.0217
> enStratus: Enterprise Cloud Management - @enStratus - http://www.enstratus.com
> To schedule a meeting with me: http://tungle.me/GeorgeReese
>
--
George Reese - Chief Technology Officer, enStratus
e: george.reese@enstratus.com Skype: nspollution t: @GeorgeReese p: +1.207.956.0217
enStratus: Enterprise Cloud Management - @enStratus - http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Chiradeep Vittal <Ch...@citrix.com>.
While embedding the publisher inside CloudStack could be one way to go, it is probably more immediately useful if we push events to a popular queueing system such as AMQP?
Having CS have its own unique way of publishing events does make it more work for others to utilize these events.
Agree on having a design that allows for other systems of notifications.
From: George Reese <ge...@enstratus.com>>
Date: Fri, 18 May 2012 05:24:42 -0700
To: "cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>" <cl...@incubator.apache.org>>
Cc: Nitin Mehta <Ni...@citrix.com>>, Kishan Kavala <Ki...@citrix.com>>, Dean <cl...@tizatron.com>>, Chiradeep Vittal <ch...@citrix.com>>
Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack Questions )
Here's the way event pub sub should work…
People interested in events will subscribe to them via API and CloudStack will maintain a record that includes:
* What type of event(s) the subscriber cares about
* The URI of an endpoint to notify on events
* An optional API key for signing event publications
If you want it to be really rich, you allow for multiple notifications endpoints, including SMS and email.
When the existing event management system in CloudStack identifies a state change, it published it to a message queue.
A new CloudStack component is subscribed to the message queue and pulls the state change off. It then sends the event to all interested parties. It does not keep delivery state. It sends and forgets. The only thing resembling state is the counting of failures, eventually killing an endpoint that fails to often.
-George
On May 17, 2012, at 10:53 PM, Abhinandan Prateek wrote:
Cloudstack already have a well-defined set of events, currently being used by the Usage module.
We can extend this to publish the events to registered listeners/subscribers.
Nitin,
We should evaluate the google guava package for this purpose.
-abhi
-----Original Message-----
From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
Sent: Saturday, May 12, 2012 3:48 AM
To: cloudstack-dev@incubator.apache.org<ma...@incubator.apache.org>
Cc: Dean
Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
+1.
This could also be used for example to update DNS records on a DNS server
whenever a VM is started / stopped in a network.
A subscriber to the event bus can do this, and potentially things like updating a
CMDB or an LDAP server.
The important caveat is that this will be asynchronous to the stuff happening
inside the management server: it has limited utility for doing stuff that is
required to succeed before an API operation can succeed.
For example, if a VM start requires a NAT rule to be programmed on the
firewall before the start can be considered successful. You could write a event
bus subscriber that does this, but the management server would not know if
it was successful or not.
There's at least a couple of entry points:
1. com.cloud.utils.fsm.StateListener. Classes that implement this listener get
notified on finite state event transitions. Examples include VM start / stop 2.
com.cloud.network.element.NetworkElement. Classes that implement this
get notified of network state transitions. Examples include adding nics and
removing nics to/from a network.
I think the first question regarding monitoring VMs on the isolated VLAN had a
different origin though.
The intention there was to have a monitoring service (e.g., Nagios) reach into
the VM and monitor stuff like CPU, IO, or even applications.
The issue there was around network reachability.
--
Chiradeep
On 5/11/12 2:09 PM, "Adrian Cole" <ad...@jclouds.org>> wrote:
+1
Makes sense to have pubsub. Inside the java codebase, we could
consider a clean and idiomatic lib like guava which is easy to unit test.
http://codingjunkie.net/guava-eventbus/
Then, expose out-of-JVM hooks for any of the popular services people use.
-A
On May 11, 2012 1:58 PM, "Dean" <cl...@tizatron.com>> wrote:
Cross reference to:
http://mail-archives.apache.org/mod_mbox/incubator-cloudstack-
dev/201204.
mbox/browser
[ from: Marlon Davids ]
< munch >
2) How do we monitor VM's that are in Cloudstack when they are in
an
isolated VLAN does
anyone have a clever workaround?
3) Has anyone developed a script for parsing and alerting on
warning
events in the
management Log yet?
I would like to propose cloudstack consider a pub/sub model for event
handling to complement API calls like listEvents.
Polling can be problematic and sensitive to scaling.
A simple example would be state change on a physical device. The
admin server can simply publish a message on a network socket
indicating that the device has changed it's state.
If a subscriber was interested in that device, it could make an api
call to the admin server for state change information for that device only.
The
admin server may choose to validate that physical device against the
current state table in the database.
The API would reply that this node changed it's state from
Operational to Prep For Maintenance. (or whatever the transition
state would be)
The message exchange could be wrapped around vm states, resource
additions/removals etc.
Using a library like zeromq, a developer can write any number of
consumers in any language they wanted to subscribe to the Event Bus.
Comments?
--
-Dean
--
George Reese - Chief Technology Officer, enStratus
e: george.reese@enstratus.com<ma...@enstratus.com> Skype: nspollution t: @GeorgeReese p: +1.207.956.0217
enStratus: Enterprise Cloud Management - @enStratus - http://www.enstratus.com<http://www.enstratus.com/>
To schedule a meeting with me: http://tungle.me/GeorgeReese
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack Questions )
Posted by George Reese <ge...@enstratus.com>.
Here's the way event pub sub should work…
People interested in events will subscribe to them via API and CloudStack will maintain a record that includes:
* What type of event(s) the subscriber cares about
* The URI of an endpoint to notify on events
* An optional API key for signing event publications
If you want it to be really rich, you allow for multiple notifications endpoints, including SMS and email.
When the existing event management system in CloudStack identifies a state change, it published it to a message queue.
A new CloudStack component is subscribed to the message queue and pulls the state change off. It then sends the event to all interested parties. It does not keep delivery state. It sends and forgets. The only thing resembling state is the counting of failures, eventually killing an endpoint that fails to often.
-George
On May 17, 2012, at 10:53 PM, Abhinandan Prateek wrote:
> Cloudstack already have a well-defined set of events, currently being used by the Usage module.
> We can extend this to publish the events to registered listeners/subscribers.
>
> Nitin,
> We should evaluate the google guava package for this purpose.
>
> -abhi
>
>> -----Original Message-----
>> From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
>> Sent: Saturday, May 12, 2012 3:48 AM
>> To: cloudstack-dev@incubator.apache.org
>> Cc: Dean
>> Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>> Questions )
>>
>> +1.
>> This could also be used for example to update DNS records on a DNS server
>> whenever a VM is started / stopped in a network.
>> A subscriber to the event bus can do this, and potentially things like updating a
>> CMDB or an LDAP server.
>>
>> The important caveat is that this will be asynchronous to the stuff happening
>> inside the management server: it has limited utility for doing stuff that is
>> required to succeed before an API operation can succeed.
>> For example, if a VM start requires a NAT rule to be programmed on the
>> firewall before the start can be considered successful. You could write a event
>> bus subscriber that does this, but the management server would not know if
>> it was successful or not.
>>
>> There's at least a couple of entry points:
>> 1. com.cloud.utils.fsm.StateListener. Classes that implement this listener get
>> notified on finite state event transitions. Examples include VM start / stop 2.
>> com.cloud.network.element.NetworkElement. Classes that implement this
>> get notified of network state transitions. Examples include adding nics and
>> removing nics to/from a network.
>>
>> I think the first question regarding monitoring VMs on the isolated VLAN had a
>> different origin though.
>> The intention there was to have a monitoring service (e.g., Nagios) reach into
>> the VM and monitor stuff like CPU, IO, or even applications.
>> The issue there was around network reachability.
>>
>> --
>> Chiradeep
>>
>>
>>
>> On 5/11/12 2:09 PM, "Adrian Cole" <ad...@jclouds.org> wrote:
>>
>>> +1
>>>
>>> Makes sense to have pubsub. Inside the java codebase, we could
>>> consider a clean and idiomatic lib like guava which is easy to unit test.
>>>
>>> http://codingjunkie.net/guava-eventbus/
>>>
>>> Then, expose out-of-JVM hooks for any of the popular services people use.
>>>
>>> -A
>>> On May 11, 2012 1:58 PM, "Dean" <cl...@tizatron.com> wrote:
>>>
>>>> Cross reference to:
>>>>
>>>>
>>>>
>>>> http://mail-archives.apache.org/mod_mbox/incubator-cloudstack-
>> dev/201204.
>>>> mbox/browser
>>>>
>>>> [ from: Marlon Davids ]
>>>> < munch >
>>>>> 2) How do we monitor VM's that are in Cloudstack when they are in
>>>>> an
>>>> isolated VLAN does
>>>>> anyone have a clever workaround?
>>>>> 3) Has anyone developed a script for parsing and alerting on
>>>>> warning
>>>> events in the
>>>>> management Log yet?
>>>>
>>>> I would like to propose cloudstack consider a pub/sub model for event
>>>> handling to complement API calls like listEvents.
>>>>
>>>> Polling can be problematic and sensitive to scaling.
>>>>
>>>> A simple example would be state change on a physical device. The
>>>> admin server can simply publish a message on a network socket
>>>> indicating that the device has changed it's state.
>>>>
>>>> If a subscriber was interested in that device, it could make an api
>>>> call to the admin server for state change information for that device only.
>>>> The
>>>> admin server may choose to validate that physical device against the
>>>> current state table in the database.
>>>>
>>>> The API would reply that this node changed it's state from
>>>> Operational to Prep For Maintenance. (or whatever the transition
>>>> state would be)
>>>>
>>>> The message exchange could be wrapped around vm states, resource
>>>> additions/removals etc.
>>>>
>>>> Using a library like zeromq, a developer can write any number of
>>>> consumers in any language they wanted to subscribe to the Event Bus.
>>>>
>>>> Comments?
>>>>
>>>> --
>>>> -Dean
>
--
George Reese - Chief Technology Officer, enStratus
e: george.reese@enstratus.com Skype: nspollution t: @GeorgeReese p: +1.207.956.0217
enStratus: Enterprise Cloud Management - @enStratus - http://www.enstratus.com
To schedule a meeting with me: http://tungle.me/GeorgeReese
RE: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Abhinandan Prateek <Ab...@citrix.com>.
Cloudstack already have a well-defined set of events, currently being used by the Usage module.
We can extend this to publish the events to registered listeners/subscribers.
Nitin,
We should evaluate the google guava package for this purpose.
-abhi
>-----Original Message-----
>From: Chiradeep Vittal [mailto:Chiradeep.Vittal@citrix.com]
>Sent: Saturday, May 12, 2012 3:48 AM
>To: cloudstack-dev@incubator.apache.org
>Cc: Dean
>Subject: Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
>Questions )
>
>+1.
>This could also be used for example to update DNS records on a DNS server
>whenever a VM is started / stopped in a network.
>A subscriber to the event bus can do this, and potentially things like updating a
>CMDB or an LDAP server.
>
>The important caveat is that this will be asynchronous to the stuff happening
>inside the management server: it has limited utility for doing stuff that is
>required to succeed before an API operation can succeed.
>For example, if a VM start requires a NAT rule to be programmed on the
>firewall before the start can be considered successful. You could write a event
>bus subscriber that does this, but the management server would not know if
>it was successful or not.
>
>There's at least a couple of entry points:
>1. com.cloud.utils.fsm.StateListener. Classes that implement this listener get
>notified on finite state event transitions. Examples include VM start / stop 2.
>com.cloud.network.element.NetworkElement. Classes that implement this
>get notified of network state transitions. Examples include adding nics and
>removing nics to/from a network.
>
>I think the first question regarding monitoring VMs on the isolated VLAN had a
>different origin though.
>The intention there was to have a monitoring service (e.g., Nagios) reach into
>the VM and monitor stuff like CPU, IO, or even applications.
>The issue there was around network reachability.
>
>--
>Chiradeep
>
>
>
>On 5/11/12 2:09 PM, "Adrian Cole" <ad...@jclouds.org> wrote:
>
>>+1
>>
>>Makes sense to have pubsub. Inside the java codebase, we could
>>consider a clean and idiomatic lib like guava which is easy to unit test.
>>
>>http://codingjunkie.net/guava-eventbus/
>>
>>Then, expose out-of-JVM hooks for any of the popular services people use.
>>
>>-A
>>On May 11, 2012 1:58 PM, "Dean" <cl...@tizatron.com> wrote:
>>
>>> Cross reference to:
>>>
>>>
>>>
>>>http://mail-archives.apache.org/mod_mbox/incubator-cloudstack-
>dev/201204.
>>>mbox/browser
>>>
>>> [ from: Marlon Davids ]
>>> < munch >
>>> > 2) How do we monitor VM's that are in Cloudstack when they are in
>>> > an
>>> isolated VLAN does
>>> > anyone have a clever workaround?
>>> > 3) Has anyone developed a script for parsing and alerting on
>>> > warning
>>> events in the
>>> > management Log yet?
>>>
>>> I would like to propose cloudstack consider a pub/sub model for event
>>> handling to complement API calls like listEvents.
>>>
>>> Polling can be problematic and sensitive to scaling.
>>>
>>> A simple example would be state change on a physical device. The
>>>admin server can simply publish a message on a network socket
>>>indicating that the device has changed it's state.
>>>
>>> If a subscriber was interested in that device, it could make an api
>>>call to the admin server for state change information for that device only.
>>>The
>>> admin server may choose to validate that physical device against the
>>>current state table in the database.
>>>
>>> The API would reply that this node changed it's state from
>>>Operational to Prep For Maintenance. (or whatever the transition
>>>state would be)
>>>
>>> The message exchange could be wrapped around vm states, resource
>>> additions/removals etc.
>>>
>>> Using a library like zeromq, a developer can write any number of
>>>consumers in any language they wanted to subscribe to the Event Bus.
>>>
>>> Comments?
>>>
>>> --
>>> -Dean
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Chiradeep Vittal <Ch...@citrix.com>.
+1.
This could also be used for example to update DNS records on a DNS server
whenever a VM is started / stopped in a network.
A subscriber to the event bus can do this, and potentially things like
updating a CMDB or an LDAP server.
The important caveat is that this will be asynchronous to the stuff
happening inside the management server: it has limited utility for doing
stuff that is required to succeed before an API operation can succeed.
For example, if a VM start requires a NAT rule to be programmed on the
firewall before the start can be considered successful. You could write a
event bus subscriber that does this, but the management server would not
know if it was successful or not.
There's at least a couple of entry points:
1. com.cloud.utils.fsm.StateListener. Classes that implement this listener
get notified on finite state event transitions. Examples include VM start
/ stop
2. com.cloud.network.element.NetworkElement. Classes that implement this
get notified of network state transitions. Examples include adding nics
and removing nics to/from a network.
I think the first question regarding monitoring VMs on the isolated VLAN
had a different origin though.
The intention there was to have a monitoring service (e.g., Nagios) reach
into the VM and monitor stuff like CPU, IO, or even applications.
The issue there was around network reachability.
--
Chiradeep
On 5/11/12 2:09 PM, "Adrian Cole" <ad...@jclouds.org> wrote:
>+1
>
>Makes sense to have pubsub. Inside the java codebase, we could consider a
>clean and idiomatic lib like guava which is easy to unit test.
>
>http://codingjunkie.net/guava-eventbus/
>
>Then, expose out-of-JVM hooks for any of the popular services people use.
>
>-A
>On May 11, 2012 1:58 PM, "Dean" <cl...@tizatron.com> wrote:
>
>> Cross reference to:
>>
>>
>>
>>http://mail-archives.apache.org/mod_mbox/incubator-cloudstack-dev/201204.
>>mbox/browser
>>
>> [ from: Marlon Davids ]
>> < munch >
>> > 2) How do we monitor VM's that are in Cloudstack when they are in an
>> isolated VLAN does
>> > anyone have a clever workaround?
>> > 3) Has anyone developed a script for parsing and alerting on warning
>> events in the
>> > management Log yet?
>>
>> I would like to propose cloudstack consider a pub/sub model for event
>> handling to complement API calls like listEvents.
>>
>> Polling can be problematic and sensitive to scaling.
>>
>> A simple example would be state change on a physical device. The admin
>> server can simply publish a message on a network socket indicating that
>>the
>> device has changed it's state.
>>
>> If a subscriber was interested in that device, it could make an api call
>> to the admin server for state change information for that device only.
>>The
>> admin server may choose to validate that physical device against the
>> current state table in the database.
>>
>> The API would reply that this node changed it's state from Operational
>>to
>> Prep For Maintenance. (or whatever the transition state would be)
>>
>> The message exchange could be wrapped around vm states, resource
>> additions/removals etc.
>>
>> Using a library like zeromq, a developer can write any number of
>>consumers
>> in any language they wanted to subscribe to the Event Bus.
>>
>> Comments?
>>
>> --
>> -Dean
Re: Event Publish and Subscribe ( was perhaps Re: Cloudstack
Questions )
Posted by Adrian Cole <ad...@jclouds.org>.
+1
Makes sense to have pubsub. Inside the java codebase, we could consider a
clean and idiomatic lib like guava which is easy to unit test.
http://codingjunkie.net/guava-eventbus/
Then, expose out-of-JVM hooks for any of the popular services people use.
-A
On May 11, 2012 1:58 PM, "Dean" <cl...@tizatron.com> wrote:
> Cross reference to:
>
>
> http://mail-archives.apache.org/mod_mbox/incubator-cloudstack-dev/201204.mbox/browser
>
> [ from: Marlon Davids ]
> < munch >
> > 2) How do we monitor VM's that are in Cloudstack when they are in an
> isolated VLAN does
> > anyone have a clever workaround?
> > 3) Has anyone developed a script for parsing and alerting on warning
> events in the
> > management Log yet?
>
> I would like to propose cloudstack consider a pub/sub model for event
> handling to complement API calls like listEvents.
>
> Polling can be problematic and sensitive to scaling.
>
> A simple example would be state change on a physical device. The admin
> server can simply publish a message on a network socket indicating that the
> device has changed it's state.
>
> If a subscriber was interested in that device, it could make an api call
> to the admin server for state change information for that device only. The
> admin server may choose to validate that physical device against the
> current state table in the database.
>
> The API would reply that this node changed it's state from Operational to
> Prep For Maintenance. (or whatever the transition state would be)
>
> The message exchange could be wrapped around vm states, resource
> additions/removals etc.
>
> Using a library like zeromq, a developer can write any number of consumers
> in any language they wanted to subscribe to the Event Bus.
>
> Comments?
>
> --
> -Dean