Hue UI & status/SLA events notification.

[Alejandro Abdelnur <tu...@cloudera.com>]

First of all thank you all for the presentations in yesterday's meetup.
Also very interesting discussions.

On the Hue UI, I think a great start would be to have a one page
documentation detailing how to install HUE and configure Oozie and Hue so
they work. Romain, Abraham, would you follow up with a JIRA for this?

Now I'd like to follow on the discussion on event notifications (do we have
a JIRA for it already, if so it would make sense to post this as comment
there):

* SLA and status events should be handled in the exact same way.
* All entities (BLD/COORD/WF * job/action) status changes should be capable
of generating events
* A server configuration should allow disabling event generation for
certain entity types (ie WF action)
* XCommands should publish events thru an EventService, the signature of
the publishEvent() method should take the EntityManager instance in context
of the XCommand (even if the current implementation that not use it). It
would be the publishEvent() the one filtering out events for entities that
are not to be published doing a NOP for them.
* The EventService should store the received events in a concurrent queue
* The EventService should register a runnable (running every 5 secs) in the
SchedulerService that drains the queue, and creates a PublishEventsXCommand
with all the drained events from the queue. Each batch of events would have
a batch ID.
* The PublishEventsXCommand would use an pluggable EventPublisher for which
there could be  different implementations: DBWriter, JMSPublisher,
EmailSender
* The PublishEventsXCommand, once the EventPublisher completes, it should
ACK the EventService with the batch ID (to allow clean up of any
persistency the EventService may have)


Note that with this proposal, we don't have to have a different threadpool
and workers, we just leverage the CommandQueue (because it has throttling
capabilities built in event commands will be capped). The CommandQueue size
and threadpool has to be increased as needed (rather than duplicating all
this logic).

The first impl of the EventService would be in-memory, later we can add
either a DB or edit-log based one.

Furthermore, i would be inclined to move the SLAEvents servlet to the
monitoring system.

Thoughts?

-- 
Alejandro

Re: Hue UI & status/SLA events notification.

[Robert Kanter <rk...@cloudera.com>]

+ Romain, Abe in case they're not on dev@oozie

For the Hue thing, would it make sense to add a maven profile that would
pull in the stripped-down version of Hue that just has the Oozie dashboard?
 Then users who want to use it can simply build Oozie with that profile and
are ready to go.  In the documentation for this, we can add a link to the
Hue project as installing the full Hue product requires more setup, a
database, etc.

For the event notification stuff, there's a JIRA here:
OOZIE-929<https://issues.apache.org/jira/browse/OOZIE-929>;
the ppt from the meetup is also there now.

- Robert


On Wed, Mar 13, 2013 at 11:56 AM, Alejandro Abdelnur <tu...@cloudera.com>wrote:

> First of all thank you all for the presentations in yesterday's meetup.
> Also very interesting discussions.
>
> On the Hue UI, I think a great start would be to have a one page
> documentation detailing how to install HUE and configure Oozie and Hue so
> they work. Romain, Abraham, would you follow up with a JIRA for this?
>
> Now I'd like to follow on the discussion on event notifications (do we have
> a JIRA for it already, if so it would make sense to post this as comment
> there):
>
> * SLA and status events should be handled in the exact same way.
> * All entities (BLD/COORD/WF * job/action) status changes should be capable
> of generating events
> * A server configuration should allow disabling event generation for
> certain entity types (ie WF action)
> * XCommands should publish events thru an EventService, the signature of
> the publishEvent() method should take the EntityManager instance in context
> of the XCommand (even if the current implementation that not use it). It
> would be the publishEvent() the one filtering out events for entities that
> are not to be published doing a NOP for them.
> * The EventService should store the received events in a concurrent queue
> * The EventService should register a runnable (running every 5 secs) in the
> SchedulerService that drains the queue, and creates a PublishEventsXCommand
> with all the drained events from the queue. Each batch of events would have
> a batch ID.
> * The PublishEventsXCommand would use an pluggable EventPublisher for which
> there could be  different implementations: DBWriter, JMSPublisher,
> EmailSender
> * The PublishEventsXCommand, once the EventPublisher completes, it should
> ACK the EventService with the batch ID (to allow clean up of any
> persistency the EventService may have)
>
>
> Note that with this proposal, we don't have to have a different threadpool
> and workers, we just leverage the CommandQueue (because it has throttling
> capabilities built in event commands will be capped). The CommandQueue size
> and threadpool has to be increased as needed (rather than duplicating all
> this logic).
>
> The first impl of the EventService would be in-memory, later we can add
> either a DB or edit-log based one.
>
> Furthermore, i would be inclined to move the SLAEvents servlet to the
> monitoring system.
>
> Thoughts?
>
> --
> Alejandro
>