Subscribing to Aurora's tasks' event changes

[Dmitriy Shirchenko <ca...@gmail.com>]

Has anyone built something that can subscribe to events and then send them
to a pub/sub system? Maybe can give pointers on how you would approach this?

Our use case is sending TaskStateChange`s into an internal Kafka topic.

Thanks!

Re: Subscribing to Aurora's tasks' event changes

[Bill Farner <wf...@apache.org>]

Fwiw I would favor a first-class mechanism instead.  I think pointing to
auth modules as a successful pattern is a mistake.

On Monday, April 25, 2016, Zameer Manji <zm...@apache.org> wrote:

> I think a good approach to take would be to have `PubsubEventModule` take
> modules as an arg like `HttpSecurityModule`. Those modules can create
> instances of `EventSubscriber` and those subscribers can consume the events
> you desire and do whatever you want.
>
> I think this is a low touch approach that is flexible and aligned with how
> we do other things.
>
> On Mon, Apr 25, 2016 at 4:40 PM, Dmitriy Shirchenko <caldima@gmail.com
> <javascript:;>>
> wrote:
>
> > Has anyone built something that can subscribe to events and then send
> them
> > to a pub/sub system? Maybe can give pointers on how you would approach
> > this?
> >
> > Our use case is sending TaskStateChange`s into an internal Kafka topic.
> >
> > Thanks!
> >
> > --
> > Zameer Manji
> >
> >
>

Re: Subscribing to Aurora's tasks' event changes

[Zameer Manji <zm...@apache.org>]

I think a good approach to take would be to have `PubsubEventModule` take
modules as an arg like `HttpSecurityModule`. Those modules can create
instances of `EventSubscriber` and those subscribers can consume the events
you desire and do whatever you want.

I think this is a low touch approach that is flexible and aligned with how
we do other things.

On Mon, Apr 25, 2016 at 4:40 PM, Dmitriy Shirchenko <ca...@gmail.com>
wrote:

> Has anyone built something that can subscribe to events and then send them
> to a pub/sub system? Maybe can give pointers on how you would approach
> this?
>
> Our use case is sending TaskStateChange`s into an internal Kafka topic.
>
> Thanks!
>
> --
> Zameer Manji
>
>

Re: Subscribing to Aurora's tasks' event changes

[John Sirois <jo...@gmail.com>]

On Thu, Apr 28, 2016 at 1:03 PM, Jeff Schroeder <je...@computer.org>
wrote:

> On Monday, April 25, 2016, John Sirois <jo...@gmail.com> wrote:
>
> > On Mon, Apr 25, 2016 at 8:15 PM, John Sirois <john.sirois@gmail.com
> > <javascript:;>> wrote:
> >
> > >
> > >
> > > On Mon, Apr 25, 2016 at 7:58 PM, Bill Farner <wfarner@apache.org
> > <javascript:;>> wrote:
> > >
> > >> I don't have a firm example in mind, I just don't think the approach
> > >> recommended by Zameer is optimal.  It's only marginally better than
> > >> forking
> > >> (and probably requires you to fork for the sake of sanity).
> > >>
> > >> Thinking out loud, it would be nice if Kafka had a JMS
> interface/bridge,
> > >> as
> > >> it would allow us to add support for a bunch of backends with one
> > >> implementation.  Unfortunately that does not appear to be the case.
> > >>
> > >
> > > That said - supporting a JMS interface doesn't sound too bad on the
> > aurora
> > > end.
> > > Motivated consumers could write a bridge presumably if they insist on
> > > using Kafka.
> > >
> >
> > Or maybe even just a webhook API.  You configure aurora with an HTTP
> > endpoint that must conform to a given api and Aurora tries, best effort
> > only, to post events to the endpoint, perhaps on a streaming connection.
> > This scales well for Aurora.
>
>
>
> Or Server Send Events, an often overlooked but really nice way of having a
> server send clients events in realtime over http. I *think* this is the way
> marathon sends events to marathon-lb (haven't looked though)
>

The reason I leaned towards 1 webhook url configured on the aurora
scheduler command line was paranoia over load mainly.
This assures aurora it will only have to service 1 best-effort streaming
client instead of an arbitrary number, and it assures aurora that only
operators can choose to enable this instead of random joe dev #9165 out of
12000 at some huge company.
I guess that concern is also easily assuaged though via admin-only api for
example.


>
>
>
>
>
> --
> Text by Jeff, typos by iPhone
>

Re: Subscribing to Aurora's tasks' event changes

[Jeff Schroeder <je...@computer.org>]

On Monday, April 25, 2016, John Sirois <jo...@gmail.com> wrote:

> On Mon, Apr 25, 2016 at 8:15 PM, John Sirois <john.sirois@gmail.com
> <javascript:;>> wrote:
>
> >
> >
> > On Mon, Apr 25, 2016 at 7:58 PM, Bill Farner <wfarner@apache.org
> <javascript:;>> wrote:
> >
> >> I don't have a firm example in mind, I just don't think the approach
> >> recommended by Zameer is optimal.  It's only marginally better than
> >> forking
> >> (and probably requires you to fork for the sake of sanity).
> >>
> >> Thinking out loud, it would be nice if Kafka had a JMS interface/bridge,
> >> as
> >> it would allow us to add support for a bunch of backends with one
> >> implementation.  Unfortunately that does not appear to be the case.
> >>
> >
> > That said - supporting a JMS interface doesn't sound too bad on the
> aurora
> > end.
> > Motivated consumers could write a bridge presumably if they insist on
> > using Kafka.
> >
>
> Or maybe even just a webhook API.  You configure aurora with an HTTP
> endpoint that must conform to a given api and Aurora tries, best effort
> only, to post events to the endpoint, perhaps on a streaming connection.
> This scales well for Aurora.



Or Server Send Events, an often overlooked but really nice way of having a
server send clients events in realtime over http. I *think* this is the way
marathon sends events to marathon-lb (haven't looked though)





-- 
Text by Jeff, typos by iPhone

Re: Subscribing to Aurora's tasks' event changes

[David McLaughlin <dm...@apache.org>]

+1 for webhooks.

On Mon, Apr 25, 2016 at 7:18 PM, John Sirois <jo...@gmail.com> wrote:

> On Mon, Apr 25, 2016 at 8:15 PM, John Sirois <jo...@gmail.com>
> wrote:
>
> >
> >
> > On Mon, Apr 25, 2016 at 7:58 PM, Bill Farner <wf...@apache.org> wrote:
> >
> >> I don't have a firm example in mind, I just don't think the approach
> >> recommended by Zameer is optimal.  It's only marginally better than
> >> forking
> >> (and probably requires you to fork for the sake of sanity).
> >>
> >> Thinking out loud, it would be nice if Kafka had a JMS interface/bridge,
> >> as
> >> it would allow us to add support for a bunch of backends with one
> >> implementation.  Unfortunately that does not appear to be the case.
> >>
> >
> > That said - supporting a JMS interface doesn't sound too bad on the
> aurora
> > end.
> > Motivated consumers could write a bridge presumably if they insist on
> > using Kafka.
> >
>
> Or maybe even just a webhook API.  You configure aurora with an HTTP
> endpoint that must conform to a given api and Aurora tries, best effort
> only, to post events to the endpoint, perhaps on a streaming connection.
> This scales well for Aurora.
>
>
> >
> >
> >> On Monday, April 25, 2016, Dmitriy Shirchenko <ca...@gmail.com>
> wrote:
> >>
> >> > @wfarner
> >> >
> >> > Can you help me out and clarify what you mean by 'First-class
> >> mechanism'?
> >> > An example would be awesome.
> >> >
> >> > On Mon, Apr 25, 2016 at 4:40 PM Dmitriy Shirchenko <caldima@gmail.com
> >> > <javascript:;>>
> >> > wrote:
> >> >
> >> > > Has anyone built something that can subscribe to events and then
> send
> >> > them
> >> > > to a pub/sub system? Maybe can give pointers on how you would
> approach
> >> > this?
> >> > >
> >> > > Our use case is sending TaskStateChange`s into an internal Kafka
> >> topic.
> >> > >
> >> > > Thanks!
> >> > >
> >> >
> >>
> >
> >
>

Re: Subscribing to Aurora's tasks' event changes

[John Sirois <jo...@gmail.com>]

On Mon, Apr 25, 2016 at 8:15 PM, John Sirois <jo...@gmail.com> wrote:

>
>
> On Mon, Apr 25, 2016 at 7:58 PM, Bill Farner <wf...@apache.org> wrote:
>
>> I don't have a firm example in mind, I just don't think the approach
>> recommended by Zameer is optimal.  It's only marginally better than
>> forking
>> (and probably requires you to fork for the sake of sanity).
>>
>> Thinking out loud, it would be nice if Kafka had a JMS interface/bridge,
>> as
>> it would allow us to add support for a bunch of backends with one
>> implementation.  Unfortunately that does not appear to be the case.
>>
>
> That said - supporting a JMS interface doesn't sound too bad on the aurora
> end.
> Motivated consumers could write a bridge presumably if they insist on
> using Kafka.
>

Or maybe even just a webhook API.  You configure aurora with an HTTP
endpoint that must conform to a given api and Aurora tries, best effort
only, to post events to the endpoint, perhaps on a streaming connection.
This scales well for Aurora.


>
>
>> On Monday, April 25, 2016, Dmitriy Shirchenko <ca...@gmail.com> wrote:
>>
>> > @wfarner
>> >
>> > Can you help me out and clarify what you mean by 'First-class
>> mechanism'?
>> > An example would be awesome.
>> >
>> > On Mon, Apr 25, 2016 at 4:40 PM Dmitriy Shirchenko <caldima@gmail.com
>> > <javascript:;>>
>> > wrote:
>> >
>> > > Has anyone built something that can subscribe to events and then send
>> > them
>> > > to a pub/sub system? Maybe can give pointers on how you would approach
>> > this?
>> > >
>> > > Our use case is sending TaskStateChange`s into an internal Kafka
>> topic.
>> > >
>> > > Thanks!
>> > >
>> >
>>
>
>

Re: Subscribing to Aurora's tasks' event changes

[John Sirois <jo...@gmail.com>]

On Mon, Apr 25, 2016 at 7:58 PM, Bill Farner <wf...@apache.org> wrote:

> I don't have a firm example in mind, I just don't think the approach
> recommended by Zameer is optimal.  It's only marginally better than forking
> (and probably requires you to fork for the sake of sanity).
>
> Thinking out loud, it would be nice if Kafka had a JMS interface/bridge, as
> it would allow us to add support for a bunch of backends with one
> implementation.  Unfortunately that does not appear to be the case.
>

That said - supporting a JMS interface doesn't sound too bad on the aurora
end.
Motivated consumers could write a bridge presumably if they insist on using
Kafka.


> On Monday, April 25, 2016, Dmitriy Shirchenko <ca...@gmail.com> wrote:
>
> > @wfarner
> >
> > Can you help me out and clarify what you mean by 'First-class mechanism'?
> > An example would be awesome.
> >
> > On Mon, Apr 25, 2016 at 4:40 PM Dmitriy Shirchenko <caldima@gmail.com
> > <javascript:;>>
> > wrote:
> >
> > > Has anyone built something that can subscribe to events and then send
> > them
> > > to a pub/sub system? Maybe can give pointers on how you would approach
> > this?
> > >
> > > Our use case is sending TaskStateChange`s into an internal Kafka topic.
> > >
> > > Thanks!
> > >
> >
>

Re: Subscribing to Aurora's tasks' event changes

[Bill Farner <wf...@apache.org>]

I don't have a firm example in mind, I just don't think the approach
recommended by Zameer is optimal.  It's only marginally better than forking
(and probably requires you to fork for the sake of sanity).

Thinking out loud, it would be nice if Kafka had a JMS interface/bridge, as
it would allow us to add support for a bunch of backends with one
implementation.  Unfortunately that does not appear to be the case.

On Monday, April 25, 2016, Dmitriy Shirchenko <ca...@gmail.com> wrote:

> @wfarner
>
> Can you help me out and clarify what you mean by 'First-class mechanism'?
> An example would be awesome.
>
> On Mon, Apr 25, 2016 at 4:40 PM Dmitriy Shirchenko <caldima@gmail.com
> <javascript:;>>
> wrote:
>
> > Has anyone built something that can subscribe to events and then send
> them
> > to a pub/sub system? Maybe can give pointers on how you would approach
> this?
> >
> > Our use case is sending TaskStateChange`s into an internal Kafka topic.
> >
> > Thanks!
> >
>

Re: Subscribing to Aurora's tasks' event changes

[Dmitriy Shirchenko <ca...@gmail.com>]

Ok, I updated the task [1] and assigned it to myself. I am onboard with
adding a webhook which will POST to our Kafka REST Proxy.

Thanks everyone.

[1] https://issues.apache.org/jira/browse/AURORA-1683

On Mon, Apr 25, 2016 at 7:18 PM, John Sirois <jo...@gmail.com> wrote:

> On Mon, Apr 25, 2016 at 8:15 PM, John Sirois <jo...@gmail.com>
> wrote:
>
> >
> >
> > On Mon, Apr 25, 2016 at 7:58 PM, Bill Farner <wf...@apache.org> wrote:
> >
> >> I don't have a firm example in mind, I just don't think the approach
> >> recommended by Zameer is optimal.  It's only marginally better than
> >> forking
> >> (and probably requires you to fork for the sake of sanity).
> >>
> >> Thinking out loud, it would be nice if Kafka had a JMS
interface/bridge,
> >> as
> >> it would allow us to add support for a bunch of backends with one
> >> implementation.  Unfortunately that does not appear to be the case.
> >>
> >
> > That said - supporting a JMS interface doesn't sound too bad on the
> aurora
> > end.
> > Motivated consumers could write a bridge presumably if they insist on
> > using Kafka.
> >
>
> Or maybe even just a webhook API.  You configure aurora with an HTTP
> endpoint that must conform to a given api and Aurora tries, best effort
> only, to post events to the endpoint, perhaps on a streaming connection.
> This scales well for Aurora.
>
>
> >
> >
> >> On Monday, April 25, 2016, Dmitriy Shirchenko <ca...@gmail.com>
> wrote:
> >>
> >> > @wfarner
> >> >
> >> > Can you help me out and clarify what you mean by 'First-class
> >> mechanism'?
> >> > An example would be awesome.
> >> >
> >> > On Mon, Apr 25, 2016 at 4:40 PM Dmitriy Shirchenko <caldima@gmail.com
> >> > <javascript:;>>
> >> > wrote:
> >> >
> >> > > Has anyone built something that can subscribe to events and then
> send
> >> > them
> >> > > to a pub/sub system? Maybe can give pointers on how you would
> approach
> >> > this?
> >> > >
> >> > > Our use case is sending TaskStateChange`s into an internal Kafka
> >> topic.
> >> > >
> >> > > Thanks!
> >> > >
> >> >
> >>
> >
> >
>

On Mon, Apr 25, 2016 at 6:42 PM Dmitriy Shirchenko <ca...@gmail.com>
wrote:

> @wfarner
>
> Can you help me out and clarify what you mean by 'First-class mechanism'?
> An example would be awesome.
>
> On Mon, Apr 25, 2016 at 4:40 PM Dmitriy Shirchenko <ca...@gmail.com>
> wrote:
>
>> Has anyone built something that can subscribe to events and then send
>> them to a pub/sub system? Maybe can give pointers on how you would approach
>> this?
>>
>> Our use case is sending TaskStateChange`s into an internal Kafka topic.
>>
>> Thanks!
>>
>

Re: Subscribing to Aurora's tasks' event changes

[Dmitriy Shirchenko <ca...@gmail.com>]

@wfarner

Can you help me out and clarify what you mean by 'First-class mechanism'?
An example would be awesome.

On Mon, Apr 25, 2016 at 4:40 PM Dmitriy Shirchenko <ca...@gmail.com>
wrote:

> Has anyone built something that can subscribe to events and then send them
> to a pub/sub system? Maybe can give pointers on how you would approach this?
>
> Our use case is sending TaskStateChange`s into an internal Kafka topic.
>
> Thanks!
>