You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Sandesh Hegde <sa...@datatorrent.com> on 2016/08/10 05:58:36 UTC
[Proposal] DAG listener
Hi All,
As we add more features to support batch use cases, there will be a need to
access more information about the DAG from an operator. One example is the need
to access the operator name while using Batched StatsListener.
The idea here is to implement DAG Listener ( similar to StatsListener ),
which can potentially give every information present in the Stram.
Operators will also have visibility into all the major activities in the
DAG.
Thoughts?
Thanks
Re: [Proposal] DAG listener
Posted by Sandesh Hegde <sa...@datatorrent.com>.
@Sanjay
Here is the issue I mentioned
https://issues.apache.org/jira/browse/APEXCORE-496
Instead of exposing one information at a time, why not publish all the
information?
To start with it could just be Physical DAG and update the subscribers
after partitioning.
Thanks
On Wed, Aug 10, 2016 at 10:40 AM Sanjay Pujare <sa...@datatorrent.com>
wrote:
> Looks like a good idea but would like more details about the use case
> (“example ... to access the operator name while using Batched
> StatsListener”).
>
> Do you propose very fine grained subscription model? Also what “major
> activities in the DAG” will be available thru this listener?
>
>
>
> On 8/10/16, 8:20 AM, "Sandesh Hegde" <sa...@datatorrent.com> wrote:
>
> Any operators can subscribe to Stram Events affecting the DAG.
>
> Implementation will most probably use heartbeat.
>
> On Tue, Aug 9, 2016 at 11:43 PM Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > Hi Sandesh,
> >
> > Dag changes are handled in Stram, and operators are running in
> > different containers.
> > Are you suggesting an RPC interface or operator request for sending
> > this information
> > from Stram to all partitions of the interested operator?
> >
> > - Tushar.
> >
> >
> >
> > On Wed, Aug 10, 2016 at 11:28 AM, Sandesh Hegde <
> sandesh@datatorrent.com>
> > wrote:
> > > Hi All,
> > >
> > > As we add more features to support batch use cases, there will be
> a need
> > to
> > > access more information about the DAG from an operator. One
> example is
> > the need
> > > to access the operator name while using Batched StatsListener.
> > >
> > > The idea here is to implement DAG Listener ( similar to
> StatsListener ),
> > > which can potentially give every information present in the Stram.
> > > Operators will also have visibility into all the major activities
> in the
> > > DAG.
> > >
> > > Thoughts?
> > >
> > > Thanks
> >
>
>
>
>
Re: [Proposal] DAG listener
Posted by Sanjay Pujare <sa...@datatorrent.com>.
Looks like a good idea but would like more details about the use case (“example ... to access the operator name while using Batched StatsListener”).
Do you propose very fine grained subscription model? Also what “major activities in the DAG” will be available thru this listener?
On 8/10/16, 8:20 AM, "Sandesh Hegde" <sa...@datatorrent.com> wrote:
Any operators can subscribe to Stram Events affecting the DAG.
Implementation will most probably use heartbeat.
On Tue, Aug 9, 2016 at 11:43 PM Tushar Gosavi <tu...@datatorrent.com>
wrote:
> Hi Sandesh,
>
> Dag changes are handled in Stram, and operators are running in
> different containers.
> Are you suggesting an RPC interface or operator request for sending
> this information
> from Stram to all partitions of the interested operator?
>
> - Tushar.
>
>
>
> On Wed, Aug 10, 2016 at 11:28 AM, Sandesh Hegde <sa...@datatorrent.com>
> wrote:
> > Hi All,
> >
> > As we add more features to support batch use cases, there will be a need
> to
> > access more information about the DAG from an operator. One example is
> the need
> > to access the operator name while using Batched StatsListener.
> >
> > The idea here is to implement DAG Listener ( similar to StatsListener ),
> > which can potentially give every information present in the Stram.
> > Operators will also have visibility into all the major activities in the
> > DAG.
> >
> > Thoughts?
> >
> > Thanks
>
Re: [Proposal] DAG listener
Posted by Sandesh Hegde <sa...@datatorrent.com>.
Any operators can subscribe to Stram Events affecting the DAG.
Implementation will most probably use heartbeat.
On Tue, Aug 9, 2016 at 11:43 PM Tushar Gosavi <tu...@datatorrent.com>
wrote:
> Hi Sandesh,
>
> Dag changes are handled in Stram, and operators are running in
> different containers.
> Are you suggesting an RPC interface or operator request for sending
> this information
> from Stram to all partitions of the interested operator?
>
> - Tushar.
>
>
>
> On Wed, Aug 10, 2016 at 11:28 AM, Sandesh Hegde <sa...@datatorrent.com>
> wrote:
> > Hi All,
> >
> > As we add more features to support batch use cases, there will be a need
> to
> > access more information about the DAG from an operator. One example is
> the need
> > to access the operator name while using Batched StatsListener.
> >
> > The idea here is to implement DAG Listener ( similar to StatsListener ),
> > which can potentially give every information present in the Stram.
> > Operators will also have visibility into all the major activities in the
> > DAG.
> >
> > Thoughts?
> >
> > Thanks
>
Re: [Proposal] DAG listener
Posted by Tushar Gosavi <tu...@datatorrent.com>.
Hi Sandesh,
Dag changes are handled in Stram, and operators are running in
different containers.
Are you suggesting an RPC interface or operator request for sending
this information
from Stram to all partitions of the interested operator?
- Tushar.
On Wed, Aug 10, 2016 at 11:28 AM, Sandesh Hegde <sa...@datatorrent.com> wrote:
> Hi All,
>
> As we add more features to support batch use cases, there will be a need to
> access more information about the DAG from an operator. One example is the need
> to access the operator name while using Batched StatsListener.
>
> The idea here is to implement DAG Listener ( similar to StatsListener ),
> which can potentially give every information present in the Stram.
> Operators will also have visibility into all the major activities in the
> DAG.
>
> Thoughts?
>
> Thanks