Improving StatsListener api so that implementing one for multiple operators is clean

[Chandni Singh <ch...@datatorrent.com>]

Hi,

Recently while implementing a stats listener which is set on 2 different
logical operators, specifically used to slow down an upstream operator
using the stats from downstream partitions, I realize the code gets
unwieldy.

A reason for this is that there is no effortless way of knowing which
logical operator the stats belong to. BatchedOperatorStats contains
operator id but that is assigned at the runtime (unknown at the time of
writing code).

I had created the following ticket to address this address this and would
like to work on it.
https://malhar.atlassian.net/browse/APEX-69

Thanks,
Chandni

Re: Improving StatsListener api so that implementing one for multiple operators is clean

[Chandni Singh <ch...@datatorrent.com>]

Below is an example of code where the same StatsListener is used for
multiple operators. If we propose this as a solution to slow down upstream
operator then we need to improve this api so that the implementation is
simplified.

This implementation can become much more complicated if operator 2 & 3 can
dynamically partition themselves.

public class MyListener implements StatsListener
{
  @Override
  public Response processStats(BatchedOperatorStats stats)
  {
    String operatorName;

    Stats.OperatorStats operatorStats =
stats.getLastWindowedStats().iterator().next();

    //this is a hack
    if (operatorStats.metrics.containsKey("operator1metric")) {
      //stats belong to operator1 metric
      operatorName = "operator1";
    } else if (operatorStats.metrics.containsKey("operator2metric")) {
      operatorName = "operator2";
    } else {
      operatorName = "operator3";
    }

    Response response = new Response();
    //Now based on the operator we need to send a Response. Lets say
based on stats of operator 2 & operator 3 we
    // need to change a property of operator 1.
    switch (operatorName) {
      case "operator1" :
          response.operatorCommands.add(new SetChangeOperator1());
          return response;
      case "operator2" :
          return null;
      case "operator3" :
          return null;
    }
  }
}


On Wed, Sep 2, 2015 at 8:03 PM, Chetan Narsude <ch...@datatorrent.com>
wrote:

> I commented on the ticket. May be we should configure the jira so that by
> default the comments on the ticket go to dev@apex?
>
> On Wed, Sep 2, 2015 at 6:58 PM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > Hi,
> >
> > Recently while implementing a stats listener which is set on 2 different
> > logical operators, specifically used to slow down an upstream operator
> > using the stats from downstream partitions, I realize the code gets
> > unwieldy.
> >
> > A reason for this is that there is no effortless way of knowing which
> > logical operator the stats belong to. BatchedOperatorStats contains
> > operator id but that is assigned at the runtime (unknown at the time of
> > writing code).
> >
> > I had created the following ticket to address this address this and would
> > like to work on it.
> > https://malhar.atlassian.net/browse/APEX-69
> >
> > Thanks,
> > Chandni
> >
>

Re: Improving StatsListener api so that implementing one for multiple operators is clean

[Chetan Narsude <ch...@datatorrent.com>]

I commented on the ticket. May be we should configure the jira so that by
default the comments on the ticket go to dev@apex?

On Wed, Sep 2, 2015 at 6:58 PM, Chandni Singh <ch...@datatorrent.com>
wrote:

> Hi,
>
> Recently while implementing a stats listener which is set on 2 different
> logical operators, specifically used to slow down an upstream operator
> using the stats from downstream partitions, I realize the code gets
> unwieldy.
>
> A reason for this is that there is no effortless way of knowing which
> logical operator the stats belong to. BatchedOperatorStats contains
> operator id but that is assigned at the runtime (unknown at the time of
> writing code).
>
> I had created the following ticket to address this address this and would
> like to work on it.
> https://malhar.atlassian.net/browse/APEX-69
>
> Thanks,
> Chandni
>