You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@apex.apache.org by Priyanka Gugale <pr...@datatorrent.com> on 2015/09/22 14:37:24 UTC

Assign property to partition instances of Operator

I have an integer property named "intProp". This property is applicable to
an operator A. I can set the property from dtcli using
"set-operator-property operator-name property-name property-value" and this
will get applied to all instances of opeartor A.

Now, I want to apply this property in a way that it's equally divided
amongst all instances of operator A. For example,  if intProp=100 and
number of partitions=4 then intProp=25 should be set on each instance of
operator A.

Is there a way to achieve this?

I have tried following approach to do this: I have overriden
definePartitions of Operator, once we do partitioning I am setting right
value on each operator instance based on new partition count. The
limitation here is that the value is applied during definePartitions. If
the value is updated dynamically it won't be applied till definePartitions
is called again.

-Priyanka

Re: Assign property to partition instances of Operator

Posted by Gaurav Gupta <ga...@datatorrent.com>.

Priyanka,

Since you are not changing the number of partitions why will manually
setting the value help here?

Thanks
-Gaurav

On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <pr...@datatorrent.com>
wrote:

> Promod,
>
> You are right about applying the value in same window. And yes I don't want
> to change number of partitions or even redeploy any partition for this.
> I need to explore on Chandini's suggestion of using StatsListener.
>
> -Priyanka
>
> On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > I am not sure about priyanka's requirements but typically you would want
> to
> > apply the value in the same window for all partitions for consistency
> with
> > the calculations. This could be done two ways using a separate logical
> > operator feeding the value to all partitions so that the value appears in
> > the same window or repartitioning the existing set and setting the value
> > during repartitioning. In the latter case you are just repartitioning so
> > that the partitions start from same point with the new value and not to
> > change the number of partitions.
> >
> > Thanks
> >
> > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <chandni@datatorrent.com
> >
> > wrote:
> >
> > > Pramod,
> > >
> > > Tim's discussion was about setting a property on multiple operators
> > > (different logical operators).
> > > Priyanka's question is about different partitions of an operator
> > >
> > > Let's say they are 5 partitions of an operator and we need to change an
> > int
> > > property.  There are following ways:
> > >
> > > 1. Divide the int property by 5 and issue set-physical operator
> property
> > > command for each partition. This is manual.
> > >
> > > 2. Set the logical operator property but wait till partition happens.
> > This
> > > will require partitioning to be triggered for which a StatsListener
> needs
> > > to be implemented. So if partitioning never happens this will never be
> > > triggered. Also un-deploying and deploying a partition just to change a
> > > property on the physical partition seems un-necessary
> > >
> > > 3. Implementing StatsListener to send operator request and not
> > re-partition
> > > seems to be appropriate way.
> > >
> > > Chandni
> > >
> > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > > > Priyanka you want to divide the value among existing partitions right
> > and
> > > > not necessarily change the number of partitions. To this end Ashwin
> is
> > > > suggesting makes sense. However different partitions can be at
> > different
> > > > windows and will see the value change in different windows. You need
> a
> > > > separate operator to receive this change and propagate it to the
> > > partitions
> > > > via data flow. Tim discussed this over the last weekend in a series
> of
> > > > emails.
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > > > priyanka@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Thanks for suggestions.
> > > > >
> > > > > Ashwin, The option you proposed is doable, but then each operator
> > which
> > > > > needs to address such use case will have to track the number of
> > > > partitions.
> > > > > So we were looking for some common mechanism.
> > > > > Chandni, Let me try the res.operatorRequests, that could solve the
> > > > problem.
> > > > >
> > > > > -Priyanka
> > > > >
> > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > > chandni@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Priyanka,
> > > > > >
> > > > > > As Isha mentioned you will can implement a StatsListener. However
> > > > instead
> > > > > > of setting res.repartitionRequired = true, you can add a
> > > > > > StatsListener.OperatorRequest to res.operatorsRequests. This will
> > not
> > > > > > re-deploy the instances.
> > > > > >
> > > > > > The OperatorRequest that you will need to implement will have to
> > > change
> > > > > the
> > > > > > property value.
> > > > > >
> > > > > > You can refer to AutoMetricTest for an example.
> > > > > >
> > > > > > - Chandni
> > > > > >
> > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > > > > > ashwinchandrap@gmail.com> wrote:
> > > > > >
> > > > > > > Why not maintain a variable containing the number of partitions
> > in
> > > > each
> > > > > > > partition; and when you set the property, just divide it by
> > number
> > > of
> > > > > > > partitions variable? Does it not solve the problem?
> > > > > > >
> > > > > > > Regards,
> > > > > > > Ashwin.
> > > > > > >
> > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > > > > priyanka@datatorrent.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I have an integer property named "intProp". This property is
> > > > > applicable
> > > > > > > to
> > > > > > > > an operator A. I can set the property from dtcli using
> > > > > > > > "set-operator-property operator-name property-name
> > > property-value"
> > > > > and
> > > > > > > this
> > > > > > > > will get applied to all instances of opeartor A.
> > > > > > > >
> > > > > > > > Now, I want to apply this property in a way that it's equally
> > > > divided
> > > > > > > > amongst all instances of operator A. For example,  if
> > intProp=100
> > > > and
> > > > > > > > number of partitions=4 then intProp=25 should be set on each
> > > > instance
> > > > > > of
> > > > > > > > operator A.
> > > > > > > >
> > > > > > > > Is there a way to achieve this?
> > > > > > > >
> > > > > > > > I have tried following approach to do this: I have overriden
> > > > > > > > definePartitions of Operator, once we do partitioning I am
> > > setting
> > > > > > right
> > > > > > > > value on each operator instance based on new partition count.
> > The
> > > > > > > > limitation here is that the value is applied during
> > > > definePartitions.
> > > > > > If
> > > > > > > > the value is updated dynamically it won't be applied till
> > > > > > > definePartitions
> > > > > > > > is called again.
> > > > > > > >
> > > > > > > > -Priyanka
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Regards,
> > > > > > > Ashwin.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Pramod Immaneni <pr...@datatorrent.com>.

Yes we should explore supporting this in the engine.

On Wed, Sep 23, 2015 at 9:22 AM, Chandni Singh <ch...@datatorrent.com>
wrote:

> Adding another operator to feed property changes to all the partitions
> requires more work. Same partition keys need to be assigned to all the
> partitions to receive all the property change triggers.
> Using a partitioner will result in re-deployment.
>
> All the approaches seem to be an overkill for this simple task. Can we not
> simplify this for the users?
>
> Chandni
>
> On Wed, Sep 23, 2015 at 12:15 AM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > Typically the window difference wouldn't be large and if its happening
> its
> > probably indicator of a larger problem in the application anyway.
> >
> > On Wed, Sep 23, 2015 at 12:07 AM, Gaurav Gupta <ga...@datatorrent.com>
> > wrote:
> >
> > > Is that acceptable? as user may want to tune operator A now so that it
> > can
> > > catch up
> > >
> > > Thanks
> > > -Gaurav
> > >
> > > On Wed, Sep 23, 2015 at 12:04 AM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > > > Yes.
> > > >
> > > > On Wed, Sep 23, 2015 at 12:03 AM, Gaurav Gupta <
> gaurav@datatorrent.com
> > >
> > > > wrote:
> > > >
> > > > > Pramod,
> > > > >
> > > > > What happens if the Separate Operator that is used to propagate
> this
> > > > change
> > > > > is on say window 1000 and all the instances of Operator A are at
> > > 300..At
> > > > > this time user changes value of property. Then the new value will
> be
> > > > > delayed to Operator A instances by 700 windows
> > > > >
> > > > > Thanks
> > > > > -Gaurav
> > > > >
> > > > > On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <
> > gaurav@datatorrent.com
> > > >
> > > > > wrote:
> > > > >
> > > > > > Pramod ,
> > > > > >
> > > > > > That is what I was thinking that Operator requests can also be
> > > received
> > > > > in
> > > > > > different windows...Looks like adding another operator is safe
> > > solution
> > > > > >
> > > > > > Thanks
> > > > > > -Gaurav
> > > > > >
> > > > > > On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <
> > > > > pramod@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > >> The operator request would also be received in different
> windows.
> > > > > >>
> > > > > >> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
> > > > > >> priyanka@datatorrent.com>
> > > > > >> wrote:
> > > > > >>
> > > > > >> > Promod,
> > > > > >> >
> > > > > >> > You are right about applying the value in same window. And
> yes I
> > > > don't
> > > > > >> want
> > > > > >> > to change number of partitions or even redeploy any partition
> > for
> > > > > this.
> > > > > >> > I need to explore on Chandini's suggestion of using
> > StatsListener.
> > > > > >> >
> > > > > >> > -Priyanka
> > > > > >> >
> > > > > >> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
> > > > > >> pramod@datatorrent.com>
> > > > > >> > wrote:
> > > > > >> >
> > > > > >> > > I am not sure about priyanka's requirements but typically
> you
> > > > would
> > > > > >> want
> > > > > >> > to
> > > > > >> > > apply the value in the same window for all partitions for
> > > > > consistency
> > > > > >> > with
> > > > > >> > > the calculations. This could be done two ways using a
> separate
> > > > > logical
> > > > > >> > > operator feeding the value to all partitions so that the
> value
> > > > > >> appears in
> > > > > >> > > the same window or repartitioning the existing set and
> setting
> > > the
> > > > > >> value
> > > > > >> > > during repartitioning. In the latter case you are just
> > > > > repartitioning
> > > > > >> so
> > > > > >> > > that the partitions start from same point with the new value
> > and
> > > > not
> > > > > >> to
> > > > > >> > > change the number of partitions.
> > > > > >> > >
> > > > > >> > > Thanks
> > > > > >> > >
> > > > > >> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
> > > > > >> chandni@datatorrent.com
> > > > > >> > >
> > > > > >> > > wrote:
> > > > > >> > >
> > > > > >> > > > Pramod,
> > > > > >> > > >
> > > > > >> > > > Tim's discussion was about setting a property on multiple
> > > > > operators
> > > > > >> > > > (different logical operators).
> > > > > >> > > > Priyanka's question is about different partitions of an
> > > operator
> > > > > >> > > >
> > > > > >> > > > Let's say they are 5 partitions of an operator and we need
> > to
> > > > > >> change an
> > > > > >> > > int
> > > > > >> > > > property.  There are following ways:
> > > > > >> > > >
> > > > > >> > > > 1. Divide the int property by 5 and issue set-physical
> > > operator
> > > > > >> > property
> > > > > >> > > > command for each partition. This is manual.
> > > > > >> > > >
> > > > > >> > > > 2. Set the logical operator property but wait till
> partition
> > > > > >> happens.
> > > > > >> > > This
> > > > > >> > > > will require partitioning to be triggered for which a
> > > > > StatsListener
> > > > > >> > needs
> > > > > >> > > > to be implemented. So if partitioning never happens this
> > will
> > > > > never
> > > > > >> be
> > > > > >> > > > triggered. Also un-deploying and deploying a partition
> just
> > to
> > > > > >> change a
> > > > > >> > > > property on the physical partition seems un-necessary
> > > > > >> > > >
> > > > > >> > > > 3. Implementing StatsListener to send operator request and
> > not
> > > > > >> > > re-partition
> > > > > >> > > > seems to be appropriate way.
> > > > > >> > > >
> > > > > >> > > > Chandni
> > > > > >> > > >
> > > > > >> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> > > > > >> > > pramod@datatorrent.com>
> > > > > >> > > > wrote:
> > > > > >> > > >
> > > > > >> > > > > Priyanka you want to divide the value among existing
> > > > partitions
> > > > > >> right
> > > > > >> > > and
> > > > > >> > > > > not necessarily change the number of partitions. To this
> > end
> > > > > >> Ashwin
> > > > > >> > is
> > > > > >> > > > > suggesting makes sense. However different partitions can
> > be
> > > at
> > > > > >> > > different
> > > > > >> > > > > windows and will see the value change in different
> > windows.
> > > > You
> > > > > >> need
> > > > > >> > a
> > > > > >> > > > > separate operator to receive this change and propagate
> it
> > to
> > > > the
> > > > > >> > > > partitions
> > > > > >> > > > > via data flow. Tim discussed this over the last weekend
> > in a
> > > > > >> series
> > > > > >> > of
> > > > > >> > > > > emails.
> > > > > >> > > > >
> > > > > >> > > > > Thanks
> > > > > >> > > > >
> > > > > >> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > > > > >> > > > > priyanka@datatorrent.com>
> > > > > >> > > > > wrote:
> > > > > >> > > > >
> > > > > >> > > > > > Thanks for suggestions.
> > > > > >> > > > > >
> > > > > >> > > > > > Ashwin, The option you proposed is doable, but then
> each
> > > > > >> operator
> > > > > >> > > which
> > > > > >> > > > > > needs to address such use case will have to track the
> > > number
> > > > > of
> > > > > >> > > > > partitions.
> > > > > >> > > > > > So we were looking for some common mechanism.
> > > > > >> > > > > > Chandni, Let me try the res.operatorRequests, that
> could
> > > > solve
> > > > > >> the
> > > > > >> > > > > problem.
> > > > > >> > > > > >
> > > > > >> > > > > > -Priyanka
> > > > > >> > > > > >
> > > > > >> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > > > > >> > > > chandni@datatorrent.com>
> > > > > >> > > > > > wrote:
> > > > > >> > > > > >
> > > > > >> > > > > > > Hi Priyanka,
> > > > > >> > > > > > >
> > > > > >> > > > > > > As Isha mentioned you will can implement a
> > > StatsListener.
> > > > > >> However
> > > > > >> > > > > instead
> > > > > >> > > > > > > of setting res.repartitionRequired = true, you can
> > add a
> > > > > >> > > > > > > StatsListener.OperatorRequest to
> > res.operatorsRequests.
> > > > This
> > > > > >> will
> > > > > >> > > not
> > > > > >> > > > > > > re-deploy the instances.
> > > > > >> > > > > > >
> > > > > >> > > > > > > The OperatorRequest that you will need to implement
> > will
> > > > > have
> > > > > >> to
> > > > > >> > > > change
> > > > > >> > > > > > the
> > > > > >> > > > > > > property value.
> > > > > >> > > > > > >
> > > > > >> > > > > > > You can refer to AutoMetricTest for an example.
> > > > > >> > > > > > >
> > > > > >> > > > > > > - Chandni
> > > > > >> > > > > > >
> > > > > >> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra
> Putta
> > <
> > > > > >> > > > > > > ashwinchandrap@gmail.com> wrote:
> > > > > >> > > > > > >
> > > > > >> > > > > > > > Why not maintain a variable containing the number
> of
> > > > > >> partitions
> > > > > >> > > in
> > > > > >> > > > > each
> > > > > >> > > > > > > > partition; and when you set the property, just
> > divide
> > > it
> > > > > by
> > > > > >> > > number
> > > > > >> > > > of
> > > > > >> > > > > > > > partitions variable? Does it not solve the
> problem?
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Regards,
> > > > > >> > > > > > > > Ashwin.
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > > > >> > > > > > > priyanka@datatorrent.com
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > wrote:
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > > I have an integer property named "intProp". This
> > > > > property
> > > > > >> is
> > > > > >> > > > > > applicable
> > > > > >> > > > > > > > to
> > > > > >> > > > > > > > > an operator A. I can set the property from dtcli
> > > using
> > > > > >> > > > > > > > > "set-operator-property operator-name
> property-name
> > > > > >> > > > property-value"
> > > > > >> > > > > > and
> > > > > >> > > > > > > > this
> > > > > >> > > > > > > > > will get applied to all instances of opeartor A.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Now, I want to apply this property in a way that
> > > it's
> > > > > >> equally
> > > > > >> > > > > divided
> > > > > >> > > > > > > > > amongst all instances of operator A. For
> example,
> > > if
> > > > > >> > > intProp=100
> > > > > >> > > > > and
> > > > > >> > > > > > > > > number of partitions=4 then intProp=25 should be
> > set
> > > > on
> > > > > >> each
> > > > > >> > > > > instance
> > > > > >> > > > > > > of
> > > > > >> > > > > > > > > operator A.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > Is there a way to achieve this?
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > I have tried following approach to do this: I
> have
> > > > > >> overriden
> > > > > >> > > > > > > > > definePartitions of Operator, once we do
> > > partitioning
> > > > I
> > > > > am
> > > > > >> > > > setting
> > > > > >> > > > > > > right
> > > > > >> > > > > > > > > value on each operator instance based on new
> > > partition
> > > > > >> count.
> > > > > >> > > The
> > > > > >> > > > > > > > > limitation here is that the value is applied
> > during
> > > > > >> > > > > definePartitions.
> > > > > >> > > > > > > If
> > > > > >> > > > > > > > > the value is updated dynamically it won't be
> > applied
> > > > > till
> > > > > >> > > > > > > > definePartitions
> > > > > >> > > > > > > > > is called again.
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > > > -Priyanka
> > > > > >> > > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > --
> > > > > >> > > > > > > >
> > > > > >> > > > > > > > Regards,
> > > > > >> > > > > > > > Ashwin.
> > > > > >> > > > > > > >
> > > > > >> > > > > > >
> > > > > >> > > > > >
> > > > > >> > > > >
> > > > > >> > > >
> > > > > >> > >
> > > > > >> >
> > > > > >>
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Chandni Singh <ch...@datatorrent.com>.

Adding another operator to feed property changes to all the partitions
requires more work. Same partition keys need to be assigned to all the
partitions to receive all the property change triggers.
Using a partitioner will result in re-deployment.

All the approaches seem to be an overkill for this simple task. Can we not
simplify this for the users?

Chandni

On Wed, Sep 23, 2015 at 12:15 AM, Pramod Immaneni <pr...@datatorrent.com>
wrote:

> Typically the window difference wouldn't be large and if its happening its
> probably indicator of a larger problem in the application anyway.
>
> On Wed, Sep 23, 2015 at 12:07 AM, Gaurav Gupta <ga...@datatorrent.com>
> wrote:
>
> > Is that acceptable? as user may want to tune operator A now so that it
> can
> > catch up
> >
> > Thanks
> > -Gaurav
> >
> > On Wed, Sep 23, 2015 at 12:04 AM, Pramod Immaneni <
> pramod@datatorrent.com>
> > wrote:
> >
> > > Yes.
> > >
> > > On Wed, Sep 23, 2015 at 12:03 AM, Gaurav Gupta <gaurav@datatorrent.com
> >
> > > wrote:
> > >
> > > > Pramod,
> > > >
> > > > What happens if the Separate Operator that is used to propagate this
> > > change
> > > > is on say window 1000 and all the instances of Operator A are at
> > 300..At
> > > > this time user changes value of property. Then the new value will be
> > > > delayed to Operator A instances by 700 windows
> > > >
> > > > Thanks
> > > > -Gaurav
> > > >
> > > > On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <
> gaurav@datatorrent.com
> > >
> > > > wrote:
> > > >
> > > > > Pramod ,
> > > > >
> > > > > That is what I was thinking that Operator requests can also be
> > received
> > > > in
> > > > > different windows...Looks like adding another operator is safe
> > solution
> > > > >
> > > > > Thanks
> > > > > -Gaurav
> > > > >
> > > > > On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <
> > > > pramod@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > >> The operator request would also be received in different windows.
> > > > >>
> > > > >> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
> > > > >> priyanka@datatorrent.com>
> > > > >> wrote:
> > > > >>
> > > > >> > Promod,
> > > > >> >
> > > > >> > You are right about applying the value in same window. And yes I
> > > don't
> > > > >> want
> > > > >> > to change number of partitions or even redeploy any partition
> for
> > > > this.
> > > > >> > I need to explore on Chandini's suggestion of using
> StatsListener.
> > > > >> >
> > > > >> > -Priyanka
> > > > >> >
> > > > >> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
> > > > >> pramod@datatorrent.com>
> > > > >> > wrote:
> > > > >> >
> > > > >> > > I am not sure about priyanka's requirements but typically you
> > > would
> > > > >> want
> > > > >> > to
> > > > >> > > apply the value in the same window for all partitions for
> > > > consistency
> > > > >> > with
> > > > >> > > the calculations. This could be done two ways using a separate
> > > > logical
> > > > >> > > operator feeding the value to all partitions so that the value
> > > > >> appears in
> > > > >> > > the same window or repartitioning the existing set and setting
> > the
> > > > >> value
> > > > >> > > during repartitioning. In the latter case you are just
> > > > repartitioning
> > > > >> so
> > > > >> > > that the partitions start from same point with the new value
> and
> > > not
> > > > >> to
> > > > >> > > change the number of partitions.
> > > > >> > >
> > > > >> > > Thanks
> > > > >> > >
> > > > >> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
> > > > >> chandni@datatorrent.com
> > > > >> > >
> > > > >> > > wrote:
> > > > >> > >
> > > > >> > > > Pramod,
> > > > >> > > >
> > > > >> > > > Tim's discussion was about setting a property on multiple
> > > > operators
> > > > >> > > > (different logical operators).
> > > > >> > > > Priyanka's question is about different partitions of an
> > operator
> > > > >> > > >
> > > > >> > > > Let's say they are 5 partitions of an operator and we need
> to
> > > > >> change an
> > > > >> > > int
> > > > >> > > > property.  There are following ways:
> > > > >> > > >
> > > > >> > > > 1. Divide the int property by 5 and issue set-physical
> > operator
> > > > >> > property
> > > > >> > > > command for each partition. This is manual.
> > > > >> > > >
> > > > >> > > > 2. Set the logical operator property but wait till partition
> > > > >> happens.
> > > > >> > > This
> > > > >> > > > will require partitioning to be triggered for which a
> > > > StatsListener
> > > > >> > needs
> > > > >> > > > to be implemented. So if partitioning never happens this
> will
> > > > never
> > > > >> be
> > > > >> > > > triggered. Also un-deploying and deploying a partition just
> to
> > > > >> change a
> > > > >> > > > property on the physical partition seems un-necessary
> > > > >> > > >
> > > > >> > > > 3. Implementing StatsListener to send operator request and
> not
> > > > >> > > re-partition
> > > > >> > > > seems to be appropriate way.
> > > > >> > > >
> > > > >> > > > Chandni
> > > > >> > > >
> > > > >> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> > > > >> > > pramod@datatorrent.com>
> > > > >> > > > wrote:
> > > > >> > > >
> > > > >> > > > > Priyanka you want to divide the value among existing
> > > partitions
> > > > >> right
> > > > >> > > and
> > > > >> > > > > not necessarily change the number of partitions. To this
> end
> > > > >> Ashwin
> > > > >> > is
> > > > >> > > > > suggesting makes sense. However different partitions can
> be
> > at
> > > > >> > > different
> > > > >> > > > > windows and will see the value change in different
> windows.
> > > You
> > > > >> need
> > > > >> > a
> > > > >> > > > > separate operator to receive this change and propagate it
> to
> > > the
> > > > >> > > > partitions
> > > > >> > > > > via data flow. Tim discussed this over the last weekend
> in a
> > > > >> series
> > > > >> > of
> > > > >> > > > > emails.
> > > > >> > > > >
> > > > >> > > > > Thanks
> > > > >> > > > >
> > > > >> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > > > >> > > > > priyanka@datatorrent.com>
> > > > >> > > > > wrote:
> > > > >> > > > >
> > > > >> > > > > > Thanks for suggestions.
> > > > >> > > > > >
> > > > >> > > > > > Ashwin, The option you proposed is doable, but then each
> > > > >> operator
> > > > >> > > which
> > > > >> > > > > > needs to address such use case will have to track the
> > number
> > > > of
> > > > >> > > > > partitions.
> > > > >> > > > > > So we were looking for some common mechanism.
> > > > >> > > > > > Chandni, Let me try the res.operatorRequests, that could
> > > solve
> > > > >> the
> > > > >> > > > > problem.
> > > > >> > > > > >
> > > > >> > > > > > -Priyanka
> > > > >> > > > > >
> > > > >> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > > > >> > > > chandni@datatorrent.com>
> > > > >> > > > > > wrote:
> > > > >> > > > > >
> > > > >> > > > > > > Hi Priyanka,
> > > > >> > > > > > >
> > > > >> > > > > > > As Isha mentioned you will can implement a
> > StatsListener.
> > > > >> However
> > > > >> > > > > instead
> > > > >> > > > > > > of setting res.repartitionRequired = true, you can
> add a
> > > > >> > > > > > > StatsListener.OperatorRequest to
> res.operatorsRequests.
> > > This
> > > > >> will
> > > > >> > > not
> > > > >> > > > > > > re-deploy the instances.
> > > > >> > > > > > >
> > > > >> > > > > > > The OperatorRequest that you will need to implement
> will
> > > > have
> > > > >> to
> > > > >> > > > change
> > > > >> > > > > > the
> > > > >> > > > > > > property value.
> > > > >> > > > > > >
> > > > >> > > > > > > You can refer to AutoMetricTest for an example.
> > > > >> > > > > > >
> > > > >> > > > > > > - Chandni
> > > > >> > > > > > >
> > > > >> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta
> <
> > > > >> > > > > > > ashwinchandrap@gmail.com> wrote:
> > > > >> > > > > > >
> > > > >> > > > > > > > Why not maintain a variable containing the number of
> > > > >> partitions
> > > > >> > > in
> > > > >> > > > > each
> > > > >> > > > > > > > partition; and when you set the property, just
> divide
> > it
> > > > by
> > > > >> > > number
> > > > >> > > > of
> > > > >> > > > > > > > partitions variable? Does it not solve the problem?
> > > > >> > > > > > > >
> > > > >> > > > > > > > Regards,
> > > > >> > > > > > > > Ashwin.
> > > > >> > > > > > > >
> > > > >> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > > >> > > > > > > priyanka@datatorrent.com
> > > > >> > > > > > > > >
> > > > >> > > > > > > > wrote:
> > > > >> > > > > > > >
> > > > >> > > > > > > > > I have an integer property named "intProp". This
> > > > property
> > > > >> is
> > > > >> > > > > > applicable
> > > > >> > > > > > > > to
> > > > >> > > > > > > > > an operator A. I can set the property from dtcli
> > using
> > > > >> > > > > > > > > "set-operator-property operator-name property-name
> > > > >> > > > property-value"
> > > > >> > > > > > and
> > > > >> > > > > > > > this
> > > > >> > > > > > > > > will get applied to all instances of opeartor A.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Now, I want to apply this property in a way that
> > it's
> > > > >> equally
> > > > >> > > > > divided
> > > > >> > > > > > > > > amongst all instances of operator A. For example,
> > if
> > > > >> > > intProp=100
> > > > >> > > > > and
> > > > >> > > > > > > > > number of partitions=4 then intProp=25 should be
> set
> > > on
> > > > >> each
> > > > >> > > > > instance
> > > > >> > > > > > > of
> > > > >> > > > > > > > > operator A.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > Is there a way to achieve this?
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > I have tried following approach to do this: I have
> > > > >> overriden
> > > > >> > > > > > > > > definePartitions of Operator, once we do
> > partitioning
> > > I
> > > > am
> > > > >> > > > setting
> > > > >> > > > > > > right
> > > > >> > > > > > > > > value on each operator instance based on new
> > partition
> > > > >> count.
> > > > >> > > The
> > > > >> > > > > > > > > limitation here is that the value is applied
> during
> > > > >> > > > > definePartitions.
> > > > >> > > > > > > If
> > > > >> > > > > > > > > the value is updated dynamically it won't be
> applied
> > > > till
> > > > >> > > > > > > > definePartitions
> > > > >> > > > > > > > > is called again.
> > > > >> > > > > > > > >
> > > > >> > > > > > > > > -Priyanka
> > > > >> > > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > >
> > > > >> > > > > > > > --
> > > > >> > > > > > > >
> > > > >> > > > > > > > Regards,
> > > > >> > > > > > > > Ashwin.
> > > > >> > > > > > > >
> > > > >> > > > > > >
> > > > >> > > > > >
> > > > >> > > > >
> > > > >> > > >
> > > > >> > >
> > > > >> >
> > > > >>
> > > > >
> > > > >
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Pramod Immaneni <pr...@datatorrent.com>.

Typically the window difference wouldn't be large and if its happening its
probably indicator of a larger problem in the application anyway.

On Wed, Sep 23, 2015 at 12:07 AM, Gaurav Gupta <ga...@datatorrent.com>
wrote:

> Is that acceptable? as user may want to tune operator A now so that it can
> catch up
>
> Thanks
> -Gaurav
>
> On Wed, Sep 23, 2015 at 12:04 AM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > Yes.
> >
> > On Wed, Sep 23, 2015 at 12:03 AM, Gaurav Gupta <ga...@datatorrent.com>
> > wrote:
> >
> > > Pramod,
> > >
> > > What happens if the Separate Operator that is used to propagate this
> > change
> > > is on say window 1000 and all the instances of Operator A are at
> 300..At
> > > this time user changes value of property. Then the new value will be
> > > delayed to Operator A instances by 700 windows
> > >
> > > Thanks
> > > -Gaurav
> > >
> > > On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <gaurav@datatorrent.com
> >
> > > wrote:
> > >
> > > > Pramod ,
> > > >
> > > > That is what I was thinking that Operator requests can also be
> received
> > > in
> > > > different windows...Looks like adding another operator is safe
> solution
> > > >
> > > > Thanks
> > > > -Gaurav
> > > >
> > > > On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <
> > > pramod@datatorrent.com>
> > > > wrote:
> > > >
> > > >> The operator request would also be received in different windows.
> > > >>
> > > >> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
> > > >> priyanka@datatorrent.com>
> > > >> wrote:
> > > >>
> > > >> > Promod,
> > > >> >
> > > >> > You are right about applying the value in same window. And yes I
> > don't
> > > >> want
> > > >> > to change number of partitions or even redeploy any partition for
> > > this.
> > > >> > I need to explore on Chandini's suggestion of using StatsListener.
> > > >> >
> > > >> > -Priyanka
> > > >> >
> > > >> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
> > > >> pramod@datatorrent.com>
> > > >> > wrote:
> > > >> >
> > > >> > > I am not sure about priyanka's requirements but typically you
> > would
> > > >> want
> > > >> > to
> > > >> > > apply the value in the same window for all partitions for
> > > consistency
> > > >> > with
> > > >> > > the calculations. This could be done two ways using a separate
> > > logical
> > > >> > > operator feeding the value to all partitions so that the value
> > > >> appears in
> > > >> > > the same window or repartitioning the existing set and setting
> the
> > > >> value
> > > >> > > during repartitioning. In the latter case you are just
> > > repartitioning
> > > >> so
> > > >> > > that the partitions start from same point with the new value and
> > not
> > > >> to
> > > >> > > change the number of partitions.
> > > >> > >
> > > >> > > Thanks
> > > >> > >
> > > >> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
> > > >> chandni@datatorrent.com
> > > >> > >
> > > >> > > wrote:
> > > >> > >
> > > >> > > > Pramod,
> > > >> > > >
> > > >> > > > Tim's discussion was about setting a property on multiple
> > > operators
> > > >> > > > (different logical operators).
> > > >> > > > Priyanka's question is about different partitions of an
> operator
> > > >> > > >
> > > >> > > > Let's say they are 5 partitions of an operator and we need to
> > > >> change an
> > > >> > > int
> > > >> > > > property.  There are following ways:
> > > >> > > >
> > > >> > > > 1. Divide the int property by 5 and issue set-physical
> operator
> > > >> > property
> > > >> > > > command for each partition. This is manual.
> > > >> > > >
> > > >> > > > 2. Set the logical operator property but wait till partition
> > > >> happens.
> > > >> > > This
> > > >> > > > will require partitioning to be triggered for which a
> > > StatsListener
> > > >> > needs
> > > >> > > > to be implemented. So if partitioning never happens this will
> > > never
> > > >> be
> > > >> > > > triggered. Also un-deploying and deploying a partition just to
> > > >> change a
> > > >> > > > property on the physical partition seems un-necessary
> > > >> > > >
> > > >> > > > 3. Implementing StatsListener to send operator request and not
> > > >> > > re-partition
> > > >> > > > seems to be appropriate way.
> > > >> > > >
> > > >> > > > Chandni
> > > >> > > >
> > > >> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> > > >> > > pramod@datatorrent.com>
> > > >> > > > wrote:
> > > >> > > >
> > > >> > > > > Priyanka you want to divide the value among existing
> > partitions
> > > >> right
> > > >> > > and
> > > >> > > > > not necessarily change the number of partitions. To this end
> > > >> Ashwin
> > > >> > is
> > > >> > > > > suggesting makes sense. However different partitions can be
> at
> > > >> > > different
> > > >> > > > > windows and will see the value change in different windows.
> > You
> > > >> need
> > > >> > a
> > > >> > > > > separate operator to receive this change and propagate it to
> > the
> > > >> > > > partitions
> > > >> > > > > via data flow. Tim discussed this over the last weekend in a
> > > >> series
> > > >> > of
> > > >> > > > > emails.
> > > >> > > > >
> > > >> > > > > Thanks
> > > >> > > > >
> > > >> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > > >> > > > > priyanka@datatorrent.com>
> > > >> > > > > wrote:
> > > >> > > > >
> > > >> > > > > > Thanks for suggestions.
> > > >> > > > > >
> > > >> > > > > > Ashwin, The option you proposed is doable, but then each
> > > >> operator
> > > >> > > which
> > > >> > > > > > needs to address such use case will have to track the
> number
> > > of
> > > >> > > > > partitions.
> > > >> > > > > > So we were looking for some common mechanism.
> > > >> > > > > > Chandni, Let me try the res.operatorRequests, that could
> > solve
> > > >> the
> > > >> > > > > problem.
> > > >> > > > > >
> > > >> > > > > > -Priyanka
> > > >> > > > > >
> > > >> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > > >> > > > chandni@datatorrent.com>
> > > >> > > > > > wrote:
> > > >> > > > > >
> > > >> > > > > > > Hi Priyanka,
> > > >> > > > > > >
> > > >> > > > > > > As Isha mentioned you will can implement a
> StatsListener.
> > > >> However
> > > >> > > > > instead
> > > >> > > > > > > of setting res.repartitionRequired = true, you can add a
> > > >> > > > > > > StatsListener.OperatorRequest to res.operatorsRequests.
> > This
> > > >> will
> > > >> > > not
> > > >> > > > > > > re-deploy the instances.
> > > >> > > > > > >
> > > >> > > > > > > The OperatorRequest that you will need to implement will
> > > have
> > > >> to
> > > >> > > > change
> > > >> > > > > > the
> > > >> > > > > > > property value.
> > > >> > > > > > >
> > > >> > > > > > > You can refer to AutoMetricTest for an example.
> > > >> > > > > > >
> > > >> > > > > > > - Chandni
> > > >> > > > > > >
> > > >> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > > >> > > > > > > ashwinchandrap@gmail.com> wrote:
> > > >> > > > > > >
> > > >> > > > > > > > Why not maintain a variable containing the number of
> > > >> partitions
> > > >> > > in
> > > >> > > > > each
> > > >> > > > > > > > partition; and when you set the property, just divide
> it
> > > by
> > > >> > > number
> > > >> > > > of
> > > >> > > > > > > > partitions variable? Does it not solve the problem?
> > > >> > > > > > > >
> > > >> > > > > > > > Regards,
> > > >> > > > > > > > Ashwin.
> > > >> > > > > > > >
> > > >> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > >> > > > > > > priyanka@datatorrent.com
> > > >> > > > > > > > >
> > > >> > > > > > > > wrote:
> > > >> > > > > > > >
> > > >> > > > > > > > > I have an integer property named "intProp". This
> > > property
> > > >> is
> > > >> > > > > > applicable
> > > >> > > > > > > > to
> > > >> > > > > > > > > an operator A. I can set the property from dtcli
> using
> > > >> > > > > > > > > "set-operator-property operator-name property-name
> > > >> > > > property-value"
> > > >> > > > > > and
> > > >> > > > > > > > this
> > > >> > > > > > > > > will get applied to all instances of opeartor A.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Now, I want to apply this property in a way that
> it's
> > > >> equally
> > > >> > > > > divided
> > > >> > > > > > > > > amongst all instances of operator A. For example,
> if
> > > >> > > intProp=100
> > > >> > > > > and
> > > >> > > > > > > > > number of partitions=4 then intProp=25 should be set
> > on
> > > >> each
> > > >> > > > > instance
> > > >> > > > > > > of
> > > >> > > > > > > > > operator A.
> > > >> > > > > > > > >
> > > >> > > > > > > > > Is there a way to achieve this?
> > > >> > > > > > > > >
> > > >> > > > > > > > > I have tried following approach to do this: I have
> > > >> overriden
> > > >> > > > > > > > > definePartitions of Operator, once we do
> partitioning
> > I
> > > am
> > > >> > > > setting
> > > >> > > > > > > right
> > > >> > > > > > > > > value on each operator instance based on new
> partition
> > > >> count.
> > > >> > > The
> > > >> > > > > > > > > limitation here is that the value is applied during
> > > >> > > > > definePartitions.
> > > >> > > > > > > If
> > > >> > > > > > > > > the value is updated dynamically it won't be applied
> > > till
> > > >> > > > > > > > definePartitions
> > > >> > > > > > > > > is called again.
> > > >> > > > > > > > >
> > > >> > > > > > > > > -Priyanka
> > > >> > > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > >
> > > >> > > > > > > > --
> > > >> > > > > > > >
> > > >> > > > > > > > Regards,
> > > >> > > > > > > > Ashwin.
> > > >> > > > > > > >
> > > >> > > > > > >
> > > >> > > > > >
> > > >> > > > >
> > > >> > > >
> > > >> > >
> > > >> >
> > > >>
> > > >
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Gaurav Gupta <ga...@datatorrent.com>.

Is that acceptable? as user may want to tune operator A now so that it can
catch up

Thanks
-Gaurav

On Wed, Sep 23, 2015 at 12:04 AM, Pramod Immaneni <pr...@datatorrent.com>
wrote:

> Yes.
>
> On Wed, Sep 23, 2015 at 12:03 AM, Gaurav Gupta <ga...@datatorrent.com>
> wrote:
>
> > Pramod,
> >
> > What happens if the Separate Operator that is used to propagate this
> change
> > is on say window 1000 and all the instances of Operator A are at 300..At
> > this time user changes value of property. Then the new value will be
> > delayed to Operator A instances by 700 windows
> >
> > Thanks
> > -Gaurav
> >
> > On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <ga...@datatorrent.com>
> > wrote:
> >
> > > Pramod ,
> > >
> > > That is what I was thinking that Operator requests can also be received
> > in
> > > different windows...Looks like adding another operator is safe solution
> > >
> > > Thanks
> > > -Gaurav
> > >
> > > On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > >> The operator request would also be received in different windows.
> > >>
> > >> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
> > >> priyanka@datatorrent.com>
> > >> wrote:
> > >>
> > >> > Promod,
> > >> >
> > >> > You are right about applying the value in same window. And yes I
> don't
> > >> want
> > >> > to change number of partitions or even redeploy any partition for
> > this.
> > >> > I need to explore on Chandini's suggestion of using StatsListener.
> > >> >
> > >> > -Priyanka
> > >> >
> > >> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
> > >> pramod@datatorrent.com>
> > >> > wrote:
> > >> >
> > >> > > I am not sure about priyanka's requirements but typically you
> would
> > >> want
> > >> > to
> > >> > > apply the value in the same window for all partitions for
> > consistency
> > >> > with
> > >> > > the calculations. This could be done two ways using a separate
> > logical
> > >> > > operator feeding the value to all partitions so that the value
> > >> appears in
> > >> > > the same window or repartitioning the existing set and setting the
> > >> value
> > >> > > during repartitioning. In the latter case you are just
> > repartitioning
> > >> so
> > >> > > that the partitions start from same point with the new value and
> not
> > >> to
> > >> > > change the number of partitions.
> > >> > >
> > >> > > Thanks
> > >> > >
> > >> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
> > >> chandni@datatorrent.com
> > >> > >
> > >> > > wrote:
> > >> > >
> > >> > > > Pramod,
> > >> > > >
> > >> > > > Tim's discussion was about setting a property on multiple
> > operators
> > >> > > > (different logical operators).
> > >> > > > Priyanka's question is about different partitions of an operator
> > >> > > >
> > >> > > > Let's say they are 5 partitions of an operator and we need to
> > >> change an
> > >> > > int
> > >> > > > property.  There are following ways:
> > >> > > >
> > >> > > > 1. Divide the int property by 5 and issue set-physical operator
> > >> > property
> > >> > > > command for each partition. This is manual.
> > >> > > >
> > >> > > > 2. Set the logical operator property but wait till partition
> > >> happens.
> > >> > > This
> > >> > > > will require partitioning to be triggered for which a
> > StatsListener
> > >> > needs
> > >> > > > to be implemented. So if partitioning never happens this will
> > never
> > >> be
> > >> > > > triggered. Also un-deploying and deploying a partition just to
> > >> change a
> > >> > > > property on the physical partition seems un-necessary
> > >> > > >
> > >> > > > 3. Implementing StatsListener to send operator request and not
> > >> > > re-partition
> > >> > > > seems to be appropriate way.
> > >> > > >
> > >> > > > Chandni
> > >> > > >
> > >> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> > >> > > pramod@datatorrent.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Priyanka you want to divide the value among existing
> partitions
> > >> right
> > >> > > and
> > >> > > > > not necessarily change the number of partitions. To this end
> > >> Ashwin
> > >> > is
> > >> > > > > suggesting makes sense. However different partitions can be at
> > >> > > different
> > >> > > > > windows and will see the value change in different windows.
> You
> > >> need
> > >> > a
> > >> > > > > separate operator to receive this change and propagate it to
> the
> > >> > > > partitions
> > >> > > > > via data flow. Tim discussed this over the last weekend in a
> > >> series
> > >> > of
> > >> > > > > emails.
> > >> > > > >
> > >> > > > > Thanks
> > >> > > > >
> > >> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > >> > > > > priyanka@datatorrent.com>
> > >> > > > > wrote:
> > >> > > > >
> > >> > > > > > Thanks for suggestions.
> > >> > > > > >
> > >> > > > > > Ashwin, The option you proposed is doable, but then each
> > >> operator
> > >> > > which
> > >> > > > > > needs to address such use case will have to track the number
> > of
> > >> > > > > partitions.
> > >> > > > > > So we were looking for some common mechanism.
> > >> > > > > > Chandni, Let me try the res.operatorRequests, that could
> solve
> > >> the
> > >> > > > > problem.
> > >> > > > > >
> > >> > > > > > -Priyanka
> > >> > > > > >
> > >> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > >> > > > chandni@datatorrent.com>
> > >> > > > > > wrote:
> > >> > > > > >
> > >> > > > > > > Hi Priyanka,
> > >> > > > > > >
> > >> > > > > > > As Isha mentioned you will can implement a StatsListener.
> > >> However
> > >> > > > > instead
> > >> > > > > > > of setting res.repartitionRequired = true, you can add a
> > >> > > > > > > StatsListener.OperatorRequest to res.operatorsRequests.
> This
> > >> will
> > >> > > not
> > >> > > > > > > re-deploy the instances.
> > >> > > > > > >
> > >> > > > > > > The OperatorRequest that you will need to implement will
> > have
> > >> to
> > >> > > > change
> > >> > > > > > the
> > >> > > > > > > property value.
> > >> > > > > > >
> > >> > > > > > > You can refer to AutoMetricTest for an example.
> > >> > > > > > >
> > >> > > > > > > - Chandni
> > >> > > > > > >
> > >> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > >> > > > > > > ashwinchandrap@gmail.com> wrote:
> > >> > > > > > >
> > >> > > > > > > > Why not maintain a variable containing the number of
> > >> partitions
> > >> > > in
> > >> > > > > each
> > >> > > > > > > > partition; and when you set the property, just divide it
> > by
> > >> > > number
> > >> > > > of
> > >> > > > > > > > partitions variable? Does it not solve the problem?
> > >> > > > > > > >
> > >> > > > > > > > Regards,
> > >> > > > > > > > Ashwin.
> > >> > > > > > > >
> > >> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > >> > > > > > > priyanka@datatorrent.com
> > >> > > > > > > > >
> > >> > > > > > > > wrote:
> > >> > > > > > > >
> > >> > > > > > > > > I have an integer property named "intProp". This
> > property
> > >> is
> > >> > > > > > applicable
> > >> > > > > > > > to
> > >> > > > > > > > > an operator A. I can set the property from dtcli using
> > >> > > > > > > > > "set-operator-property operator-name property-name
> > >> > > > property-value"
> > >> > > > > > and
> > >> > > > > > > > this
> > >> > > > > > > > > will get applied to all instances of opeartor A.
> > >> > > > > > > > >
> > >> > > > > > > > > Now, I want to apply this property in a way that it's
> > >> equally
> > >> > > > > divided
> > >> > > > > > > > > amongst all instances of operator A. For example,  if
> > >> > > intProp=100
> > >> > > > > and
> > >> > > > > > > > > number of partitions=4 then intProp=25 should be set
> on
> > >> each
> > >> > > > > instance
> > >> > > > > > > of
> > >> > > > > > > > > operator A.
> > >> > > > > > > > >
> > >> > > > > > > > > Is there a way to achieve this?
> > >> > > > > > > > >
> > >> > > > > > > > > I have tried following approach to do this: I have
> > >> overriden
> > >> > > > > > > > > definePartitions of Operator, once we do partitioning
> I
> > am
> > >> > > > setting
> > >> > > > > > > right
> > >> > > > > > > > > value on each operator instance based on new partition
> > >> count.
> > >> > > The
> > >> > > > > > > > > limitation here is that the value is applied during
> > >> > > > > definePartitions.
> > >> > > > > > > If
> > >> > > > > > > > > the value is updated dynamically it won't be applied
> > till
> > >> > > > > > > > definePartitions
> > >> > > > > > > > > is called again.
> > >> > > > > > > > >
> > >> > > > > > > > > -Priyanka
> > >> > > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > >
> > >> > > > > > > > --
> > >> > > > > > > >
> > >> > > > > > > > Regards,
> > >> > > > > > > > Ashwin.
> > >> > > > > > > >
> > >> > > > > > >
> > >> > > > > >
> > >> > > > >
> > >> > > >
> > >> > >
> > >> >
> > >>
> > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Pramod Immaneni <pr...@datatorrent.com>.

Yes.

On Wed, Sep 23, 2015 at 12:03 AM, Gaurav Gupta <ga...@datatorrent.com>
wrote:

> Pramod,
>
> What happens if the Separate Operator that is used to propagate this change
> is on say window 1000 and all the instances of Operator A are at 300..At
> this time user changes value of property. Then the new value will be
> delayed to Operator A instances by 700 windows
>
> Thanks
> -Gaurav
>
> On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <ga...@datatorrent.com>
> wrote:
>
> > Pramod ,
> >
> > That is what I was thinking that Operator requests can also be received
> in
> > different windows...Looks like adding another operator is safe solution
> >
> > Thanks
> > -Gaurav
> >
> > On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <
> pramod@datatorrent.com>
> > wrote:
> >
> >> The operator request would also be received in different windows.
> >>
> >> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
> >> priyanka@datatorrent.com>
> >> wrote:
> >>
> >> > Promod,
> >> >
> >> > You are right about applying the value in same window. And yes I don't
> >> want
> >> > to change number of partitions or even redeploy any partition for
> this.
> >> > I need to explore on Chandini's suggestion of using StatsListener.
> >> >
> >> > -Priyanka
> >> >
> >> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
> >> pramod@datatorrent.com>
> >> > wrote:
> >> >
> >> > > I am not sure about priyanka's requirements but typically you would
> >> want
> >> > to
> >> > > apply the value in the same window for all partitions for
> consistency
> >> > with
> >> > > the calculations. This could be done two ways using a separate
> logical
> >> > > operator feeding the value to all partitions so that the value
> >> appears in
> >> > > the same window or repartitioning the existing set and setting the
> >> value
> >> > > during repartitioning. In the latter case you are just
> repartitioning
> >> so
> >> > > that the partitions start from same point with the new value and not
> >> to
> >> > > change the number of partitions.
> >> > >
> >> > > Thanks
> >> > >
> >> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
> >> chandni@datatorrent.com
> >> > >
> >> > > wrote:
> >> > >
> >> > > > Pramod,
> >> > > >
> >> > > > Tim's discussion was about setting a property on multiple
> operators
> >> > > > (different logical operators).
> >> > > > Priyanka's question is about different partitions of an operator
> >> > > >
> >> > > > Let's say they are 5 partitions of an operator and we need to
> >> change an
> >> > > int
> >> > > > property.  There are following ways:
> >> > > >
> >> > > > 1. Divide the int property by 5 and issue set-physical operator
> >> > property
> >> > > > command for each partition. This is manual.
> >> > > >
> >> > > > 2. Set the logical operator property but wait till partition
> >> happens.
> >> > > This
> >> > > > will require partitioning to be triggered for which a
> StatsListener
> >> > needs
> >> > > > to be implemented. So if partitioning never happens this will
> never
> >> be
> >> > > > triggered. Also un-deploying and deploying a partition just to
> >> change a
> >> > > > property on the physical partition seems un-necessary
> >> > > >
> >> > > > 3. Implementing StatsListener to send operator request and not
> >> > > re-partition
> >> > > > seems to be appropriate way.
> >> > > >
> >> > > > Chandni
> >> > > >
> >> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> >> > > pramod@datatorrent.com>
> >> > > > wrote:
> >> > > >
> >> > > > > Priyanka you want to divide the value among existing partitions
> >> right
> >> > > and
> >> > > > > not necessarily change the number of partitions. To this end
> >> Ashwin
> >> > is
> >> > > > > suggesting makes sense. However different partitions can be at
> >> > > different
> >> > > > > windows and will see the value change in different windows. You
> >> need
> >> > a
> >> > > > > separate operator to receive this change and propagate it to the
> >> > > > partitions
> >> > > > > via data flow. Tim discussed this over the last weekend in a
> >> series
> >> > of
> >> > > > > emails.
> >> > > > >
> >> > > > > Thanks
> >> > > > >
> >> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> >> > > > > priyanka@datatorrent.com>
> >> > > > > wrote:
> >> > > > >
> >> > > > > > Thanks for suggestions.
> >> > > > > >
> >> > > > > > Ashwin, The option you proposed is doable, but then each
> >> operator
> >> > > which
> >> > > > > > needs to address such use case will have to track the number
> of
> >> > > > > partitions.
> >> > > > > > So we were looking for some common mechanism.
> >> > > > > > Chandni, Let me try the res.operatorRequests, that could solve
> >> the
> >> > > > > problem.
> >> > > > > >
> >> > > > > > -Priyanka
> >> > > > > >
> >> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> >> > > > chandni@datatorrent.com>
> >> > > > > > wrote:
> >> > > > > >
> >> > > > > > > Hi Priyanka,
> >> > > > > > >
> >> > > > > > > As Isha mentioned you will can implement a StatsListener.
> >> However
> >> > > > > instead
> >> > > > > > > of setting res.repartitionRequired = true, you can add a
> >> > > > > > > StatsListener.OperatorRequest to res.operatorsRequests. This
> >> will
> >> > > not
> >> > > > > > > re-deploy the instances.
> >> > > > > > >
> >> > > > > > > The OperatorRequest that you will need to implement will
> have
> >> to
> >> > > > change
> >> > > > > > the
> >> > > > > > > property value.
> >> > > > > > >
> >> > > > > > > You can refer to AutoMetricTest for an example.
> >> > > > > > >
> >> > > > > > > - Chandni
> >> > > > > > >
> >> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> >> > > > > > > ashwinchandrap@gmail.com> wrote:
> >> > > > > > >
> >> > > > > > > > Why not maintain a variable containing the number of
> >> partitions
> >> > > in
> >> > > > > each
> >> > > > > > > > partition; and when you set the property, just divide it
> by
> >> > > number
> >> > > > of
> >> > > > > > > > partitions variable? Does it not solve the problem?
> >> > > > > > > >
> >> > > > > > > > Regards,
> >> > > > > > > > Ashwin.
> >> > > > > > > >
> >> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> >> > > > > > > priyanka@datatorrent.com
> >> > > > > > > > >
> >> > > > > > > > wrote:
> >> > > > > > > >
> >> > > > > > > > > I have an integer property named "intProp". This
> property
> >> is
> >> > > > > > applicable
> >> > > > > > > > to
> >> > > > > > > > > an operator A. I can set the property from dtcli using
> >> > > > > > > > > "set-operator-property operator-name property-name
> >> > > > property-value"
> >> > > > > > and
> >> > > > > > > > this
> >> > > > > > > > > will get applied to all instances of opeartor A.
> >> > > > > > > > >
> >> > > > > > > > > Now, I want to apply this property in a way that it's
> >> equally
> >> > > > > divided
> >> > > > > > > > > amongst all instances of operator A. For example,  if
> >> > > intProp=100
> >> > > > > and
> >> > > > > > > > > number of partitions=4 then intProp=25 should be set on
> >> each
> >> > > > > instance
> >> > > > > > > of
> >> > > > > > > > > operator A.
> >> > > > > > > > >
> >> > > > > > > > > Is there a way to achieve this?
> >> > > > > > > > >
> >> > > > > > > > > I have tried following approach to do this: I have
> >> overriden
> >> > > > > > > > > definePartitions of Operator, once we do partitioning I
> am
> >> > > > setting
> >> > > > > > > right
> >> > > > > > > > > value on each operator instance based on new partition
> >> count.
> >> > > The
> >> > > > > > > > > limitation here is that the value is applied during
> >> > > > > definePartitions.
> >> > > > > > > If
> >> > > > > > > > > the value is updated dynamically it won't be applied
> till
> >> > > > > > > > definePartitions
> >> > > > > > > > > is called again.
> >> > > > > > > > >
> >> > > > > > > > > -Priyanka
> >> > > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > >
> >> > > > > > > > --
> >> > > > > > > >
> >> > > > > > > > Regards,
> >> > > > > > > > Ashwin.
> >> > > > > > > >
> >> > > > > > >
> >> > > > > >
> >> > > > >
> >> > > >
> >> > >
> >> >
> >>
> >
> >
>

Re: Assign property to partition instances of Operator

Posted by Gaurav Gupta <ga...@datatorrent.com>.

Pramod,

What happens if the Separate Operator that is used to propagate this change
is on say window 1000 and all the instances of Operator A are at 300..At
this time user changes value of property. Then the new value will be
delayed to Operator A instances by 700 windows

Thanks
-Gaurav

On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <ga...@datatorrent.com>
wrote:

> Pramod ,
>
> That is what I was thinking that Operator requests can also be received in
> different windows...Looks like adding another operator is safe solution
>
> Thanks
> -Gaurav
>
> On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
>> The operator request would also be received in different windows.
>>
>> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
>> priyanka@datatorrent.com>
>> wrote:
>>
>> > Promod,
>> >
>> > You are right about applying the value in same window. And yes I don't
>> want
>> > to change number of partitions or even redeploy any partition for this.
>> > I need to explore on Chandini's suggestion of using StatsListener.
>> >
>> > -Priyanka
>> >
>> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
>> pramod@datatorrent.com>
>> > wrote:
>> >
>> > > I am not sure about priyanka's requirements but typically you would
>> want
>> > to
>> > > apply the value in the same window for all partitions for consistency
>> > with
>> > > the calculations. This could be done two ways using a separate logical
>> > > operator feeding the value to all partitions so that the value
>> appears in
>> > > the same window or repartitioning the existing set and setting the
>> value
>> > > during repartitioning. In the latter case you are just repartitioning
>> so
>> > > that the partitions start from same point with the new value and not
>> to
>> > > change the number of partitions.
>> > >
>> > > Thanks
>> > >
>> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
>> chandni@datatorrent.com
>> > >
>> > > wrote:
>> > >
>> > > > Pramod,
>> > > >
>> > > > Tim's discussion was about setting a property on multiple operators
>> > > > (different logical operators).
>> > > > Priyanka's question is about different partitions of an operator
>> > > >
>> > > > Let's say they are 5 partitions of an operator and we need to
>> change an
>> > > int
>> > > > property.  There are following ways:
>> > > >
>> > > > 1. Divide the int property by 5 and issue set-physical operator
>> > property
>> > > > command for each partition. This is manual.
>> > > >
>> > > > 2. Set the logical operator property but wait till partition
>> happens.
>> > > This
>> > > > will require partitioning to be triggered for which a StatsListener
>> > needs
>> > > > to be implemented. So if partitioning never happens this will never
>> be
>> > > > triggered. Also un-deploying and deploying a partition just to
>> change a
>> > > > property on the physical partition seems un-necessary
>> > > >
>> > > > 3. Implementing StatsListener to send operator request and not
>> > > re-partition
>> > > > seems to be appropriate way.
>> > > >
>> > > > Chandni
>> > > >
>> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
>> > > pramod@datatorrent.com>
>> > > > wrote:
>> > > >
>> > > > > Priyanka you want to divide the value among existing partitions
>> right
>> > > and
>> > > > > not necessarily change the number of partitions. To this end
>> Ashwin
>> > is
>> > > > > suggesting makes sense. However different partitions can be at
>> > > different
>> > > > > windows and will see the value change in different windows. You
>> need
>> > a
>> > > > > separate operator to receive this change and propagate it to the
>> > > > partitions
>> > > > > via data flow. Tim discussed this over the last weekend in a
>> series
>> > of
>> > > > > emails.
>> > > > >
>> > > > > Thanks
>> > > > >
>> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
>> > > > > priyanka@datatorrent.com>
>> > > > > wrote:
>> > > > >
>> > > > > > Thanks for suggestions.
>> > > > > >
>> > > > > > Ashwin, The option you proposed is doable, but then each
>> operator
>> > > which
>> > > > > > needs to address such use case will have to track the number of
>> > > > > partitions.
>> > > > > > So we were looking for some common mechanism.
>> > > > > > Chandni, Let me try the res.operatorRequests, that could solve
>> the
>> > > > > problem.
>> > > > > >
>> > > > > > -Priyanka
>> > > > > >
>> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
>> > > > chandni@datatorrent.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Hi Priyanka,
>> > > > > > >
>> > > > > > > As Isha mentioned you will can implement a StatsListener.
>> However
>> > > > > instead
>> > > > > > > of setting res.repartitionRequired = true, you can add a
>> > > > > > > StatsListener.OperatorRequest to res.operatorsRequests. This
>> will
>> > > not
>> > > > > > > re-deploy the instances.
>> > > > > > >
>> > > > > > > The OperatorRequest that you will need to implement will have
>> to
>> > > > change
>> > > > > > the
>> > > > > > > property value.
>> > > > > > >
>> > > > > > > You can refer to AutoMetricTest for an example.
>> > > > > > >
>> > > > > > > - Chandni
>> > > > > > >
>> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
>> > > > > > > ashwinchandrap@gmail.com> wrote:
>> > > > > > >
>> > > > > > > > Why not maintain a variable containing the number of
>> partitions
>> > > in
>> > > > > each
>> > > > > > > > partition; and when you set the property, just divide it by
>> > > number
>> > > > of
>> > > > > > > > partitions variable? Does it not solve the problem?
>> > > > > > > >
>> > > > > > > > Regards,
>> > > > > > > > Ashwin.
>> > > > > > > >
>> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
>> > > > > > > priyanka@datatorrent.com
>> > > > > > > > >
>> > > > > > > > wrote:
>> > > > > > > >
>> > > > > > > > > I have an integer property named "intProp". This property
>> is
>> > > > > > applicable
>> > > > > > > > to
>> > > > > > > > > an operator A. I can set the property from dtcli using
>> > > > > > > > > "set-operator-property operator-name property-name
>> > > > property-value"
>> > > > > > and
>> > > > > > > > this
>> > > > > > > > > will get applied to all instances of opeartor A.
>> > > > > > > > >
>> > > > > > > > > Now, I want to apply this property in a way that it's
>> equally
>> > > > > divided
>> > > > > > > > > amongst all instances of operator A. For example,  if
>> > > intProp=100
>> > > > > and
>> > > > > > > > > number of partitions=4 then intProp=25 should be set on
>> each
>> > > > > instance
>> > > > > > > of
>> > > > > > > > > operator A.
>> > > > > > > > >
>> > > > > > > > > Is there a way to achieve this?
>> > > > > > > > >
>> > > > > > > > > I have tried following approach to do this: I have
>> overriden
>> > > > > > > > > definePartitions of Operator, once we do partitioning I am
>> > > > setting
>> > > > > > > right
>> > > > > > > > > value on each operator instance based on new partition
>> count.
>> > > The
>> > > > > > > > > limitation here is that the value is applied during
>> > > > > definePartitions.
>> > > > > > > If
>> > > > > > > > > the value is updated dynamically it won't be applied till
>> > > > > > > > definePartitions
>> > > > > > > > > is called again.
>> > > > > > > > >
>> > > > > > > > > -Priyanka
>> > > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > --
>> > > > > > > >
>> > > > > > > > Regards,
>> > > > > > > > Ashwin.
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>>
>
>

Re: Assign property to partition instances of Operator

Posted by Pramod Immaneni <pr...@datatorrent.com>.

Yes two ways separate operator or repartition.

On Tue, Sep 22, 2015 at 11:58 PM, Gaurav Gupta <ga...@datatorrent.com>
wrote:

> Pramod ,
>
> That is what I was thinking that Operator requests can also be received in
> different windows...Looks like adding another operator is safe solution
>
> Thanks
> -Gaurav
>
> On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > The operator request would also be received in different windows.
> >
> > On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
> > priyanka@datatorrent.com>
> > wrote:
> >
> > > Promod,
> > >
> > > You are right about applying the value in same window. And yes I don't
> > want
> > > to change number of partitions or even redeploy any partition for this.
> > > I need to explore on Chandini's suggestion of using StatsListener.
> > >
> > > -Priyanka
> > >
> > > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > > > I am not sure about priyanka's requirements but typically you would
> > want
> > > to
> > > > apply the value in the same window for all partitions for consistency
> > > with
> > > > the calculations. This could be done two ways using a separate
> logical
> > > > operator feeding the value to all partitions so that the value
> appears
> > in
> > > > the same window or repartitioning the existing set and setting the
> > value
> > > > during repartitioning. In the latter case you are just repartitioning
> > so
> > > > that the partitions start from same point with the new value and not
> to
> > > > change the number of partitions.
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
> > chandni@datatorrent.com
> > > >
> > > > wrote:
> > > >
> > > > > Pramod,
> > > > >
> > > > > Tim's discussion was about setting a property on multiple operators
> > > > > (different logical operators).
> > > > > Priyanka's question is about different partitions of an operator
> > > > >
> > > > > Let's say they are 5 partitions of an operator and we need to
> change
> > an
> > > > int
> > > > > property.  There are following ways:
> > > > >
> > > > > 1. Divide the int property by 5 and issue set-physical operator
> > > property
> > > > > command for each partition. This is manual.
> > > > >
> > > > > 2. Set the logical operator property but wait till partition
> happens.
> > > > This
> > > > > will require partitioning to be triggered for which a StatsListener
> > > needs
> > > > > to be implemented. So if partitioning never happens this will never
> > be
> > > > > triggered. Also un-deploying and deploying a partition just to
> > change a
> > > > > property on the physical partition seems un-necessary
> > > > >
> > > > > 3. Implementing StatsListener to send operator request and not
> > > > re-partition
> > > > > seems to be appropriate way.
> > > > >
> > > > > Chandni
> > > > >
> > > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> > > > pramod@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Priyanka you want to divide the value among existing partitions
> > right
> > > > and
> > > > > > not necessarily change the number of partitions. To this end
> Ashwin
> > > is
> > > > > > suggesting makes sense. However different partitions can be at
> > > > different
> > > > > > windows and will see the value change in different windows. You
> > need
> > > a
> > > > > > separate operator to receive this change and propagate it to the
> > > > > partitions
> > > > > > via data flow. Tim discussed this over the last weekend in a
> series
> > > of
> > > > > > emails.
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > > > > > priyanka@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Thanks for suggestions.
> > > > > > >
> > > > > > > Ashwin, The option you proposed is doable, but then each
> operator
> > > > which
> > > > > > > needs to address such use case will have to track the number of
> > > > > > partitions.
> > > > > > > So we were looking for some common mechanism.
> > > > > > > Chandni, Let me try the res.operatorRequests, that could solve
> > the
> > > > > > problem.
> > > > > > >
> > > > > > > -Priyanka
> > > > > > >
> > > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > > > > chandni@datatorrent.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Priyanka,
> > > > > > > >
> > > > > > > > As Isha mentioned you will can implement a StatsListener.
> > However
> > > > > > instead
> > > > > > > > of setting res.repartitionRequired = true, you can add a
> > > > > > > > StatsListener.OperatorRequest to res.operatorsRequests. This
> > will
> > > > not
> > > > > > > > re-deploy the instances.
> > > > > > > >
> > > > > > > > The OperatorRequest that you will need to implement will have
> > to
> > > > > change
> > > > > > > the
> > > > > > > > property value.
> > > > > > > >
> > > > > > > > You can refer to AutoMetricTest for an example.
> > > > > > > >
> > > > > > > > - Chandni
> > > > > > > >
> > > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > > > > > > > ashwinchandrap@gmail.com> wrote:
> > > > > > > >
> > > > > > > > > Why not maintain a variable containing the number of
> > partitions
> > > > in
> > > > > > each
> > > > > > > > > partition; and when you set the property, just divide it by
> > > > number
> > > > > of
> > > > > > > > > partitions variable? Does it not solve the problem?
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Ashwin.
> > > > > > > > >
> > > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > > > > > > priyanka@datatorrent.com
> > > > > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > I have an integer property named "intProp". This property
> > is
> > > > > > > applicable
> > > > > > > > > to
> > > > > > > > > > an operator A. I can set the property from dtcli using
> > > > > > > > > > "set-operator-property operator-name property-name
> > > > > property-value"
> > > > > > > and
> > > > > > > > > this
> > > > > > > > > > will get applied to all instances of opeartor A.
> > > > > > > > > >
> > > > > > > > > > Now, I want to apply this property in a way that it's
> > equally
> > > > > > divided
> > > > > > > > > > amongst all instances of operator A. For example,  if
> > > > intProp=100
> > > > > > and
> > > > > > > > > > number of partitions=4 then intProp=25 should be set on
> > each
> > > > > > instance
> > > > > > > > of
> > > > > > > > > > operator A.
> > > > > > > > > >
> > > > > > > > > > Is there a way to achieve this?
> > > > > > > > > >
> > > > > > > > > > I have tried following approach to do this: I have
> > overriden
> > > > > > > > > > definePartitions of Operator, once we do partitioning I
> am
> > > > > setting
> > > > > > > > right
> > > > > > > > > > value on each operator instance based on new partition
> > count.
> > > > The
> > > > > > > > > > limitation here is that the value is applied during
> > > > > > definePartitions.
> > > > > > > > If
> > > > > > > > > > the value is updated dynamically it won't be applied till
> > > > > > > > > definePartitions
> > > > > > > > > > is called again.
> > > > > > > > > >
> > > > > > > > > > -Priyanka
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Ashwin.
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Gaurav Gupta <ga...@datatorrent.com>.

Pramod ,

That is what I was thinking that Operator requests can also be received in
different windows...Looks like adding another operator is safe solution

Thanks
-Gaurav

On Tue, Sep 22, 2015 at 11:43 PM, Pramod Immaneni <pr...@datatorrent.com>
wrote:

> The operator request would also be received in different windows.
>
> On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <
> priyanka@datatorrent.com>
> wrote:
>
> > Promod,
> >
> > You are right about applying the value in same window. And yes I don't
> want
> > to change number of partitions or even redeploy any partition for this.
> > I need to explore on Chandini's suggestion of using StatsListener.
> >
> > -Priyanka
> >
> > On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <
> pramod@datatorrent.com>
> > wrote:
> >
> > > I am not sure about priyanka's requirements but typically you would
> want
> > to
> > > apply the value in the same window for all partitions for consistency
> > with
> > > the calculations. This could be done two ways using a separate logical
> > > operator feeding the value to all partitions so that the value appears
> in
> > > the same window or repartitioning the existing set and setting the
> value
> > > during repartitioning. In the latter case you are just repartitioning
> so
> > > that the partitions start from same point with the new value and not to
> > > change the number of partitions.
> > >
> > > Thanks
> > >
> > > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <
> chandni@datatorrent.com
> > >
> > > wrote:
> > >
> > > > Pramod,
> > > >
> > > > Tim's discussion was about setting a property on multiple operators
> > > > (different logical operators).
> > > > Priyanka's question is about different partitions of an operator
> > > >
> > > > Let's say they are 5 partitions of an operator and we need to change
> an
> > > int
> > > > property.  There are following ways:
> > > >
> > > > 1. Divide the int property by 5 and issue set-physical operator
> > property
> > > > command for each partition. This is manual.
> > > >
> > > > 2. Set the logical operator property but wait till partition happens.
> > > This
> > > > will require partitioning to be triggered for which a StatsListener
> > needs
> > > > to be implemented. So if partitioning never happens this will never
> be
> > > > triggered. Also un-deploying and deploying a partition just to
> change a
> > > > property on the physical partition seems un-necessary
> > > >
> > > > 3. Implementing StatsListener to send operator request and not
> > > re-partition
> > > > seems to be appropriate way.
> > > >
> > > > Chandni
> > > >
> > > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> > > pramod@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Priyanka you want to divide the value among existing partitions
> right
> > > and
> > > > > not necessarily change the number of partitions. To this end Ashwin
> > is
> > > > > suggesting makes sense. However different partitions can be at
> > > different
> > > > > windows and will see the value change in different windows. You
> need
> > a
> > > > > separate operator to receive this change and propagate it to the
> > > > partitions
> > > > > via data flow. Tim discussed this over the last weekend in a series
> > of
> > > > > emails.
> > > > >
> > > > > Thanks
> > > > >
> > > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > > > > priyanka@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Thanks for suggestions.
> > > > > >
> > > > > > Ashwin, The option you proposed is doable, but then each operator
> > > which
> > > > > > needs to address such use case will have to track the number of
> > > > > partitions.
> > > > > > So we were looking for some common mechanism.
> > > > > > Chandni, Let me try the res.operatorRequests, that could solve
> the
> > > > > problem.
> > > > > >
> > > > > > -Priyanka
> > > > > >
> > > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > > > chandni@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Priyanka,
> > > > > > >
> > > > > > > As Isha mentioned you will can implement a StatsListener.
> However
> > > > > instead
> > > > > > > of setting res.repartitionRequired = true, you can add a
> > > > > > > StatsListener.OperatorRequest to res.operatorsRequests. This
> will
> > > not
> > > > > > > re-deploy the instances.
> > > > > > >
> > > > > > > The OperatorRequest that you will need to implement will have
> to
> > > > change
> > > > > > the
> > > > > > > property value.
> > > > > > >
> > > > > > > You can refer to AutoMetricTest for an example.
> > > > > > >
> > > > > > > - Chandni
> > > > > > >
> > > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > > > > > > ashwinchandrap@gmail.com> wrote:
> > > > > > >
> > > > > > > > Why not maintain a variable containing the number of
> partitions
> > > in
> > > > > each
> > > > > > > > partition; and when you set the property, just divide it by
> > > number
> > > > of
> > > > > > > > partitions variable? Does it not solve the problem?
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Ashwin.
> > > > > > > >
> > > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > > > > > priyanka@datatorrent.com
> > > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I have an integer property named "intProp". This property
> is
> > > > > > applicable
> > > > > > > > to
> > > > > > > > > an operator A. I can set the property from dtcli using
> > > > > > > > > "set-operator-property operator-name property-name
> > > > property-value"
> > > > > > and
> > > > > > > > this
> > > > > > > > > will get applied to all instances of opeartor A.
> > > > > > > > >
> > > > > > > > > Now, I want to apply this property in a way that it's
> equally
> > > > > divided
> > > > > > > > > amongst all instances of operator A. For example,  if
> > > intProp=100
> > > > > and
> > > > > > > > > number of partitions=4 then intProp=25 should be set on
> each
> > > > > instance
> > > > > > > of
> > > > > > > > > operator A.
> > > > > > > > >
> > > > > > > > > Is there a way to achieve this?
> > > > > > > > >
> > > > > > > > > I have tried following approach to do this: I have
> overriden
> > > > > > > > > definePartitions of Operator, once we do partitioning I am
> > > > setting
> > > > > > > right
> > > > > > > > > value on each operator instance based on new partition
> count.
> > > The
> > > > > > > > > limitation here is that the value is applied during
> > > > > definePartitions.
> > > > > > > If
> > > > > > > > > the value is updated dynamically it won't be applied till
> > > > > > > > definePartitions
> > > > > > > > > is called again.
> > > > > > > > >
> > > > > > > > > -Priyanka
> > > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > --
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Ashwin.
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Pramod Immaneni <pr...@datatorrent.com>.

The operator request would also be received in different windows.

On Tue, Sep 22, 2015 at 11:34 PM, Priyanka Gugale <pr...@datatorrent.com>
wrote:

> Promod,
>
> You are right about applying the value in same window. And yes I don't want
> to change number of partitions or even redeploy any partition for this.
> I need to explore on Chandini's suggestion of using StatsListener.
>
> -Priyanka
>
> On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > I am not sure about priyanka's requirements but typically you would want
> to
> > apply the value in the same window for all partitions for consistency
> with
> > the calculations. This could be done two ways using a separate logical
> > operator feeding the value to all partitions so that the value appears in
> > the same window or repartitioning the existing set and setting the value
> > during repartitioning. In the latter case you are just repartitioning so
> > that the partitions start from same point with the new value and not to
> > change the number of partitions.
> >
> > Thanks
> >
> > On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <chandni@datatorrent.com
> >
> > wrote:
> >
> > > Pramod,
> > >
> > > Tim's discussion was about setting a property on multiple operators
> > > (different logical operators).
> > > Priyanka's question is about different partitions of an operator
> > >
> > > Let's say they are 5 partitions of an operator and we need to change an
> > int
> > > property.  There are following ways:
> > >
> > > 1. Divide the int property by 5 and issue set-physical operator
> property
> > > command for each partition. This is manual.
> > >
> > > 2. Set the logical operator property but wait till partition happens.
> > This
> > > will require partitioning to be triggered for which a StatsListener
> needs
> > > to be implemented. So if partitioning never happens this will never be
> > > triggered. Also un-deploying and deploying a partition just to change a
> > > property on the physical partition seems un-necessary
> > >
> > > 3. Implementing StatsListener to send operator request and not
> > re-partition
> > > seems to be appropriate way.
> > >
> > > Chandni
> > >
> > > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> > pramod@datatorrent.com>
> > > wrote:
> > >
> > > > Priyanka you want to divide the value among existing partitions right
> > and
> > > > not necessarily change the number of partitions. To this end Ashwin
> is
> > > > suggesting makes sense. However different partitions can be at
> > different
> > > > windows and will see the value change in different windows. You need
> a
> > > > separate operator to receive this change and propagate it to the
> > > partitions
> > > > via data flow. Tim discussed this over the last weekend in a series
> of
> > > > emails.
> > > >
> > > > Thanks
> > > >
> > > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > > > priyanka@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Thanks for suggestions.
> > > > >
> > > > > Ashwin, The option you proposed is doable, but then each operator
> > which
> > > > > needs to address such use case will have to track the number of
> > > > partitions.
> > > > > So we were looking for some common mechanism.
> > > > > Chandni, Let me try the res.operatorRequests, that could solve the
> > > > problem.
> > > > >
> > > > > -Priyanka
> > > > >
> > > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > > chandni@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Priyanka,
> > > > > >
> > > > > > As Isha mentioned you will can implement a StatsListener. However
> > > > instead
> > > > > > of setting res.repartitionRequired = true, you can add a
> > > > > > StatsListener.OperatorRequest to res.operatorsRequests. This will
> > not
> > > > > > re-deploy the instances.
> > > > > >
> > > > > > The OperatorRequest that you will need to implement will have to
> > > change
> > > > > the
> > > > > > property value.
> > > > > >
> > > > > > You can refer to AutoMetricTest for an example.
> > > > > >
> > > > > > - Chandni
> > > > > >
> > > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > > > > > ashwinchandrap@gmail.com> wrote:
> > > > > >
> > > > > > > Why not maintain a variable containing the number of partitions
> > in
> > > > each
> > > > > > > partition; and when you set the property, just divide it by
> > number
> > > of
> > > > > > > partitions variable? Does it not solve the problem?
> > > > > > >
> > > > > > > Regards,
> > > > > > > Ashwin.
> > > > > > >
> > > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > > > > priyanka@datatorrent.com
> > > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I have an integer property named "intProp". This property is
> > > > > applicable
> > > > > > > to
> > > > > > > > an operator A. I can set the property from dtcli using
> > > > > > > > "set-operator-property operator-name property-name
> > > property-value"
> > > > > and
> > > > > > > this
> > > > > > > > will get applied to all instances of opeartor A.
> > > > > > > >
> > > > > > > > Now, I want to apply this property in a way that it's equally
> > > > divided
> > > > > > > > amongst all instances of operator A. For example,  if
> > intProp=100
> > > > and
> > > > > > > > number of partitions=4 then intProp=25 should be set on each
> > > > instance
> > > > > > of
> > > > > > > > operator A.
> > > > > > > >
> > > > > > > > Is there a way to achieve this?
> > > > > > > >
> > > > > > > > I have tried following approach to do this: I have overriden
> > > > > > > > definePartitions of Operator, once we do partitioning I am
> > > setting
> > > > > > right
> > > > > > > > value on each operator instance based on new partition count.
> > The
> > > > > > > > limitation here is that the value is applied during
> > > > definePartitions.
> > > > > > If
> > > > > > > > the value is updated dynamically it won't be applied till
> > > > > > > definePartitions
> > > > > > > > is called again.
> > > > > > > >
> > > > > > > > -Priyanka
> > > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > >
> > > > > > > Regards,
> > > > > > > Ashwin.
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Priyanka Gugale <pr...@datatorrent.com>.

Promod,

You are right about applying the value in same window. And yes I don't want
to change number of partitions or even redeploy any partition for this.
I need to explore on Chandini's suggestion of using StatsListener.

-Priyanka

On Wed, Sep 23, 2015 at 12:00 PM, Pramod Immaneni <pr...@datatorrent.com>
wrote:

> I am not sure about priyanka's requirements but typically you would want to
> apply the value in the same window for all partitions for consistency with
> the calculations. This could be done two ways using a separate logical
> operator feeding the value to all partitions so that the value appears in
> the same window or repartitioning the existing set and setting the value
> during repartitioning. In the latter case you are just repartitioning so
> that the partitions start from same point with the new value and not to
> change the number of partitions.
>
> Thanks
>
> On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > Pramod,
> >
> > Tim's discussion was about setting a property on multiple operators
> > (different logical operators).
> > Priyanka's question is about different partitions of an operator
> >
> > Let's say they are 5 partitions of an operator and we need to change an
> int
> > property.  There are following ways:
> >
> > 1. Divide the int property by 5 and issue set-physical operator property
> > command for each partition. This is manual.
> >
> > 2. Set the logical operator property but wait till partition happens.
> This
> > will require partitioning to be triggered for which a StatsListener needs
> > to be implemented. So if partitioning never happens this will never be
> > triggered. Also un-deploying and deploying a partition just to change a
> > property on the physical partition seems un-necessary
> >
> > 3. Implementing StatsListener to send operator request and not
> re-partition
> > seems to be appropriate way.
> >
> > Chandni
> >
> > On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <
> pramod@datatorrent.com>
> > wrote:
> >
> > > Priyanka you want to divide the value among existing partitions right
> and
> > > not necessarily change the number of partitions. To this end Ashwin is
> > > suggesting makes sense. However different partitions can be at
> different
> > > windows and will see the value change in different windows. You need a
> > > separate operator to receive this change and propagate it to the
> > partitions
> > > via data flow. Tim discussed this over the last weekend in a series of
> > > emails.
> > >
> > > Thanks
> > >
> > > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > > priyanka@datatorrent.com>
> > > wrote:
> > >
> > > > Thanks for suggestions.
> > > >
> > > > Ashwin, The option you proposed is doable, but then each operator
> which
> > > > needs to address such use case will have to track the number of
> > > partitions.
> > > > So we were looking for some common mechanism.
> > > > Chandni, Let me try the res.operatorRequests, that could solve the
> > > problem.
> > > >
> > > > -Priyanka
> > > >
> > > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> > chandni@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Hi Priyanka,
> > > > >
> > > > > As Isha mentioned you will can implement a StatsListener. However
> > > instead
> > > > > of setting res.repartitionRequired = true, you can add a
> > > > > StatsListener.OperatorRequest to res.operatorsRequests. This will
> not
> > > > > re-deploy the instances.
> > > > >
> > > > > The OperatorRequest that you will need to implement will have to
> > change
> > > > the
> > > > > property value.
> > > > >
> > > > > You can refer to AutoMetricTest for an example.
> > > > >
> > > > > - Chandni
> > > > >
> > > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > > > > ashwinchandrap@gmail.com> wrote:
> > > > >
> > > > > > Why not maintain a variable containing the number of partitions
> in
> > > each
> > > > > > partition; and when you set the property, just divide it by
> number
> > of
> > > > > > partitions variable? Does it not solve the problem?
> > > > > >
> > > > > > Regards,
> > > > > > Ashwin.
> > > > > >
> > > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > > > priyanka@datatorrent.com
> > > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I have an integer property named "intProp". This property is
> > > > applicable
> > > > > > to
> > > > > > > an operator A. I can set the property from dtcli using
> > > > > > > "set-operator-property operator-name property-name
> > property-value"
> > > > and
> > > > > > this
> > > > > > > will get applied to all instances of opeartor A.
> > > > > > >
> > > > > > > Now, I want to apply this property in a way that it's equally
> > > divided
> > > > > > > amongst all instances of operator A. For example,  if
> intProp=100
> > > and
> > > > > > > number of partitions=4 then intProp=25 should be set on each
> > > instance
> > > > > of
> > > > > > > operator A.
> > > > > > >
> > > > > > > Is there a way to achieve this?
> > > > > > >
> > > > > > > I have tried following approach to do this: I have overriden
> > > > > > > definePartitions of Operator, once we do partitioning I am
> > setting
> > > > > right
> > > > > > > value on each operator instance based on new partition count.
> The
> > > > > > > limitation here is that the value is applied during
> > > definePartitions.
> > > > > If
> > > > > > > the value is updated dynamically it won't be applied till
> > > > > > definePartitions
> > > > > > > is called again.
> > > > > > >
> > > > > > > -Priyanka
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > >
> > > > > > Regards,
> > > > > > Ashwin.
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Pramod Immaneni <pr...@datatorrent.com>.

I am not sure about priyanka's requirements but typically you would want to
apply the value in the same window for all partitions for consistency with
the calculations. This could be done two ways using a separate logical
operator feeding the value to all partitions so that the value appears in
the same window or repartitioning the existing set and setting the value
during repartitioning. In the latter case you are just repartitioning so
that the partitions start from same point with the new value and not to
change the number of partitions.

Thanks

On Tue, Sep 22, 2015 at 11:18 PM, Chandni Singh <ch...@datatorrent.com>
wrote:

> Pramod,
>
> Tim's discussion was about setting a property on multiple operators
> (different logical operators).
> Priyanka's question is about different partitions of an operator
>
> Let's say they are 5 partitions of an operator and we need to change an int
> property.  There are following ways:
>
> 1. Divide the int property by 5 and issue set-physical operator property
> command for each partition. This is manual.
>
> 2. Set the logical operator property but wait till partition happens. This
> will require partitioning to be triggered for which a StatsListener needs
> to be implemented. So if partitioning never happens this will never be
> triggered. Also un-deploying and deploying a partition just to change a
> property on the physical partition seems un-necessary
>
> 3. Implementing StatsListener to send operator request and not re-partition
> seems to be appropriate way.
>
> Chandni
>
> On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > Priyanka you want to divide the value among existing partitions right and
> > not necessarily change the number of partitions. To this end Ashwin is
> > suggesting makes sense. However different partitions can be at different
> > windows and will see the value change in different windows. You need a
> > separate operator to receive this change and propagate it to the
> partitions
> > via data flow. Tim discussed this over the last weekend in a series of
> > emails.
> >
> > Thanks
> >
> > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > priyanka@datatorrent.com>
> > wrote:
> >
> > > Thanks for suggestions.
> > >
> > > Ashwin, The option you proposed is doable, but then each operator which
> > > needs to address such use case will have to track the number of
> > partitions.
> > > So we were looking for some common mechanism.
> > > Chandni, Let me try the res.operatorRequests, that could solve the
> > problem.
> > >
> > > -Priyanka
> > >
> > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> chandni@datatorrent.com>
> > > wrote:
> > >
> > > > Hi Priyanka,
> > > >
> > > > As Isha mentioned you will can implement a StatsListener. However
> > instead
> > > > of setting res.repartitionRequired = true, you can add a
> > > > StatsListener.OperatorRequest to res.operatorsRequests. This will not
> > > > re-deploy the instances.
> > > >
> > > > The OperatorRequest that you will need to implement will have to
> change
> > > the
> > > > property value.
> > > >
> > > > You can refer to AutoMetricTest for an example.
> > > >
> > > > - Chandni
> > > >
> > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > > > ashwinchandrap@gmail.com> wrote:
> > > >
> > > > > Why not maintain a variable containing the number of partitions in
> > each
> > > > > partition; and when you set the property, just divide it by number
> of
> > > > > partitions variable? Does it not solve the problem?
> > > > >
> > > > > Regards,
> > > > > Ashwin.
> > > > >
> > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > > priyanka@datatorrent.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I have an integer property named "intProp". This property is
> > > applicable
> > > > > to
> > > > > > an operator A. I can set the property from dtcli using
> > > > > > "set-operator-property operator-name property-name
> property-value"
> > > and
> > > > > this
> > > > > > will get applied to all instances of opeartor A.
> > > > > >
> > > > > > Now, I want to apply this property in a way that it's equally
> > divided
> > > > > > amongst all instances of operator A. For example,  if intProp=100
> > and
> > > > > > number of partitions=4 then intProp=25 should be set on each
> > instance
> > > > of
> > > > > > operator A.
> > > > > >
> > > > > > Is there a way to achieve this?
> > > > > >
> > > > > > I have tried following approach to do this: I have overriden
> > > > > > definePartitions of Operator, once we do partitioning I am
> setting
> > > > right
> > > > > > value on each operator instance based on new partition count. The
> > > > > > limitation here is that the value is applied during
> > definePartitions.
> > > > If
> > > > > > the value is updated dynamically it won't be applied till
> > > > > definePartitions
> > > > > > is called again.
> > > > > >
> > > > > > -Priyanka
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Regards,
> > > > > Ashwin.
> > > > >
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Priyanka Gugale <pr...@datatorrent.com>.

Gaurav,
I am doing the exact thing which you suggested. But limitation here is I
need to wait for next partitioning call to aggregate old value and
re-distribute to new partitions. And the next partitioning may not happen
for long time.

Pramod,
Having a different operator could be an overhead for this simple usecase.

-Priyanka

On Wed, Sep 23, 2015 at 11:48 AM, Chandni Singh <ch...@datatorrent.com>
wrote:

> Pramod,
>
> Tim's discussion was about setting a property on multiple operators
> (different logical operators).
> Priyanka's question is about different partitions of an operator
>
> Let's say they are 5 partitions of an operator and we need to change an int
> property.  There are following ways:
>
> 1. Divide the int property by 5 and issue set-physical operator property
> command for each partition. This is manual.
>
> 2. Set the logical operator property but wait till partition happens. This
> will require partitioning to be triggered for which a StatsListener needs
> to be implemented. So if partitioning never happens this will never be
> triggered. Also un-deploying and deploying a partition just to change a
> property on the physical partition seems un-necessary
>
> 3. Implementing StatsListener to send operator request and not re-partition
> seems to be appropriate way.
>
> Chandni
>
> On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > Priyanka you want to divide the value among existing partitions right and
> > not necessarily change the number of partitions. To this end Ashwin is
> > suggesting makes sense. However different partitions can be at different
> > windows and will see the value change in different windows. You need a
> > separate operator to receive this change and propagate it to the
> partitions
> > via data flow. Tim discussed this over the last weekend in a series of
> > emails.
> >
> > Thanks
> >
> > On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> > priyanka@datatorrent.com>
> > wrote:
> >
> > > Thanks for suggestions.
> > >
> > > Ashwin, The option you proposed is doable, but then each operator which
> > > needs to address such use case will have to track the number of
> > partitions.
> > > So we were looking for some common mechanism.
> > > Chandni, Let me try the res.operatorRequests, that could solve the
> > problem.
> > >
> > > -Priyanka
> > >
> > > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <
> chandni@datatorrent.com>
> > > wrote:
> > >
> > > > Hi Priyanka,
> > > >
> > > > As Isha mentioned you will can implement a StatsListener. However
> > instead
> > > > of setting res.repartitionRequired = true, you can add a
> > > > StatsListener.OperatorRequest to res.operatorsRequests. This will not
> > > > re-deploy the instances.
> > > >
> > > > The OperatorRequest that you will need to implement will have to
> change
> > > the
> > > > property value.
> > > >
> > > > You can refer to AutoMetricTest for an example.
> > > >
> > > > - Chandni
> > > >
> > > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > > > ashwinchandrap@gmail.com> wrote:
> > > >
> > > > > Why not maintain a variable containing the number of partitions in
> > each
> > > > > partition; and when you set the property, just divide it by number
> of
> > > > > partitions variable? Does it not solve the problem?
> > > > >
> > > > > Regards,
> > > > > Ashwin.
> > > > >
> > > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > > priyanka@datatorrent.com
> > > > > >
> > > > > wrote:
> > > > >
> > > > > > I have an integer property named "intProp". This property is
> > > applicable
> > > > > to
> > > > > > an operator A. I can set the property from dtcli using
> > > > > > "set-operator-property operator-name property-name
> property-value"
> > > and
> > > > > this
> > > > > > will get applied to all instances of opeartor A.
> > > > > >
> > > > > > Now, I want to apply this property in a way that it's equally
> > divided
> > > > > > amongst all instances of operator A. For example,  if intProp=100
> > and
> > > > > > number of partitions=4 then intProp=25 should be set on each
> > instance
> > > > of
> > > > > > operator A.
> > > > > >
> > > > > > Is there a way to achieve this?
> > > > > >
> > > > > > I have tried following approach to do this: I have overriden
> > > > > > definePartitions of Operator, once we do partitioning I am
> setting
> > > > right
> > > > > > value on each operator instance based on new partition count. The
> > > > > > limitation here is that the value is applied during
> > definePartitions.
> > > > If
> > > > > > the value is updated dynamically it won't be applied till
> > > > > definePartitions
> > > > > > is called again.
> > > > > >
> > > > > > -Priyanka
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > >
> > > > > Regards,
> > > > > Ashwin.
> > > > >
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Chandni Singh <ch...@datatorrent.com>.

Pramod,

Tim's discussion was about setting a property on multiple operators
(different logical operators).
Priyanka's question is about different partitions of an operator

Let's say they are 5 partitions of an operator and we need to change an int
property.  There are following ways:

1. Divide the int property by 5 and issue set-physical operator property
command for each partition. This is manual.

2. Set the logical operator property but wait till partition happens. This
will require partitioning to be triggered for which a StatsListener needs
to be implemented. So if partitioning never happens this will never be
triggered. Also un-deploying and deploying a partition just to change a
property on the physical partition seems un-necessary

3. Implementing StatsListener to send operator request and not re-partition
seems to be appropriate way.

Chandni

On Tue, Sep 22, 2015 at 11:02 PM, Pramod Immaneni <pr...@datatorrent.com>
wrote:

> Priyanka you want to divide the value among existing partitions right and
> not necessarily change the number of partitions. To this end Ashwin is
> suggesting makes sense. However different partitions can be at different
> windows and will see the value change in different windows. You need a
> separate operator to receive this change and propagate it to the partitions
> via data flow. Tim discussed this over the last weekend in a series of
> emails.
>
> Thanks
>
> On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <
> priyanka@datatorrent.com>
> wrote:
>
> > Thanks for suggestions.
> >
> > Ashwin, The option you proposed is doable, but then each operator which
> > needs to address such use case will have to track the number of
> partitions.
> > So we were looking for some common mechanism.
> > Chandni, Let me try the res.operatorRequests, that could solve the
> problem.
> >
> > -Priyanka
> >
> > On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <ch...@datatorrent.com>
> > wrote:
> >
> > > Hi Priyanka,
> > >
> > > As Isha mentioned you will can implement a StatsListener. However
> instead
> > > of setting res.repartitionRequired = true, you can add a
> > > StatsListener.OperatorRequest to res.operatorsRequests. This will not
> > > re-deploy the instances.
> > >
> > > The OperatorRequest that you will need to implement will have to change
> > the
> > > property value.
> > >
> > > You can refer to AutoMetricTest for an example.
> > >
> > > - Chandni
> > >
> > > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > > ashwinchandrap@gmail.com> wrote:
> > >
> > > > Why not maintain a variable containing the number of partitions in
> each
> > > > partition; and when you set the property, just divide it by number of
> > > > partitions variable? Does it not solve the problem?
> > > >
> > > > Regards,
> > > > Ashwin.
> > > >
> > > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > > priyanka@datatorrent.com
> > > > >
> > > > wrote:
> > > >
> > > > > I have an integer property named "intProp". This property is
> > applicable
> > > > to
> > > > > an operator A. I can set the property from dtcli using
> > > > > "set-operator-property operator-name property-name property-value"
> > and
> > > > this
> > > > > will get applied to all instances of opeartor A.
> > > > >
> > > > > Now, I want to apply this property in a way that it's equally
> divided
> > > > > amongst all instances of operator A. For example,  if intProp=100
> and
> > > > > number of partitions=4 then intProp=25 should be set on each
> instance
> > > of
> > > > > operator A.
> > > > >
> > > > > Is there a way to achieve this?
> > > > >
> > > > > I have tried following approach to do this: I have overriden
> > > > > definePartitions of Operator, once we do partitioning I am setting
> > > right
> > > > > value on each operator instance based on new partition count. The
> > > > > limitation here is that the value is applied during
> definePartitions.
> > > If
> > > > > the value is updated dynamically it won't be applied till
> > > > definePartitions
> > > > > is called again.
> > > > >
> > > > > -Priyanka
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >
> > > > Regards,
> > > > Ashwin.
> > > >
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Pramod Immaneni <pr...@datatorrent.com>.

Priyanka you want to divide the value among existing partitions right and
not necessarily change the number of partitions. To this end Ashwin is
suggesting makes sense. However different partitions can be at different
windows and will see the value change in different windows. You need a
separate operator to receive this change and propagate it to the partitions
via data flow. Tim discussed this over the last weekend in a series of
emails.

Thanks

On Tue, Sep 22, 2015 at 10:43 PM, Priyanka Gugale <pr...@datatorrent.com>
wrote:

> Thanks for suggestions.
>
> Ashwin, The option you proposed is doable, but then each operator which
> needs to address such use case will have to track the number of partitions.
> So we were looking for some common mechanism.
> Chandni, Let me try the res.operatorRequests, that could solve the problem.
>
> -Priyanka
>
> On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > Hi Priyanka,
> >
> > As Isha mentioned you will can implement a StatsListener. However instead
> > of setting res.repartitionRequired = true, you can add a
> > StatsListener.OperatorRequest to res.operatorsRequests. This will not
> > re-deploy the instances.
> >
> > The OperatorRequest that you will need to implement will have to change
> the
> > property value.
> >
> > You can refer to AutoMetricTest for an example.
> >
> > - Chandni
> >
> > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > ashwinchandrap@gmail.com> wrote:
> >
> > > Why not maintain a variable containing the number of partitions in each
> > > partition; and when you set the property, just divide it by number of
> > > partitions variable? Does it not solve the problem?
> > >
> > > Regards,
> > > Ashwin.
> > >
> > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > priyanka@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > I have an integer property named "intProp". This property is
> applicable
> > > to
> > > > an operator A. I can set the property from dtcli using
> > > > "set-operator-property operator-name property-name property-value"
> and
> > > this
> > > > will get applied to all instances of opeartor A.
> > > >
> > > > Now, I want to apply this property in a way that it's equally divided
> > > > amongst all instances of operator A. For example,  if intProp=100 and
> > > > number of partitions=4 then intProp=25 should be set on each instance
> > of
> > > > operator A.
> > > >
> > > > Is there a way to achieve this?
> > > >
> > > > I have tried following approach to do this: I have overriden
> > > > definePartitions of Operator, once we do partitioning I am setting
> > right
> > > > value on each operator instance based on new partition count. The
> > > > limitation here is that the value is applied during definePartitions.
> > If
> > > > the value is updated dynamically it won't be applied till
> > > definePartitions
> > > > is called again.
> > > >
> > > > -Priyanka
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashwin.
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Gaurav Gupta <ga...@datatorrent.com>.

Priyanka

Why not just set 25 on operator A and during dynamic partitioning if any
you can aggregate old value and distribute among new partitions

Thanks
-Gaurav
On Sep 22, 2015 10:43 PM, "Priyanka Gugale" <pr...@datatorrent.com>
wrote:

> Thanks for suggestions.
>
> Ashwin, The option you proposed is doable, but then each operator which
> needs to address such use case will have to track the number of partitions.
> So we were looking for some common mechanism.
> Chandni, Let me try the res.operatorRequests, that could solve the problem.
>
> -Priyanka
>
> On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > Hi Priyanka,
> >
> > As Isha mentioned you will can implement a StatsListener. However instead
> > of setting res.repartitionRequired = true, you can add a
> > StatsListener.OperatorRequest to res.operatorsRequests. This will not
> > re-deploy the instances.
> >
> > The OperatorRequest that you will need to implement will have to change
> the
> > property value.
> >
> > You can refer to AutoMetricTest for an example.
> >
> > - Chandni
> >
> > On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> > ashwinchandrap@gmail.com> wrote:
> >
> > > Why not maintain a variable containing the number of partitions in each
> > > partition; and when you set the property, just divide it by number of
> > > partitions variable? Does it not solve the problem?
> > >
> > > Regards,
> > > Ashwin.
> > >
> > > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> > priyanka@datatorrent.com
> > > >
> > > wrote:
> > >
> > > > I have an integer property named "intProp". This property is
> applicable
> > > to
> > > > an operator A. I can set the property from dtcli using
> > > > "set-operator-property operator-name property-name property-value"
> and
> > > this
> > > > will get applied to all instances of opeartor A.
> > > >
> > > > Now, I want to apply this property in a way that it's equally divided
> > > > amongst all instances of operator A. For example,  if intProp=100 and
> > > > number of partitions=4 then intProp=25 should be set on each instance
> > of
> > > > operator A.
> > > >
> > > > Is there a way to achieve this?
> > > >
> > > > I have tried following approach to do this: I have overriden
> > > > definePartitions of Operator, once we do partitioning I am setting
> > right
> > > > value on each operator instance based on new partition count. The
> > > > limitation here is that the value is applied during definePartitions.
> > If
> > > > the value is updated dynamically it won't be applied till
> > > definePartitions
> > > > is called again.
> > > >
> > > > -Priyanka
> > > >
> > >
> > >
> > >
> > > --
> > >
> > > Regards,
> > > Ashwin.
> > >
> >
>

Re: Assign property to partition instances of Operator

Posted by Priyanka Gugale <pr...@datatorrent.com>.

Thanks for suggestions.

Ashwin, The option you proposed is doable, but then each operator which
needs to address such use case will have to track the number of partitions.
So we were looking for some common mechanism.
Chandni, Let me try the res.operatorRequests, that could solve the problem.

-Priyanka

On Tue, Sep 22, 2015 at 8:05 PM, Chandni Singh <ch...@datatorrent.com>
wrote:

> Hi Priyanka,
>
> As Isha mentioned you will can implement a StatsListener. However instead
> of setting res.repartitionRequired = true, you can add a
> StatsListener.OperatorRequest to res.operatorsRequests. This will not
> re-deploy the instances.
>
> The OperatorRequest that you will need to implement will have to change the
> property value.
>
> You can refer to AutoMetricTest for an example.
>
> - Chandni
>
> On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
> ashwinchandrap@gmail.com> wrote:
>
> > Why not maintain a variable containing the number of partitions in each
> > partition; and when you set the property, just divide it by number of
> > partitions variable? Does it not solve the problem?
> >
> > Regards,
> > Ashwin.
> >
> > On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <
> priyanka@datatorrent.com
> > >
> > wrote:
> >
> > > I have an integer property named "intProp". This property is applicable
> > to
> > > an operator A. I can set the property from dtcli using
> > > "set-operator-property operator-name property-name property-value" and
> > this
> > > will get applied to all instances of opeartor A.
> > >
> > > Now, I want to apply this property in a way that it's equally divided
> > > amongst all instances of operator A. For example,  if intProp=100 and
> > > number of partitions=4 then intProp=25 should be set on each instance
> of
> > > operator A.
> > >
> > > Is there a way to achieve this?
> > >
> > > I have tried following approach to do this: I have overriden
> > > definePartitions of Operator, once we do partitioning I am setting
> right
> > > value on each operator instance based on new partition count. The
> > > limitation here is that the value is applied during definePartitions.
> If
> > > the value is updated dynamically it won't be applied till
> > definePartitions
> > > is called again.
> > >
> > > -Priyanka
> > >
> >
> >
> >
> > --
> >
> > Regards,
> > Ashwin.
> >
>

Re: Assign property to partition instances of Operator

Posted by Chandni Singh <ch...@datatorrent.com>.

Hi Priyanka,

As Isha mentioned you will can implement a StatsListener. However instead
of setting res.repartitionRequired = true, you can add a
StatsListener.OperatorRequest to res.operatorsRequests. This will not
re-deploy the instances.

The OperatorRequest that you will need to implement will have to change the
property value.

You can refer to AutoMetricTest for an example.

- Chandni

On Tue, Sep 22, 2015 at 7:11 AM, Ashwin Chandra Putta <
ashwinchandrap@gmail.com> wrote:

> Why not maintain a variable containing the number of partitions in each
> partition; and when you set the property, just divide it by number of
> partitions variable? Does it not solve the problem?
>
> Regards,
> Ashwin.
>
> On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <priyanka@datatorrent.com
> >
> wrote:
>
> > I have an integer property named "intProp". This property is applicable
> to
> > an operator A. I can set the property from dtcli using
> > "set-operator-property operator-name property-name property-value" and
> this
> > will get applied to all instances of opeartor A.
> >
> > Now, I want to apply this property in a way that it's equally divided
> > amongst all instances of operator A. For example,  if intProp=100 and
> > number of partitions=4 then intProp=25 should be set on each instance of
> > operator A.
> >
> > Is there a way to achieve this?
> >
> > I have tried following approach to do this: I have overriden
> > definePartitions of Operator, once we do partitioning I am setting right
> > value on each operator instance based on new partition count. The
> > limitation here is that the value is applied during definePartitions. If
> > the value is updated dynamically it won't be applied till
> definePartitions
> > is called again.
> >
> > -Priyanka
> >
>
>
>
> --
>
> Regards,
> Ashwin.
>

Re: Assign property to partition instances of Operator

Posted by Ashwin Chandra Putta <as...@gmail.com>.

Why not maintain a variable containing the number of partitions in each
partition; and when you set the property, just divide it by number of
partitions variable? Does it not solve the problem?

Regards,
Ashwin.

On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <pr...@datatorrent.com>
wrote:

> I have an integer property named "intProp". This property is applicable to
> an operator A. I can set the property from dtcli using
> "set-operator-property operator-name property-name property-value" and this
> will get applied to all instances of opeartor A.
>
> Now, I want to apply this property in a way that it's equally divided
> amongst all instances of operator A. For example,  if intProp=100 and
> number of partitions=4 then intProp=25 should be set on each instance of
> operator A.
>
> Is there a way to achieve this?
>
> I have tried following approach to do this: I have overriden
> definePartitions of Operator, once we do partitioning I am setting right
> value on each operator instance based on new partition count. The
> limitation here is that the value is applied during definePartitions. If
> the value is updated dynamically it won't be applied till definePartitions
> is called again.
>
> -Priyanka
>



-- 

Regards,
Ashwin.

Re: Assign property to partition instances of Operator

Posted by Isha Arkatkar <is...@datatorrent.com>.

Hi Priyanka,

   To invoke definePartitions method after updating new value dynamically,
you can implement StatsListener interface in the operator. And in the
overridden processStats method, you can have something like:

public Response processStats(BatchedOperatorStats stats)
  {
    Response res = new Response();
    res.repartitionRequired = false;
    if (intProp != oldIntProp) {
      res.repartitionRequired = true;
    }
    return res;
  }

  However, this will redeploy the partitions again, even if partitioning is
not changed. May be you can try storing the number of partitions as an
operator property and use that to divide the intProp value?

Thanks,
Isha


On Tue, Sep 22, 2015 at 5:37 AM, Priyanka Gugale <pr...@datatorrent.com>
wrote:

> I have an integer property named "intProp". This property is applicable to
> an operator A. I can set the property from dtcli using
> "set-operator-property operator-name property-name property-value" and this
> will get applied to all instances of opeartor A.
>
> Now, I want to apply this property in a way that it's equally divided
> amongst all instances of operator A. For example,  if intProp=100 and
> number of partitions=4 then intProp=25 should be set on each instance of
> operator A.
>
> Is there a way to achieve this?
>
> I have tried following approach to do this: I have overriden
> definePartitions of Operator, once we do partitioning I am setting right
> value on each operator instance based on new partition count. The
> limitation here is that the value is applied during definePartitions. If
> the value is updated dynamically it won't be applied till definePartitions
> is called again.
>
> -Priyanka
>