You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@apex.apache.org by Chandni Singh <ch...@datatorrent.com> on 2015/12/14 01:07:49 UTC
Enable semantic versioning only for specific operators in Malhar
Hi,
In Malhar there are relatively smaller number of operators that we use in
our demo applications, customer applications, POCs etc that are mature.
The library is cluttered with operators especially in lib/util, lib/algo,
lib/math packages which can be cleaned up by either removing them or
improving them but that breaks semantic versioning.
When we add new operators/utilities it takes certain time for them to
mature. Japicmp doesn't help because it doesn't honor @Evolving @Unstable
annotations for now.
I wanted to propose that we add an annotation, let's say, re-use hadoop's
@Stable and mark the operators which are stable with it and perform semver
check on just these operators.
The 0.7.0 version of japi cmp has the support for inclusions (as well as
exclusions) based on annotations.
Here is the info:
https://github.com/siom79/japicmp/issues/88
The reason I am inclined to the inclusion approach is that there are
relatively smaller number of operators which IMO are stable. A lot of them
aren't.
So instead of going and marking so many as Evolving, we will mark
relatively few of them as stable.
Also new development can be facilitated by this. We wouldn't have to add
@Evolving to everything which is new. Instead we will mark it @Stable when
it is.
Please let me know what you think?
Thanks,
Chandni
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Thomas Weise <th...@datatorrent.com>.
-1 on custom annotations.
hadoop-common is an API dependency.
[INFO] org.apache.apex:apex-api:jar:3.3.0-incubating-SNAPSHOT
[INFO] +- org.apache.hadoop:hadoop-common:jar:2.2.0:compile
[INFO] | +- org.apache.hadoop:hadoop-annotations:jar:2.2.0:compile
On Tue, Dec 22, 2015 at 2:37 PM, Chandni Singh <ch...@datatorrent.com>
wrote:
> Pramod,
> Agreed on this point. Will create our own annotations.
>
> Thanks,
> Chandni
>
> On Tue, Dec 22, 2015 at 2:30 PM, Pramod Immaneni <pr...@datatorrent.com>
> wrote:
>
> > I suggest we don't go with hadoop annotations as that will add a direct
> > dependency from operators to hadoop even though most of them only need to
> > be dependent on apex. In future if we port apex to run on other systems
> > this will pose a problem. We either implement our own annotations or pick
> > some other independent standard.
> >
> > On Mon, Dec 14, 2015 at 10:16 PM, Chandni Singh <chandni@datatorrent.com
> >
> > wrote:
> >
> > > We need to identify the operators and components that are stable if we
> > want
> > > to go with semver check of only Stable classes.
> > > I can create an initial list.
> > >
> > > Thanks,
> > > Chandni
> > >
> > > On Mon, Dec 14, 2015 at 9:24 PM, Isha Arkatkar <is...@datatorrent.com>
> > > wrote:
> > >
> > > > Yep, That's what I am doing now :)
> > > >
> > > > Thanks,
> > > > Isha
> > > >
> > > > On Mon, Dec 14, 2015 at 9:22 PM, Chandni Singh <
> > chandni@datatorrent.com>
> > > > wrote:
> > > >
> > > > > Isha,
> > > > >
> > > > > I think for now you can configure the japicmp plugin to exclude the
> > > > package
> > > > > as follows in the pom.xml.
> > > > >
> > > > > <excludes>
> > > > >
> > > > > <exclude>com.datatorrent.lib.parser.*</exclude>
> > > > >
> > > > > </excludes>
> > > > >
> > > > > This is an example where we can benefit from inclusion approach
> with
> > > > > japicmp 0.7 version.
> > > > >
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Chandni
> > > > >
> > > > >
> > > > > On Mon, Dec 14, 2015 at 8:34 PM, Isha Arkatkar <
> isha@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > > When 0.7 version of japicmp is available, we can add exclusions
> for
> > > > > > @Evolving or inclusions for @Stable, whichever way is finalized.
> > > > > >
> > > > > > But before that should we add package exclusions individually if
> > all
> > > > the
> > > > > > operators inside the package are marked Evolving?
> > > > > > I wanted to make changes to some of the parser operators in
> Malhar.
> > > But
> > > > > > changing those breaks sem version check.
> > > > > >
> > > > > > Please suggest.
> > > > > >
> > > > > > Thanks,
> > > > > > Isha
> > > > > >
> > > > > > On Sun, Dec 13, 2015 at 10:46 PM, Tushar Gosavi <
> > > > tushar@datatorrent.com>
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > > The new operators added will most likely to undergo change
> > > frequently
> > > > > > until
> > > > > > > they become stable.
> > > > > > >
> > > > > > > - Tushar.
> > > > > > >
> > > > > > >
> > > > > > > On Mon, Dec 14, 2015 at 12:05 PM, Siyuan Hua <
> > > siyuan@datatorrent.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > >
> > > > > > > > On Sun, Dec 13, 2015 at 10:01 PM, Yogi Devendra <
> > > > > > yogidevendra@apache.org
> > > > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > I am not against the Idea of using @Stable instead of
> marking
> > > > every
> > > > > > new
> > > > > > > > > @Evolving. I agree that, it would convenient to have
> @Stable.
> > > > > > > > >
> > > > > > > > > Just raised the point which needs further discussion, so
> that
> > > we
> > > > > get
> > > > > > > > > suggestions from the mentors and community.
> > > > > > > > >
> > > > > > > > > ~ Yogi
> > > > > > > > >
> > > > > > > > > On 14 December 2015 at 11:23, Chandni Singh <
> > > > > chandni@datatorrent.com
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Can we define some criteria for deciding when to consider
> > > > > operator
> > > > > > as
> > > > > > > > > > @Stable?
> > > > > > > > > > Yes, we should follow Hadoop's example and formalize some
> > > > > criteria.
> > > > > > > > > >
> > > > > > > > > > [It would be difficult for an open source project to
> track
> > > > which
> > > > > > user
> > > > > > > > is
> > > > > > > > > > using which operators. So, above strategy may not work. ]
> > > > > > > > > > Hadoop is an open source project which actually created
> > these
> > > > > > > > annotations
> > > > > > > > > > and it's widely used. I think any new development takes
> > time
> > > to
> > > > > > > become
> > > > > > > > > > stable.
> > > > > > > > > > If the operators are NOT marked as @Stable, users will
> know
> > > > that
> > > > > > when
> > > > > > > > > they
> > > > > > > > > > upgrade backward compatibility may be broken.
> > > > > > > > > > I think it has the same affect of marking every new
> > > > > operator/class
> > > > > > as
> > > > > > > > > > @Evolving. The benefit of checking semantic versioning of
> > > > > "Stable"
> > > > > > > > > > operators is that they are currently fewer in number and
> > IMO
> > > > easy
> > > > > > to
> > > > > > > > > manage
> > > > > > > > > > and new development will be implicitly "Evolving".
> > > > > > > > > >
> > > > > > > > > > Chandni
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <
> > > > > > > > yogidevendra@apache.org>
> > > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > When to mark certain operator as @Stable is not clearly
> > > > > defined.
> > > > > > > > > > >
> > > > > > > > > > > Can we define some criteria for deciding when to
> consider
> > > > > > operator
> > > > > > > as
> > > > > > > > > > > @Stable?
> > > > > > > > > > >
> > > > > > > > > > > For example one criteria could be, if operator is
> running
> > > for
> > > > > >1
> > > > > > > year
> > > > > > > > > in
> > > > > > > > > > > production environment for some user. Can we come with
> > some
> > > > > > > strategy
> > > > > > > > > like
> > > > > > > > > > > this?
> > > > > > > > > > > [It would be difficult for an open source project to
> > track
> > > > > which
> > > > > > > user
> > > > > > > > > is
> > > > > > > > > > > using which operators. So, above strategy may not
> work. ]
> > > > > > > > > > >
> > > > > > > > > > > ~ Yogi
> > > > > > > > > > >
> > > > > > > > > > > On 14 December 2015 at 05:42, Timothy Farkas <
> > > > > > tim@datatorrent.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > +1
> > > > > > > > > > > > On Dec 13, 2015 4:08 PM, "Chandni Singh" <
> > > > > > > chandni@datatorrent.com>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi,
> > > > > > > > > > > > >
> > > > > > > > > > > > > In Malhar there are relatively smaller number of
> > > > > operators
> > > > > > > that
> > > > > > > > > we
> > > > > > > > > > > use
> > > > > > > > > > > > in
> > > > > > > > > > > > > our demo applications, customer applications, POCs
> > etc
> > > > that
> > > > > > are
> > > > > > > > > > mature.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The library is cluttered with operators especially
> in
> > > > > > lib/util,
> > > > > > > > > > > lib/algo,
> > > > > > > > > > > > > lib/math packages which can be cleaned up by either
> > > > > removing
> > > > > > > them
> > > > > > > > > or
> > > > > > > > > > > > > improving them but that breaks semantic versioning.
> > > > > > > > > > > > >
> > > > > > > > > > > > > When we add new operators/utilities it takes
> certain
> > > time
> > > > > for
> > > > > > > > them
> > > > > > > > > to
> > > > > > > > > > > > > mature. Japicmp doesn't help because it doesn't
> honor
> > > > > > @Evolving
> > > > > > > > > > > @Unstable
> > > > > > > > > > > > > annotations for now.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I wanted to propose that we add an annotation,
> let's
> > > say,
> > > > > > > re-use
> > > > > > > > > > > hadoop's
> > > > > > > > > > > > > @Stable and mark the operators which are stable
> with
> > it
> > > > and
> > > > > > > > perform
> > > > > > > > > > > > semver
> > > > > > > > > > > > > check on just these operators.
> > > > > > > > > > > > >
> > > > > > > > > > > > > The 0.7.0 version of japi cmp has the support for
> > > > > inclusions
> > > > > > > (as
> > > > > > > > > well
> > > > > > > > > > > as
> > > > > > > > > > > > > exclusions) based on annotations.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Here is the info:
> > > > > > > > > > > > > https://github.com/siom79/japicmp/issues/88
> > > > > > > > > > > > >
> > > > > > > > > > > > > The reason I am inclined to the inclusion approach
> is
> > > > that
> > > > > > > there
> > > > > > > > > are
> > > > > > > > > > > > > relatively smaller number of operators which IMO
> are
> > > > > stable.
> > > > > > A
> > > > > > > > lot
> > > > > > > > > of
> > > > > > > > > > > > them
> > > > > > > > > > > > > aren't.
> > > > > > > > > > > > > So instead of going and marking so many as
> Evolving,
> > we
> > > > > will
> > > > > > > mark
> > > > > > > > > > > > > relatively few of them as stable.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Also new development can be facilitated by this. We
> > > > > wouldn't
> > > > > > > have
> > > > > > > > > to
> > > > > > > > > > > add
> > > > > > > > > > > > > @Evolving to everything which is new. Instead we
> will
> > > > mark
> > > > > it
> > > > > > > > > @Stable
> > > > > > > > > > > > when
> > > > > > > > > > > > > it is.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Please let me know what you think?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Chandni
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Chandni Singh <ch...@datatorrent.com>.
Pramod,
Agreed on this point. Will create our own annotations.
Thanks,
Chandni
On Tue, Dec 22, 2015 at 2:30 PM, Pramod Immaneni <pr...@datatorrent.com>
wrote:
> I suggest we don't go with hadoop annotations as that will add a direct
> dependency from operators to hadoop even though most of them only need to
> be dependent on apex. In future if we port apex to run on other systems
> this will pose a problem. We either implement our own annotations or pick
> some other independent standard.
>
> On Mon, Dec 14, 2015 at 10:16 PM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > We need to identify the operators and components that are stable if we
> want
> > to go with semver check of only Stable classes.
> > I can create an initial list.
> >
> > Thanks,
> > Chandni
> >
> > On Mon, Dec 14, 2015 at 9:24 PM, Isha Arkatkar <is...@datatorrent.com>
> > wrote:
> >
> > > Yep, That's what I am doing now :)
> > >
> > > Thanks,
> > > Isha
> > >
> > > On Mon, Dec 14, 2015 at 9:22 PM, Chandni Singh <
> chandni@datatorrent.com>
> > > wrote:
> > >
> > > > Isha,
> > > >
> > > > I think for now you can configure the japicmp plugin to exclude the
> > > package
> > > > as follows in the pom.xml.
> > > >
> > > > <excludes>
> > > >
> > > > <exclude>com.datatorrent.lib.parser.*</exclude>
> > > >
> > > > </excludes>
> > > >
> > > > This is an example where we can benefit from inclusion approach with
> > > > japicmp 0.7 version.
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > Chandni
> > > >
> > > >
> > > > On Mon, Dec 14, 2015 at 8:34 PM, Isha Arkatkar <isha@datatorrent.com
> >
> > > > wrote:
> > > >
> > > > > +1
> > > > > When 0.7 version of japicmp is available, we can add exclusions for
> > > > > @Evolving or inclusions for @Stable, whichever way is finalized.
> > > > >
> > > > > But before that should we add package exclusions individually if
> all
> > > the
> > > > > operators inside the package are marked Evolving?
> > > > > I wanted to make changes to some of the parser operators in Malhar.
> > But
> > > > > changing those breaks sem version check.
> > > > >
> > > > > Please suggest.
> > > > >
> > > > > Thanks,
> > > > > Isha
> > > > >
> > > > > On Sun, Dec 13, 2015 at 10:46 PM, Tushar Gosavi <
> > > tushar@datatorrent.com>
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > > The new operators added will most likely to undergo change
> > frequently
> > > > > until
> > > > > > they become stable.
> > > > > >
> > > > > > - Tushar.
> > > > > >
> > > > > >
> > > > > > On Mon, Dec 14, 2015 at 12:05 PM, Siyuan Hua <
> > siyuan@datatorrent.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > >
> > > > > > > On Sun, Dec 13, 2015 at 10:01 PM, Yogi Devendra <
> > > > > yogidevendra@apache.org
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I am not against the Idea of using @Stable instead of marking
> > > every
> > > > > new
> > > > > > > > @Evolving. I agree that, it would convenient to have @Stable.
> > > > > > > >
> > > > > > > > Just raised the point which needs further discussion, so that
> > we
> > > > get
> > > > > > > > suggestions from the mentors and community.
> > > > > > > >
> > > > > > > > ~ Yogi
> > > > > > > >
> > > > > > > > On 14 December 2015 at 11:23, Chandni Singh <
> > > > chandni@datatorrent.com
> > > > > >
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Can we define some criteria for deciding when to consider
> > > > operator
> > > > > as
> > > > > > > > > @Stable?
> > > > > > > > > Yes, we should follow Hadoop's example and formalize some
> > > > criteria.
> > > > > > > > >
> > > > > > > > > [It would be difficult for an open source project to track
> > > which
> > > > > user
> > > > > > > is
> > > > > > > > > using which operators. So, above strategy may not work. ]
> > > > > > > > > Hadoop is an open source project which actually created
> these
> > > > > > > annotations
> > > > > > > > > and it's widely used. I think any new development takes
> time
> > to
> > > > > > become
> > > > > > > > > stable.
> > > > > > > > > If the operators are NOT marked as @Stable, users will know
> > > that
> > > > > when
> > > > > > > > they
> > > > > > > > > upgrade backward compatibility may be broken.
> > > > > > > > > I think it has the same affect of marking every new
> > > > operator/class
> > > > > as
> > > > > > > > > @Evolving. The benefit of checking semantic versioning of
> > > > "Stable"
> > > > > > > > > operators is that they are currently fewer in number and
> IMO
> > > easy
> > > > > to
> > > > > > > > manage
> > > > > > > > > and new development will be implicitly "Evolving".
> > > > > > > > >
> > > > > > > > > Chandni
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <
> > > > > > > yogidevendra@apache.org>
> > > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > When to mark certain operator as @Stable is not clearly
> > > > defined.
> > > > > > > > > >
> > > > > > > > > > Can we define some criteria for deciding when to consider
> > > > > operator
> > > > > > as
> > > > > > > > > > @Stable?
> > > > > > > > > >
> > > > > > > > > > For example one criteria could be, if operator is running
> > for
> > > > >1
> > > > > > year
> > > > > > > > in
> > > > > > > > > > production environment for some user. Can we come with
> some
> > > > > > strategy
> > > > > > > > like
> > > > > > > > > > this?
> > > > > > > > > > [It would be difficult for an open source project to
> track
> > > > which
> > > > > > user
> > > > > > > > is
> > > > > > > > > > using which operators. So, above strategy may not work. ]
> > > > > > > > > >
> > > > > > > > > > ~ Yogi
> > > > > > > > > >
> > > > > > > > > > On 14 December 2015 at 05:42, Timothy Farkas <
> > > > > tim@datatorrent.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > +1
> > > > > > > > > > > On Dec 13, 2015 4:08 PM, "Chandni Singh" <
> > > > > > chandni@datatorrent.com>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi,
> > > > > > > > > > > >
> > > > > > > > > > > > In Malhar there are relatively smaller number of
> > > > operators
> > > > > > that
> > > > > > > > we
> > > > > > > > > > use
> > > > > > > > > > > in
> > > > > > > > > > > > our demo applications, customer applications, POCs
> etc
> > > that
> > > > > are
> > > > > > > > > mature.
> > > > > > > > > > > >
> > > > > > > > > > > > The library is cluttered with operators especially in
> > > > > lib/util,
> > > > > > > > > > lib/algo,
> > > > > > > > > > > > lib/math packages which can be cleaned up by either
> > > > removing
> > > > > > them
> > > > > > > > or
> > > > > > > > > > > > improving them but that breaks semantic versioning.
> > > > > > > > > > > >
> > > > > > > > > > > > When we add new operators/utilities it takes certain
> > time
> > > > for
> > > > > > > them
> > > > > > > > to
> > > > > > > > > > > > mature. Japicmp doesn't help because it doesn't honor
> > > > > @Evolving
> > > > > > > > > > @Unstable
> > > > > > > > > > > > annotations for now.
> > > > > > > > > > > >
> > > > > > > > > > > > I wanted to propose that we add an annotation, let's
> > say,
> > > > > > re-use
> > > > > > > > > > hadoop's
> > > > > > > > > > > > @Stable and mark the operators which are stable with
> it
> > > and
> > > > > > > perform
> > > > > > > > > > > semver
> > > > > > > > > > > > check on just these operators.
> > > > > > > > > > > >
> > > > > > > > > > > > The 0.7.0 version of japi cmp has the support for
> > > > inclusions
> > > > > > (as
> > > > > > > > well
> > > > > > > > > > as
> > > > > > > > > > > > exclusions) based on annotations.
> > > > > > > > > > > >
> > > > > > > > > > > > Here is the info:
> > > > > > > > > > > > https://github.com/siom79/japicmp/issues/88
> > > > > > > > > > > >
> > > > > > > > > > > > The reason I am inclined to the inclusion approach is
> > > that
> > > > > > there
> > > > > > > > are
> > > > > > > > > > > > relatively smaller number of operators which IMO are
> > > > stable.
> > > > > A
> > > > > > > lot
> > > > > > > > of
> > > > > > > > > > > them
> > > > > > > > > > > > aren't.
> > > > > > > > > > > > So instead of going and marking so many as Evolving,
> we
> > > > will
> > > > > > mark
> > > > > > > > > > > > relatively few of them as stable.
> > > > > > > > > > > >
> > > > > > > > > > > > Also new development can be facilitated by this. We
> > > > wouldn't
> > > > > > have
> > > > > > > > to
> > > > > > > > > > add
> > > > > > > > > > > > @Evolving to everything which is new. Instead we will
> > > mark
> > > > it
> > > > > > > > @Stable
> > > > > > > > > > > when
> > > > > > > > > > > > it is.
> > > > > > > > > > > >
> > > > > > > > > > > > Please let me know what you think?
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Chandni
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Pramod Immaneni <pr...@datatorrent.com>.
I suggest we don't go with hadoop annotations as that will add a direct
dependency from operators to hadoop even though most of them only need to
be dependent on apex. In future if we port apex to run on other systems
this will pose a problem. We either implement our own annotations or pick
some other independent standard.
On Mon, Dec 14, 2015 at 10:16 PM, Chandni Singh <ch...@datatorrent.com>
wrote:
> We need to identify the operators and components that are stable if we want
> to go with semver check of only Stable classes.
> I can create an initial list.
>
> Thanks,
> Chandni
>
> On Mon, Dec 14, 2015 at 9:24 PM, Isha Arkatkar <is...@datatorrent.com>
> wrote:
>
> > Yep, That's what I am doing now :)
> >
> > Thanks,
> > Isha
> >
> > On Mon, Dec 14, 2015 at 9:22 PM, Chandni Singh <ch...@datatorrent.com>
> > wrote:
> >
> > > Isha,
> > >
> > > I think for now you can configure the japicmp plugin to exclude the
> > package
> > > as follows in the pom.xml.
> > >
> > > <excludes>
> > >
> > > <exclude>com.datatorrent.lib.parser.*</exclude>
> > >
> > > </excludes>
> > >
> > > This is an example where we can benefit from inclusion approach with
> > > japicmp 0.7 version.
> > >
> > >
> > > Thanks,
> > >
> > > Chandni
> > >
> > >
> > > On Mon, Dec 14, 2015 at 8:34 PM, Isha Arkatkar <is...@datatorrent.com>
> > > wrote:
> > >
> > > > +1
> > > > When 0.7 version of japicmp is available, we can add exclusions for
> > > > @Evolving or inclusions for @Stable, whichever way is finalized.
> > > >
> > > > But before that should we add package exclusions individually if all
> > the
> > > > operators inside the package are marked Evolving?
> > > > I wanted to make changes to some of the parser operators in Malhar.
> But
> > > > changing those breaks sem version check.
> > > >
> > > > Please suggest.
> > > >
> > > > Thanks,
> > > > Isha
> > > >
> > > > On Sun, Dec 13, 2015 at 10:46 PM, Tushar Gosavi <
> > tushar@datatorrent.com>
> > > > wrote:
> > > >
> > > > > +1
> > > > > The new operators added will most likely to undergo change
> frequently
> > > > until
> > > > > they become stable.
> > > > >
> > > > > - Tushar.
> > > > >
> > > > >
> > > > > On Mon, Dec 14, 2015 at 12:05 PM, Siyuan Hua <
> siyuan@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > > > +1
> > > > > >
> > > > > > On Sun, Dec 13, 2015 at 10:01 PM, Yogi Devendra <
> > > > yogidevendra@apache.org
> > > > > >
> > > > > > wrote:
> > > > > >
> > > > > > > I am not against the Idea of using @Stable instead of marking
> > every
> > > > new
> > > > > > > @Evolving. I agree that, it would convenient to have @Stable.
> > > > > > >
> > > > > > > Just raised the point which needs further discussion, so that
> we
> > > get
> > > > > > > suggestions from the mentors and community.
> > > > > > >
> > > > > > > ~ Yogi
> > > > > > >
> > > > > > > On 14 December 2015 at 11:23, Chandni Singh <
> > > chandni@datatorrent.com
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > > > Can we define some criteria for deciding when to consider
> > > operator
> > > > as
> > > > > > > > @Stable?
> > > > > > > > Yes, we should follow Hadoop's example and formalize some
> > > criteria.
> > > > > > > >
> > > > > > > > [It would be difficult for an open source project to track
> > which
> > > > user
> > > > > > is
> > > > > > > > using which operators. So, above strategy may not work. ]
> > > > > > > > Hadoop is an open source project which actually created these
> > > > > > annotations
> > > > > > > > and it's widely used. I think any new development takes time
> to
> > > > > become
> > > > > > > > stable.
> > > > > > > > If the operators are NOT marked as @Stable, users will know
> > that
> > > > when
> > > > > > > they
> > > > > > > > upgrade backward compatibility may be broken.
> > > > > > > > I think it has the same affect of marking every new
> > > operator/class
> > > > as
> > > > > > > > @Evolving. The benefit of checking semantic versioning of
> > > "Stable"
> > > > > > > > operators is that they are currently fewer in number and IMO
> > easy
> > > > to
> > > > > > > manage
> > > > > > > > and new development will be implicitly "Evolving".
> > > > > > > >
> > > > > > > > Chandni
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <
> > > > > > yogidevendra@apache.org>
> > > > > > > > wrote:
> > > > > > > >
> > > > > > > > > When to mark certain operator as @Stable is not clearly
> > > defined.
> > > > > > > > >
> > > > > > > > > Can we define some criteria for deciding when to consider
> > > > operator
> > > > > as
> > > > > > > > > @Stable?
> > > > > > > > >
> > > > > > > > > For example one criteria could be, if operator is running
> for
> > > >1
> > > > > year
> > > > > > > in
> > > > > > > > > production environment for some user. Can we come with some
> > > > > strategy
> > > > > > > like
> > > > > > > > > this?
> > > > > > > > > [It would be difficult for an open source project to track
> > > which
> > > > > user
> > > > > > > is
> > > > > > > > > using which operators. So, above strategy may not work. ]
> > > > > > > > >
> > > > > > > > > ~ Yogi
> > > > > > > > >
> > > > > > > > > On 14 December 2015 at 05:42, Timothy Farkas <
> > > > tim@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > +1
> > > > > > > > > > On Dec 13, 2015 4:08 PM, "Chandni Singh" <
> > > > > chandni@datatorrent.com>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi,
> > > > > > > > > > >
> > > > > > > > > > > In Malhar there are relatively smaller number of
> > > operators
> > > > > that
> > > > > > > we
> > > > > > > > > use
> > > > > > > > > > in
> > > > > > > > > > > our demo applications, customer applications, POCs etc
> > that
> > > > are
> > > > > > > > mature.
> > > > > > > > > > >
> > > > > > > > > > > The library is cluttered with operators especially in
> > > > lib/util,
> > > > > > > > > lib/algo,
> > > > > > > > > > > lib/math packages which can be cleaned up by either
> > > removing
> > > > > them
> > > > > > > or
> > > > > > > > > > > improving them but that breaks semantic versioning.
> > > > > > > > > > >
> > > > > > > > > > > When we add new operators/utilities it takes certain
> time
> > > for
> > > > > > them
> > > > > > > to
> > > > > > > > > > > mature. Japicmp doesn't help because it doesn't honor
> > > > @Evolving
> > > > > > > > > @Unstable
> > > > > > > > > > > annotations for now.
> > > > > > > > > > >
> > > > > > > > > > > I wanted to propose that we add an annotation, let's
> say,
> > > > > re-use
> > > > > > > > > hadoop's
> > > > > > > > > > > @Stable and mark the operators which are stable with it
> > and
> > > > > > perform
> > > > > > > > > > semver
> > > > > > > > > > > check on just these operators.
> > > > > > > > > > >
> > > > > > > > > > > The 0.7.0 version of japi cmp has the support for
> > > inclusions
> > > > > (as
> > > > > > > well
> > > > > > > > > as
> > > > > > > > > > > exclusions) based on annotations.
> > > > > > > > > > >
> > > > > > > > > > > Here is the info:
> > > > > > > > > > > https://github.com/siom79/japicmp/issues/88
> > > > > > > > > > >
> > > > > > > > > > > The reason I am inclined to the inclusion approach is
> > that
> > > > > there
> > > > > > > are
> > > > > > > > > > > relatively smaller number of operators which IMO are
> > > stable.
> > > > A
> > > > > > lot
> > > > > > > of
> > > > > > > > > > them
> > > > > > > > > > > aren't.
> > > > > > > > > > > So instead of going and marking so many as Evolving, we
> > > will
> > > > > mark
> > > > > > > > > > > relatively few of them as stable.
> > > > > > > > > > >
> > > > > > > > > > > Also new development can be facilitated by this. We
> > > wouldn't
> > > > > have
> > > > > > > to
> > > > > > > > > add
> > > > > > > > > > > @Evolving to everything which is new. Instead we will
> > mark
> > > it
> > > > > > > @Stable
> > > > > > > > > > when
> > > > > > > > > > > it is.
> > > > > > > > > > >
> > > > > > > > > > > Please let me know what you think?
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Chandni
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Chandni Singh <ch...@datatorrent.com>.
We need to identify the operators and components that are stable if we want
to go with semver check of only Stable classes.
I can create an initial list.
Thanks,
Chandni
On Mon, Dec 14, 2015 at 9:24 PM, Isha Arkatkar <is...@datatorrent.com> wrote:
> Yep, That's what I am doing now :)
>
> Thanks,
> Isha
>
> On Mon, Dec 14, 2015 at 9:22 PM, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > Isha,
> >
> > I think for now you can configure the japicmp plugin to exclude the
> package
> > as follows in the pom.xml.
> >
> > <excludes>
> >
> > <exclude>com.datatorrent.lib.parser.*</exclude>
> >
> > </excludes>
> >
> > This is an example where we can benefit from inclusion approach with
> > japicmp 0.7 version.
> >
> >
> > Thanks,
> >
> > Chandni
> >
> >
> > On Mon, Dec 14, 2015 at 8:34 PM, Isha Arkatkar <is...@datatorrent.com>
> > wrote:
> >
> > > +1
> > > When 0.7 version of japicmp is available, we can add exclusions for
> > > @Evolving or inclusions for @Stable, whichever way is finalized.
> > >
> > > But before that should we add package exclusions individually if all
> the
> > > operators inside the package are marked Evolving?
> > > I wanted to make changes to some of the parser operators in Malhar. But
> > > changing those breaks sem version check.
> > >
> > > Please suggest.
> > >
> > > Thanks,
> > > Isha
> > >
> > > On Sun, Dec 13, 2015 at 10:46 PM, Tushar Gosavi <
> tushar@datatorrent.com>
> > > wrote:
> > >
> > > > +1
> > > > The new operators added will most likely to undergo change frequently
> > > until
> > > > they become stable.
> > > >
> > > > - Tushar.
> > > >
> > > >
> > > > On Mon, Dec 14, 2015 at 12:05 PM, Siyuan Hua <siyuan@datatorrent.com
> >
> > > > wrote:
> > > >
> > > > > +1
> > > > >
> > > > > On Sun, Dec 13, 2015 at 10:01 PM, Yogi Devendra <
> > > yogidevendra@apache.org
> > > > >
> > > > > wrote:
> > > > >
> > > > > > I am not against the Idea of using @Stable instead of marking
> every
> > > new
> > > > > > @Evolving. I agree that, it would convenient to have @Stable.
> > > > > >
> > > > > > Just raised the point which needs further discussion, so that we
> > get
> > > > > > suggestions from the mentors and community.
> > > > > >
> > > > > > ~ Yogi
> > > > > >
> > > > > > On 14 December 2015 at 11:23, Chandni Singh <
> > chandni@datatorrent.com
> > > >
> > > > > > wrote:
> > > > > >
> > > > > > > Can we define some criteria for deciding when to consider
> > operator
> > > as
> > > > > > > @Stable?
> > > > > > > Yes, we should follow Hadoop's example and formalize some
> > criteria.
> > > > > > >
> > > > > > > [It would be difficult for an open source project to track
> which
> > > user
> > > > > is
> > > > > > > using which operators. So, above strategy may not work. ]
> > > > > > > Hadoop is an open source project which actually created these
> > > > > annotations
> > > > > > > and it's widely used. I think any new development takes time to
> > > > become
> > > > > > > stable.
> > > > > > > If the operators are NOT marked as @Stable, users will know
> that
> > > when
> > > > > > they
> > > > > > > upgrade backward compatibility may be broken.
> > > > > > > I think it has the same affect of marking every new
> > operator/class
> > > as
> > > > > > > @Evolving. The benefit of checking semantic versioning of
> > "Stable"
> > > > > > > operators is that they are currently fewer in number and IMO
> easy
> > > to
> > > > > > manage
> > > > > > > and new development will be implicitly "Evolving".
> > > > > > >
> > > > > > > Chandni
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <
> > > > > yogidevendra@apache.org>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > When to mark certain operator as @Stable is not clearly
> > defined.
> > > > > > > >
> > > > > > > > Can we define some criteria for deciding when to consider
> > > operator
> > > > as
> > > > > > > > @Stable?
> > > > > > > >
> > > > > > > > For example one criteria could be, if operator is running for
> > >1
> > > > year
> > > > > > in
> > > > > > > > production environment for some user. Can we come with some
> > > > strategy
> > > > > > like
> > > > > > > > this?
> > > > > > > > [It would be difficult for an open source project to track
> > which
> > > > user
> > > > > > is
> > > > > > > > using which operators. So, above strategy may not work. ]
> > > > > > > >
> > > > > > > > ~ Yogi
> > > > > > > >
> > > > > > > > On 14 December 2015 at 05:42, Timothy Farkas <
> > > tim@datatorrent.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > +1
> > > > > > > > > On Dec 13, 2015 4:08 PM, "Chandni Singh" <
> > > > chandni@datatorrent.com>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi,
> > > > > > > > > >
> > > > > > > > > > In Malhar there are relatively smaller number of
> > operators
> > > > that
> > > > > > we
> > > > > > > > use
> > > > > > > > > in
> > > > > > > > > > our demo applications, customer applications, POCs etc
> that
> > > are
> > > > > > > mature.
> > > > > > > > > >
> > > > > > > > > > The library is cluttered with operators especially in
> > > lib/util,
> > > > > > > > lib/algo,
> > > > > > > > > > lib/math packages which can be cleaned up by either
> > removing
> > > > them
> > > > > > or
> > > > > > > > > > improving them but that breaks semantic versioning.
> > > > > > > > > >
> > > > > > > > > > When we add new operators/utilities it takes certain time
> > for
> > > > > them
> > > > > > to
> > > > > > > > > > mature. Japicmp doesn't help because it doesn't honor
> > > @Evolving
> > > > > > > > @Unstable
> > > > > > > > > > annotations for now.
> > > > > > > > > >
> > > > > > > > > > I wanted to propose that we add an annotation, let's say,
> > > > re-use
> > > > > > > > hadoop's
> > > > > > > > > > @Stable and mark the operators which are stable with it
> and
> > > > > perform
> > > > > > > > > semver
> > > > > > > > > > check on just these operators.
> > > > > > > > > >
> > > > > > > > > > The 0.7.0 version of japi cmp has the support for
> > inclusions
> > > > (as
> > > > > > well
> > > > > > > > as
> > > > > > > > > > exclusions) based on annotations.
> > > > > > > > > >
> > > > > > > > > > Here is the info:
> > > > > > > > > > https://github.com/siom79/japicmp/issues/88
> > > > > > > > > >
> > > > > > > > > > The reason I am inclined to the inclusion approach is
> that
> > > > there
> > > > > > are
> > > > > > > > > > relatively smaller number of operators which IMO are
> > stable.
> > > A
> > > > > lot
> > > > > > of
> > > > > > > > > them
> > > > > > > > > > aren't.
> > > > > > > > > > So instead of going and marking so many as Evolving, we
> > will
> > > > mark
> > > > > > > > > > relatively few of them as stable.
> > > > > > > > > >
> > > > > > > > > > Also new development can be facilitated by this. We
> > wouldn't
> > > > have
> > > > > > to
> > > > > > > > add
> > > > > > > > > > @Evolving to everything which is new. Instead we will
> mark
> > it
> > > > > > @Stable
> > > > > > > > > when
> > > > > > > > > > it is.
> > > > > > > > > >
> > > > > > > > > > Please let me know what you think?
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Chandni
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Isha Arkatkar <is...@datatorrent.com>.
Yep, That's what I am doing now :)
Thanks,
Isha
On Mon, Dec 14, 2015 at 9:22 PM, Chandni Singh <ch...@datatorrent.com>
wrote:
> Isha,
>
> I think for now you can configure the japicmp plugin to exclude the package
> as follows in the pom.xml.
>
> <excludes>
>
> <exclude>com.datatorrent.lib.parser.*</exclude>
>
> </excludes>
>
> This is an example where we can benefit from inclusion approach with
> japicmp 0.7 version.
>
>
> Thanks,
>
> Chandni
>
>
> On Mon, Dec 14, 2015 at 8:34 PM, Isha Arkatkar <is...@datatorrent.com>
> wrote:
>
> > +1
> > When 0.7 version of japicmp is available, we can add exclusions for
> > @Evolving or inclusions for @Stable, whichever way is finalized.
> >
> > But before that should we add package exclusions individually if all the
> > operators inside the package are marked Evolving?
> > I wanted to make changes to some of the parser operators in Malhar. But
> > changing those breaks sem version check.
> >
> > Please suggest.
> >
> > Thanks,
> > Isha
> >
> > On Sun, Dec 13, 2015 at 10:46 PM, Tushar Gosavi <tu...@datatorrent.com>
> > wrote:
> >
> > > +1
> > > The new operators added will most likely to undergo change frequently
> > until
> > > they become stable.
> > >
> > > - Tushar.
> > >
> > >
> > > On Mon, Dec 14, 2015 at 12:05 PM, Siyuan Hua <si...@datatorrent.com>
> > > wrote:
> > >
> > > > +1
> > > >
> > > > On Sun, Dec 13, 2015 at 10:01 PM, Yogi Devendra <
> > yogidevendra@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > I am not against the Idea of using @Stable instead of marking every
> > new
> > > > > @Evolving. I agree that, it would convenient to have @Stable.
> > > > >
> > > > > Just raised the point which needs further discussion, so that we
> get
> > > > > suggestions from the mentors and community.
> > > > >
> > > > > ~ Yogi
> > > > >
> > > > > On 14 December 2015 at 11:23, Chandni Singh <
> chandni@datatorrent.com
> > >
> > > > > wrote:
> > > > >
> > > > > > Can we define some criteria for deciding when to consider
> operator
> > as
> > > > > > @Stable?
> > > > > > Yes, we should follow Hadoop's example and formalize some
> criteria.
> > > > > >
> > > > > > [It would be difficult for an open source project to track which
> > user
> > > > is
> > > > > > using which operators. So, above strategy may not work. ]
> > > > > > Hadoop is an open source project which actually created these
> > > > annotations
> > > > > > and it's widely used. I think any new development takes time to
> > > become
> > > > > > stable.
> > > > > > If the operators are NOT marked as @Stable, users will know that
> > when
> > > > > they
> > > > > > upgrade backward compatibility may be broken.
> > > > > > I think it has the same affect of marking every new
> operator/class
> > as
> > > > > > @Evolving. The benefit of checking semantic versioning of
> "Stable"
> > > > > > operators is that they are currently fewer in number and IMO easy
> > to
> > > > > manage
> > > > > > and new development will be implicitly "Evolving".
> > > > > >
> > > > > > Chandni
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <
> > > > yogidevendra@apache.org>
> > > > > > wrote:
> > > > > >
> > > > > > > When to mark certain operator as @Stable is not clearly
> defined.
> > > > > > >
> > > > > > > Can we define some criteria for deciding when to consider
> > operator
> > > as
> > > > > > > @Stable?
> > > > > > >
> > > > > > > For example one criteria could be, if operator is running for
> >1
> > > year
> > > > > in
> > > > > > > production environment for some user. Can we come with some
> > > strategy
> > > > > like
> > > > > > > this?
> > > > > > > [It would be difficult for an open source project to track
> which
> > > user
> > > > > is
> > > > > > > using which operators. So, above strategy may not work. ]
> > > > > > >
> > > > > > > ~ Yogi
> > > > > > >
> > > > > > > On 14 December 2015 at 05:42, Timothy Farkas <
> > tim@datatorrent.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > +1
> > > > > > > > On Dec 13, 2015 4:08 PM, "Chandni Singh" <
> > > chandni@datatorrent.com>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi,
> > > > > > > > >
> > > > > > > > > In Malhar there are relatively smaller number of
> operators
> > > that
> > > > > we
> > > > > > > use
> > > > > > > > in
> > > > > > > > > our demo applications, customer applications, POCs etc that
> > are
> > > > > > mature.
> > > > > > > > >
> > > > > > > > > The library is cluttered with operators especially in
> > lib/util,
> > > > > > > lib/algo,
> > > > > > > > > lib/math packages which can be cleaned up by either
> removing
> > > them
> > > > > or
> > > > > > > > > improving them but that breaks semantic versioning.
> > > > > > > > >
> > > > > > > > > When we add new operators/utilities it takes certain time
> for
> > > > them
> > > > > to
> > > > > > > > > mature. Japicmp doesn't help because it doesn't honor
> > @Evolving
> > > > > > > @Unstable
> > > > > > > > > annotations for now.
> > > > > > > > >
> > > > > > > > > I wanted to propose that we add an annotation, let's say,
> > > re-use
> > > > > > > hadoop's
> > > > > > > > > @Stable and mark the operators which are stable with it and
> > > > perform
> > > > > > > > semver
> > > > > > > > > check on just these operators.
> > > > > > > > >
> > > > > > > > > The 0.7.0 version of japi cmp has the support for
> inclusions
> > > (as
> > > > > well
> > > > > > > as
> > > > > > > > > exclusions) based on annotations.
> > > > > > > > >
> > > > > > > > > Here is the info:
> > > > > > > > > https://github.com/siom79/japicmp/issues/88
> > > > > > > > >
> > > > > > > > > The reason I am inclined to the inclusion approach is that
> > > there
> > > > > are
> > > > > > > > > relatively smaller number of operators which IMO are
> stable.
> > A
> > > > lot
> > > > > of
> > > > > > > > them
> > > > > > > > > aren't.
> > > > > > > > > So instead of going and marking so many as Evolving, we
> will
> > > mark
> > > > > > > > > relatively few of them as stable.
> > > > > > > > >
> > > > > > > > > Also new development can be facilitated by this. We
> wouldn't
> > > have
> > > > > to
> > > > > > > add
> > > > > > > > > @Evolving to everything which is new. Instead we will mark
> it
> > > > > @Stable
> > > > > > > > when
> > > > > > > > > it is.
> > > > > > > > >
> > > > > > > > > Please let me know what you think?
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Chandni
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Chandni Singh <ch...@datatorrent.com>.
Isha,
I think for now you can configure the japicmp plugin to exclude the package
as follows in the pom.xml.
<excludes>
<exclude>com.datatorrent.lib.parser.*</exclude>
</excludes>
This is an example where we can benefit from inclusion approach with
japicmp 0.7 version.
Thanks,
Chandni
On Mon, Dec 14, 2015 at 8:34 PM, Isha Arkatkar <is...@datatorrent.com> wrote:
> +1
> When 0.7 version of japicmp is available, we can add exclusions for
> @Evolving or inclusions for @Stable, whichever way is finalized.
>
> But before that should we add package exclusions individually if all the
> operators inside the package are marked Evolving?
> I wanted to make changes to some of the parser operators in Malhar. But
> changing those breaks sem version check.
>
> Please suggest.
>
> Thanks,
> Isha
>
> On Sun, Dec 13, 2015 at 10:46 PM, Tushar Gosavi <tu...@datatorrent.com>
> wrote:
>
> > +1
> > The new operators added will most likely to undergo change frequently
> until
> > they become stable.
> >
> > - Tushar.
> >
> >
> > On Mon, Dec 14, 2015 at 12:05 PM, Siyuan Hua <si...@datatorrent.com>
> > wrote:
> >
> > > +1
> > >
> > > On Sun, Dec 13, 2015 at 10:01 PM, Yogi Devendra <
> yogidevendra@apache.org
> > >
> > > wrote:
> > >
> > > > I am not against the Idea of using @Stable instead of marking every
> new
> > > > @Evolving. I agree that, it would convenient to have @Stable.
> > > >
> > > > Just raised the point which needs further discussion, so that we get
> > > > suggestions from the mentors and community.
> > > >
> > > > ~ Yogi
> > > >
> > > > On 14 December 2015 at 11:23, Chandni Singh <chandni@datatorrent.com
> >
> > > > wrote:
> > > >
> > > > > Can we define some criteria for deciding when to consider operator
> as
> > > > > @Stable?
> > > > > Yes, we should follow Hadoop's example and formalize some criteria.
> > > > >
> > > > > [It would be difficult for an open source project to track which
> user
> > > is
> > > > > using which operators. So, above strategy may not work. ]
> > > > > Hadoop is an open source project which actually created these
> > > annotations
> > > > > and it's widely used. I think any new development takes time to
> > become
> > > > > stable.
> > > > > If the operators are NOT marked as @Stable, users will know that
> when
> > > > they
> > > > > upgrade backward compatibility may be broken.
> > > > > I think it has the same affect of marking every new operator/class
> as
> > > > > @Evolving. The benefit of checking semantic versioning of "Stable"
> > > > > operators is that they are currently fewer in number and IMO easy
> to
> > > > manage
> > > > > and new development will be implicitly "Evolving".
> > > > >
> > > > > Chandni
> > > > >
> > > > >
> > > > >
> > > > > On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <
> > > yogidevendra@apache.org>
> > > > > wrote:
> > > > >
> > > > > > When to mark certain operator as @Stable is not clearly defined.
> > > > > >
> > > > > > Can we define some criteria for deciding when to consider
> operator
> > as
> > > > > > @Stable?
> > > > > >
> > > > > > For example one criteria could be, if operator is running for >1
> > year
> > > > in
> > > > > > production environment for some user. Can we come with some
> > strategy
> > > > like
> > > > > > this?
> > > > > > [It would be difficult for an open source project to track which
> > user
> > > > is
> > > > > > using which operators. So, above strategy may not work. ]
> > > > > >
> > > > > > ~ Yogi
> > > > > >
> > > > > > On 14 December 2015 at 05:42, Timothy Farkas <
> tim@datatorrent.com>
> > > > > wrote:
> > > > > >
> > > > > > > +1
> > > > > > > On Dec 13, 2015 4:08 PM, "Chandni Singh" <
> > chandni@datatorrent.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi,
> > > > > > > >
> > > > > > > > In Malhar there are relatively smaller number of operators
> > that
> > > > we
> > > > > > use
> > > > > > > in
> > > > > > > > our demo applications, customer applications, POCs etc that
> are
> > > > > mature.
> > > > > > > >
> > > > > > > > The library is cluttered with operators especially in
> lib/util,
> > > > > > lib/algo,
> > > > > > > > lib/math packages which can be cleaned up by either removing
> > them
> > > > or
> > > > > > > > improving them but that breaks semantic versioning.
> > > > > > > >
> > > > > > > > When we add new operators/utilities it takes certain time for
> > > them
> > > > to
> > > > > > > > mature. Japicmp doesn't help because it doesn't honor
> @Evolving
> > > > > > @Unstable
> > > > > > > > annotations for now.
> > > > > > > >
> > > > > > > > I wanted to propose that we add an annotation, let's say,
> > re-use
> > > > > > hadoop's
> > > > > > > > @Stable and mark the operators which are stable with it and
> > > perform
> > > > > > > semver
> > > > > > > > check on just these operators.
> > > > > > > >
> > > > > > > > The 0.7.0 version of japi cmp has the support for inclusions
> > (as
> > > > well
> > > > > > as
> > > > > > > > exclusions) based on annotations.
> > > > > > > >
> > > > > > > > Here is the info:
> > > > > > > > https://github.com/siom79/japicmp/issues/88
> > > > > > > >
> > > > > > > > The reason I am inclined to the inclusion approach is that
> > there
> > > > are
> > > > > > > > relatively smaller number of operators which IMO are stable.
> A
> > > lot
> > > > of
> > > > > > > them
> > > > > > > > aren't.
> > > > > > > > So instead of going and marking so many as Evolving, we will
> > mark
> > > > > > > > relatively few of them as stable.
> > > > > > > >
> > > > > > > > Also new development can be facilitated by this. We wouldn't
> > have
> > > > to
> > > > > > add
> > > > > > > > @Evolving to everything which is new. Instead we will mark it
> > > > @Stable
> > > > > > > when
> > > > > > > > it is.
> > > > > > > >
> > > > > > > > Please let me know what you think?
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Chandni
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Isha Arkatkar <is...@datatorrent.com>.
+1
When 0.7 version of japicmp is available, we can add exclusions for
@Evolving or inclusions for @Stable, whichever way is finalized.
But before that should we add package exclusions individually if all the
operators inside the package are marked Evolving?
I wanted to make changes to some of the parser operators in Malhar. But
changing those breaks sem version check.
Please suggest.
Thanks,
Isha
On Sun, Dec 13, 2015 at 10:46 PM, Tushar Gosavi <tu...@datatorrent.com>
wrote:
> +1
> The new operators added will most likely to undergo change frequently until
> they become stable.
>
> - Tushar.
>
>
> On Mon, Dec 14, 2015 at 12:05 PM, Siyuan Hua <si...@datatorrent.com>
> wrote:
>
> > +1
> >
> > On Sun, Dec 13, 2015 at 10:01 PM, Yogi Devendra <yogidevendra@apache.org
> >
> > wrote:
> >
> > > I am not against the Idea of using @Stable instead of marking every new
> > > @Evolving. I agree that, it would convenient to have @Stable.
> > >
> > > Just raised the point which needs further discussion, so that we get
> > > suggestions from the mentors and community.
> > >
> > > ~ Yogi
> > >
> > > On 14 December 2015 at 11:23, Chandni Singh <ch...@datatorrent.com>
> > > wrote:
> > >
> > > > Can we define some criteria for deciding when to consider operator as
> > > > @Stable?
> > > > Yes, we should follow Hadoop's example and formalize some criteria.
> > > >
> > > > [It would be difficult for an open source project to track which user
> > is
> > > > using which operators. So, above strategy may not work. ]
> > > > Hadoop is an open source project which actually created these
> > annotations
> > > > and it's widely used. I think any new development takes time to
> become
> > > > stable.
> > > > If the operators are NOT marked as @Stable, users will know that when
> > > they
> > > > upgrade backward compatibility may be broken.
> > > > I think it has the same affect of marking every new operator/class as
> > > > @Evolving. The benefit of checking semantic versioning of "Stable"
> > > > operators is that they are currently fewer in number and IMO easy to
> > > manage
> > > > and new development will be implicitly "Evolving".
> > > >
> > > > Chandni
> > > >
> > > >
> > > >
> > > > On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <
> > yogidevendra@apache.org>
> > > > wrote:
> > > >
> > > > > When to mark certain operator as @Stable is not clearly defined.
> > > > >
> > > > > Can we define some criteria for deciding when to consider operator
> as
> > > > > @Stable?
> > > > >
> > > > > For example one criteria could be, if operator is running for >1
> year
> > > in
> > > > > production environment for some user. Can we come with some
> strategy
> > > like
> > > > > this?
> > > > > [It would be difficult for an open source project to track which
> user
> > > is
> > > > > using which operators. So, above strategy may not work. ]
> > > > >
> > > > > ~ Yogi
> > > > >
> > > > > On 14 December 2015 at 05:42, Timothy Farkas <ti...@datatorrent.com>
> > > > wrote:
> > > > >
> > > > > > +1
> > > > > > On Dec 13, 2015 4:08 PM, "Chandni Singh" <
> chandni@datatorrent.com>
> > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > In Malhar there are relatively smaller number of operators
> that
> > > we
> > > > > use
> > > > > > in
> > > > > > > our demo applications, customer applications, POCs etc that are
> > > > mature.
> > > > > > >
> > > > > > > The library is cluttered with operators especially in lib/util,
> > > > > lib/algo,
> > > > > > > lib/math packages which can be cleaned up by either removing
> them
> > > or
> > > > > > > improving them but that breaks semantic versioning.
> > > > > > >
> > > > > > > When we add new operators/utilities it takes certain time for
> > them
> > > to
> > > > > > > mature. Japicmp doesn't help because it doesn't honor @Evolving
> > > > > @Unstable
> > > > > > > annotations for now.
> > > > > > >
> > > > > > > I wanted to propose that we add an annotation, let's say,
> re-use
> > > > > hadoop's
> > > > > > > @Stable and mark the operators which are stable with it and
> > perform
> > > > > > semver
> > > > > > > check on just these operators.
> > > > > > >
> > > > > > > The 0.7.0 version of japi cmp has the support for inclusions
> (as
> > > well
> > > > > as
> > > > > > > exclusions) based on annotations.
> > > > > > >
> > > > > > > Here is the info:
> > > > > > > https://github.com/siom79/japicmp/issues/88
> > > > > > >
> > > > > > > The reason I am inclined to the inclusion approach is that
> there
> > > are
> > > > > > > relatively smaller number of operators which IMO are stable. A
> > lot
> > > of
> > > > > > them
> > > > > > > aren't.
> > > > > > > So instead of going and marking so many as Evolving, we will
> mark
> > > > > > > relatively few of them as stable.
> > > > > > >
> > > > > > > Also new development can be facilitated by this. We wouldn't
> have
> > > to
> > > > > add
> > > > > > > @Evolving to everything which is new. Instead we will mark it
> > > @Stable
> > > > > > when
> > > > > > > it is.
> > > > > > >
> > > > > > > Please let me know what you think?
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Chandni
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Tushar Gosavi <tu...@datatorrent.com>.
+1
The new operators added will most likely to undergo change frequently until
they become stable.
- Tushar.
On Mon, Dec 14, 2015 at 12:05 PM, Siyuan Hua <si...@datatorrent.com> wrote:
> +1
>
> On Sun, Dec 13, 2015 at 10:01 PM, Yogi Devendra <yo...@apache.org>
> wrote:
>
> > I am not against the Idea of using @Stable instead of marking every new
> > @Evolving. I agree that, it would convenient to have @Stable.
> >
> > Just raised the point which needs further discussion, so that we get
> > suggestions from the mentors and community.
> >
> > ~ Yogi
> >
> > On 14 December 2015 at 11:23, Chandni Singh <ch...@datatorrent.com>
> > wrote:
> >
> > > Can we define some criteria for deciding when to consider operator as
> > > @Stable?
> > > Yes, we should follow Hadoop's example and formalize some criteria.
> > >
> > > [It would be difficult for an open source project to track which user
> is
> > > using which operators. So, above strategy may not work. ]
> > > Hadoop is an open source project which actually created these
> annotations
> > > and it's widely used. I think any new development takes time to become
> > > stable.
> > > If the operators are NOT marked as @Stable, users will know that when
> > they
> > > upgrade backward compatibility may be broken.
> > > I think it has the same affect of marking every new operator/class as
> > > @Evolving. The benefit of checking semantic versioning of "Stable"
> > > operators is that they are currently fewer in number and IMO easy to
> > manage
> > > and new development will be implicitly "Evolving".
> > >
> > > Chandni
> > >
> > >
> > >
> > > On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <
> yogidevendra@apache.org>
> > > wrote:
> > >
> > > > When to mark certain operator as @Stable is not clearly defined.
> > > >
> > > > Can we define some criteria for deciding when to consider operator as
> > > > @Stable?
> > > >
> > > > For example one criteria could be, if operator is running for >1 year
> > in
> > > > production environment for some user. Can we come with some strategy
> > like
> > > > this?
> > > > [It would be difficult for an open source project to track which user
> > is
> > > > using which operators. So, above strategy may not work. ]
> > > >
> > > > ~ Yogi
> > > >
> > > > On 14 December 2015 at 05:42, Timothy Farkas <ti...@datatorrent.com>
> > > wrote:
> > > >
> > > > > +1
> > > > > On Dec 13, 2015 4:08 PM, "Chandni Singh" <ch...@datatorrent.com>
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > In Malhar there are relatively smaller number of operators that
> > we
> > > > use
> > > > > in
> > > > > > our demo applications, customer applications, POCs etc that are
> > > mature.
> > > > > >
> > > > > > The library is cluttered with operators especially in lib/util,
> > > > lib/algo,
> > > > > > lib/math packages which can be cleaned up by either removing them
> > or
> > > > > > improving them but that breaks semantic versioning.
> > > > > >
> > > > > > When we add new operators/utilities it takes certain time for
> them
> > to
> > > > > > mature. Japicmp doesn't help because it doesn't honor @Evolving
> > > > @Unstable
> > > > > > annotations for now.
> > > > > >
> > > > > > I wanted to propose that we add an annotation, let's say, re-use
> > > > hadoop's
> > > > > > @Stable and mark the operators which are stable with it and
> perform
> > > > > semver
> > > > > > check on just these operators.
> > > > > >
> > > > > > The 0.7.0 version of japi cmp has the support for inclusions (as
> > well
> > > > as
> > > > > > exclusions) based on annotations.
> > > > > >
> > > > > > Here is the info:
> > > > > > https://github.com/siom79/japicmp/issues/88
> > > > > >
> > > > > > The reason I am inclined to the inclusion approach is that there
> > are
> > > > > > relatively smaller number of operators which IMO are stable. A
> lot
> > of
> > > > > them
> > > > > > aren't.
> > > > > > So instead of going and marking so many as Evolving, we will mark
> > > > > > relatively few of them as stable.
> > > > > >
> > > > > > Also new development can be facilitated by this. We wouldn't have
> > to
> > > > add
> > > > > > @Evolving to everything which is new. Instead we will mark it
> > @Stable
> > > > > when
> > > > > > it is.
> > > > > >
> > > > > > Please let me know what you think?
> > > > > >
> > > > > > Thanks,
> > > > > > Chandni
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Siyuan Hua <si...@datatorrent.com>.
+1
On Sun, Dec 13, 2015 at 10:01 PM, Yogi Devendra <yo...@apache.org>
wrote:
> I am not against the Idea of using @Stable instead of marking every new
> @Evolving. I agree that, it would convenient to have @Stable.
>
> Just raised the point which needs further discussion, so that we get
> suggestions from the mentors and community.
>
> ~ Yogi
>
> On 14 December 2015 at 11:23, Chandni Singh <ch...@datatorrent.com>
> wrote:
>
> > Can we define some criteria for deciding when to consider operator as
> > @Stable?
> > Yes, we should follow Hadoop's example and formalize some criteria.
> >
> > [It would be difficult for an open source project to track which user is
> > using which operators. So, above strategy may not work. ]
> > Hadoop is an open source project which actually created these annotations
> > and it's widely used. I think any new development takes time to become
> > stable.
> > If the operators are NOT marked as @Stable, users will know that when
> they
> > upgrade backward compatibility may be broken.
> > I think it has the same affect of marking every new operator/class as
> > @Evolving. The benefit of checking semantic versioning of "Stable"
> > operators is that they are currently fewer in number and IMO easy to
> manage
> > and new development will be implicitly "Evolving".
> >
> > Chandni
> >
> >
> >
> > On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <yo...@apache.org>
> > wrote:
> >
> > > When to mark certain operator as @Stable is not clearly defined.
> > >
> > > Can we define some criteria for deciding when to consider operator as
> > > @Stable?
> > >
> > > For example one criteria could be, if operator is running for >1 year
> in
> > > production environment for some user. Can we come with some strategy
> like
> > > this?
> > > [It would be difficult for an open source project to track which user
> is
> > > using which operators. So, above strategy may not work. ]
> > >
> > > ~ Yogi
> > >
> > > On 14 December 2015 at 05:42, Timothy Farkas <ti...@datatorrent.com>
> > wrote:
> > >
> > > > +1
> > > > On Dec 13, 2015 4:08 PM, "Chandni Singh" <ch...@datatorrent.com>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > In Malhar there are relatively smaller number of operators that
> we
> > > use
> > > > in
> > > > > our demo applications, customer applications, POCs etc that are
> > mature.
> > > > >
> > > > > The library is cluttered with operators especially in lib/util,
> > > lib/algo,
> > > > > lib/math packages which can be cleaned up by either removing them
> or
> > > > > improving them but that breaks semantic versioning.
> > > > >
> > > > > When we add new operators/utilities it takes certain time for them
> to
> > > > > mature. Japicmp doesn't help because it doesn't honor @Evolving
> > > @Unstable
> > > > > annotations for now.
> > > > >
> > > > > I wanted to propose that we add an annotation, let's say, re-use
> > > hadoop's
> > > > > @Stable and mark the operators which are stable with it and perform
> > > > semver
> > > > > check on just these operators.
> > > > >
> > > > > The 0.7.0 version of japi cmp has the support for inclusions (as
> well
> > > as
> > > > > exclusions) based on annotations.
> > > > >
> > > > > Here is the info:
> > > > > https://github.com/siom79/japicmp/issues/88
> > > > >
> > > > > The reason I am inclined to the inclusion approach is that there
> are
> > > > > relatively smaller number of operators which IMO are stable. A lot
> of
> > > > them
> > > > > aren't.
> > > > > So instead of going and marking so many as Evolving, we will mark
> > > > > relatively few of them as stable.
> > > > >
> > > > > Also new development can be facilitated by this. We wouldn't have
> to
> > > add
> > > > > @Evolving to everything which is new. Instead we will mark it
> @Stable
> > > > when
> > > > > it is.
> > > > >
> > > > > Please let me know what you think?
> > > > >
> > > > > Thanks,
> > > > > Chandni
> > > > >
> > > >
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Yogi Devendra <yo...@apache.org>.
I am not against the Idea of using @Stable instead of marking every new
@Evolving. I agree that, it would convenient to have @Stable.
Just raised the point which needs further discussion, so that we get
suggestions from the mentors and community.
~ Yogi
On 14 December 2015 at 11:23, Chandni Singh <ch...@datatorrent.com> wrote:
> Can we define some criteria for deciding when to consider operator as
> @Stable?
> Yes, we should follow Hadoop's example and formalize some criteria.
>
> [It would be difficult for an open source project to track which user is
> using which operators. So, above strategy may not work. ]
> Hadoop is an open source project which actually created these annotations
> and it's widely used. I think any new development takes time to become
> stable.
> If the operators are NOT marked as @Stable, users will know that when they
> upgrade backward compatibility may be broken.
> I think it has the same affect of marking every new operator/class as
> @Evolving. The benefit of checking semantic versioning of "Stable"
> operators is that they are currently fewer in number and IMO easy to manage
> and new development will be implicitly "Evolving".
>
> Chandni
>
>
>
> On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <yo...@apache.org>
> wrote:
>
> > When to mark certain operator as @Stable is not clearly defined.
> >
> > Can we define some criteria for deciding when to consider operator as
> > @Stable?
> >
> > For example one criteria could be, if operator is running for >1 year in
> > production environment for some user. Can we come with some strategy like
> > this?
> > [It would be difficult for an open source project to track which user is
> > using which operators. So, above strategy may not work. ]
> >
> > ~ Yogi
> >
> > On 14 December 2015 at 05:42, Timothy Farkas <ti...@datatorrent.com>
> wrote:
> >
> > > +1
> > > On Dec 13, 2015 4:08 PM, "Chandni Singh" <ch...@datatorrent.com>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > In Malhar there are relatively smaller number of operators that we
> > use
> > > in
> > > > our demo applications, customer applications, POCs etc that are
> mature.
> > > >
> > > > The library is cluttered with operators especially in lib/util,
> > lib/algo,
> > > > lib/math packages which can be cleaned up by either removing them or
> > > > improving them but that breaks semantic versioning.
> > > >
> > > > When we add new operators/utilities it takes certain time for them to
> > > > mature. Japicmp doesn't help because it doesn't honor @Evolving
> > @Unstable
> > > > annotations for now.
> > > >
> > > > I wanted to propose that we add an annotation, let's say, re-use
> > hadoop's
> > > > @Stable and mark the operators which are stable with it and perform
> > > semver
> > > > check on just these operators.
> > > >
> > > > The 0.7.0 version of japi cmp has the support for inclusions (as well
> > as
> > > > exclusions) based on annotations.
> > > >
> > > > Here is the info:
> > > > https://github.com/siom79/japicmp/issues/88
> > > >
> > > > The reason I am inclined to the inclusion approach is that there are
> > > > relatively smaller number of operators which IMO are stable. A lot of
> > > them
> > > > aren't.
> > > > So instead of going and marking so many as Evolving, we will mark
> > > > relatively few of them as stable.
> > > >
> > > > Also new development can be facilitated by this. We wouldn't have to
> > add
> > > > @Evolving to everything which is new. Instead we will mark it @Stable
> > > when
> > > > it is.
> > > >
> > > > Please let me know what you think?
> > > >
> > > > Thanks,
> > > > Chandni
> > > >
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Chandni Singh <ch...@datatorrent.com>.
Can we define some criteria for deciding when to consider operator as
@Stable?
Yes, we should follow Hadoop's example and formalize some criteria.
[It would be difficult for an open source project to track which user is
using which operators. So, above strategy may not work. ]
Hadoop is an open source project which actually created these annotations
and it's widely used. I think any new development takes time to become
stable.
If the operators are NOT marked as @Stable, users will know that when they
upgrade backward compatibility may be broken.
I think it has the same affect of marking every new operator/class as
@Evolving. The benefit of checking semantic versioning of "Stable"
operators is that they are currently fewer in number and IMO easy to manage
and new development will be implicitly "Evolving".
Chandni
On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <yo...@apache.org>
wrote:
> When to mark certain operator as @Stable is not clearly defined.
>
> Can we define some criteria for deciding when to consider operator as
> @Stable?
>
> For example one criteria could be, if operator is running for >1 year in
> production environment for some user. Can we come with some strategy like
> this?
> [It would be difficult for an open source project to track which user is
> using which operators. So, above strategy may not work. ]
>
> ~ Yogi
>
> On 14 December 2015 at 05:42, Timothy Farkas <ti...@datatorrent.com> wrote:
>
> > +1
> > On Dec 13, 2015 4:08 PM, "Chandni Singh" <ch...@datatorrent.com>
> wrote:
> >
> > > Hi,
> > >
> > > In Malhar there are relatively smaller number of operators that we
> use
> > in
> > > our demo applications, customer applications, POCs etc that are mature.
> > >
> > > The library is cluttered with operators especially in lib/util,
> lib/algo,
> > > lib/math packages which can be cleaned up by either removing them or
> > > improving them but that breaks semantic versioning.
> > >
> > > When we add new operators/utilities it takes certain time for them to
> > > mature. Japicmp doesn't help because it doesn't honor @Evolving
> @Unstable
> > > annotations for now.
> > >
> > > I wanted to propose that we add an annotation, let's say, re-use
> hadoop's
> > > @Stable and mark the operators which are stable with it and perform
> > semver
> > > check on just these operators.
> > >
> > > The 0.7.0 version of japi cmp has the support for inclusions (as well
> as
> > > exclusions) based on annotations.
> > >
> > > Here is the info:
> > > https://github.com/siom79/japicmp/issues/88
> > >
> > > The reason I am inclined to the inclusion approach is that there are
> > > relatively smaller number of operators which IMO are stable. A lot of
> > them
> > > aren't.
> > > So instead of going and marking so many as Evolving, we will mark
> > > relatively few of them as stable.
> > >
> > > Also new development can be facilitated by this. We wouldn't have to
> add
> > > @Evolving to everything which is new. Instead we will mark it @Stable
> > when
> > > it is.
> > >
> > > Please let me know what you think?
> > >
> > > Thanks,
> > > Chandni
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Amol Kekre <am...@datatorrent.com>.
Mentors,
Is there a common convention for usage of @Stable in Hadoop community?
Thks,
Amol
On Sun, Dec 13, 2015 at 9:11 PM, Yogi Devendra <yo...@apache.org>
wrote:
> When to mark certain operator as @Stable is not clearly defined.
>
> Can we define some criteria for deciding when to consider operator as
> @Stable?
>
> For example one criteria could be, if operator is running for >1 year in
> production environment for some user. Can we come with some strategy like
> this?
> [It would be difficult for an open source project to track which user is
> using which operators. So, above strategy may not work. ]
>
> ~ Yogi
>
> On 14 December 2015 at 05:42, Timothy Farkas <ti...@datatorrent.com> wrote:
>
> > +1
> > On Dec 13, 2015 4:08 PM, "Chandni Singh" <ch...@datatorrent.com>
> wrote:
> >
> > > Hi,
> > >
> > > In Malhar there are relatively smaller number of operators that we
> use
> > in
> > > our demo applications, customer applications, POCs etc that are mature.
> > >
> > > The library is cluttered with operators especially in lib/util,
> lib/algo,
> > > lib/math packages which can be cleaned up by either removing them or
> > > improving them but that breaks semantic versioning.
> > >
> > > When we add new operators/utilities it takes certain time for them to
> > > mature. Japicmp doesn't help because it doesn't honor @Evolving
> @Unstable
> > > annotations for now.
> > >
> > > I wanted to propose that we add an annotation, let's say, re-use
> hadoop's
> > > @Stable and mark the operators which are stable with it and perform
> > semver
> > > check on just these operators.
> > >
> > > The 0.7.0 version of japi cmp has the support for inclusions (as well
> as
> > > exclusions) based on annotations.
> > >
> > > Here is the info:
> > > https://github.com/siom79/japicmp/issues/88
> > >
> > > The reason I am inclined to the inclusion approach is that there are
> > > relatively smaller number of operators which IMO are stable. A lot of
> > them
> > > aren't.
> > > So instead of going and marking so many as Evolving, we will mark
> > > relatively few of them as stable.
> > >
> > > Also new development can be facilitated by this. We wouldn't have to
> add
> > > @Evolving to everything which is new. Instead we will mark it @Stable
> > when
> > > it is.
> > >
> > > Please let me know what you think?
> > >
> > > Thanks,
> > > Chandni
> > >
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Yogi Devendra <yo...@apache.org>.
When to mark certain operator as @Stable is not clearly defined.
Can we define some criteria for deciding when to consider operator as
@Stable?
For example one criteria could be, if operator is running for >1 year in
production environment for some user. Can we come with some strategy like
this?
[It would be difficult for an open source project to track which user is
using which operators. So, above strategy may not work. ]
~ Yogi
On 14 December 2015 at 05:42, Timothy Farkas <ti...@datatorrent.com> wrote:
> +1
> On Dec 13, 2015 4:08 PM, "Chandni Singh" <ch...@datatorrent.com> wrote:
>
> > Hi,
> >
> > In Malhar there are relatively smaller number of operators that we use
> in
> > our demo applications, customer applications, POCs etc that are mature.
> >
> > The library is cluttered with operators especially in lib/util, lib/algo,
> > lib/math packages which can be cleaned up by either removing them or
> > improving them but that breaks semantic versioning.
> >
> > When we add new operators/utilities it takes certain time for them to
> > mature. Japicmp doesn't help because it doesn't honor @Evolving @Unstable
> > annotations for now.
> >
> > I wanted to propose that we add an annotation, let's say, re-use hadoop's
> > @Stable and mark the operators which are stable with it and perform
> semver
> > check on just these operators.
> >
> > The 0.7.0 version of japi cmp has the support for inclusions (as well as
> > exclusions) based on annotations.
> >
> > Here is the info:
> > https://github.com/siom79/japicmp/issues/88
> >
> > The reason I am inclined to the inclusion approach is that there are
> > relatively smaller number of operators which IMO are stable. A lot of
> them
> > aren't.
> > So instead of going and marking so many as Evolving, we will mark
> > relatively few of them as stable.
> >
> > Also new development can be facilitated by this. We wouldn't have to add
> > @Evolving to everything which is new. Instead we will mark it @Stable
> when
> > it is.
> >
> > Please let me know what you think?
> >
> > Thanks,
> > Chandni
> >
>
Re: Enable semantic versioning only for specific operators in Malhar
Posted by Timothy Farkas <ti...@datatorrent.com>.
+1
On Dec 13, 2015 4:08 PM, "Chandni Singh" <ch...@datatorrent.com> wrote:
> Hi,
>
> In Malhar there are relatively smaller number of operators that we use in
> our demo applications, customer applications, POCs etc that are mature.
>
> The library is cluttered with operators especially in lib/util, lib/algo,
> lib/math packages which can be cleaned up by either removing them or
> improving them but that breaks semantic versioning.
>
> When we add new operators/utilities it takes certain time for them to
> mature. Japicmp doesn't help because it doesn't honor @Evolving @Unstable
> annotations for now.
>
> I wanted to propose that we add an annotation, let's say, re-use hadoop's
> @Stable and mark the operators which are stable with it and perform semver
> check on just these operators.
>
> The 0.7.0 version of japi cmp has the support for inclusions (as well as
> exclusions) based on annotations.
>
> Here is the info:
> https://github.com/siom79/japicmp/issues/88
>
> The reason I am inclined to the inclusion approach is that there are
> relatively smaller number of operators which IMO are stable. A lot of them
> aren't.
> So instead of going and marking so many as Evolving, we will mark
> relatively few of them as stable.
>
> Also new development can be facilitated by this. We wouldn't have to add
> @Evolving to everything which is new. Instead we will mark it @Stable when
> it is.
>
> Please let me know what you think?
>
> Thanks,
> Chandni
>